Does Anyone Really Grasp The Complexity of Operations?

[Deleted member 5399]

Unless you work within any of the attached environments; most of us take for granted that Google, Yahoo, Microsoft, or the internet is always there for us.
Only Free2Play understands or knows just how MANY servers GI needs. Not just for running AC, but for all its games. These inside looks are just from Google sever farms alone. And it isn't all wires and lights that go into this. There are massive cooling fans, fire suppressant systems as well. Without keeping the heat and humidity at bay, it would get seriously hot inside these rooms.
And when they say they are having server problems, well WHICH ONE IS IT?

 

[Deleted member 5399]

The next time you think you've got it bad troubleshooting your desktop or laptop, think of what it's like for these people.

 

[Canadian Cowboy]

I worked in IT for 33 years, as a systems support person, mainframe mostly. I can sympathize with GI Techs. I had to take my turn going oncall (cell phone with 24/7 call out), which meant that you had to stay home evenings for the week when you were oncall. You had to be able to logon within 10 minutes of getting the phone call. The worst was getting a Severity 1 call at 3:30 AM, when you are fighting to wake up from being in a deep sleep. I do not miss it, though I do miss the great people I worked with over the years.

 

[AshPort]

Unless you work within any of the attached environments; most of us take for granted that Google, Yahoo, Microsoft, or the internet is always there for us.
Only Free2Play understands or knows just how MANY servers GI needs. Not just for running AC, but for all its games. These inside looks are just from Google sever farms alone. And it isn't all wires and lights that go into this. There are massive cooling fans, fire suppressant systems as well. Without keeping the heat and humidity at bay, it would get seriously hot inside these rooms.
And when they say they are having server problems, well WHICH ONE IS IT?View attachment 11761 View attachment 11762 View attachment 11763 View attachment 11764

well theres your global warming problem

 

[Davescharf]

Those centers are so energy efficient now. I’m an IT manager and technology isn’t easy. I have a lot of empathy but I’m also pretty critical about the ‘dumb bugs’ because it tells me a lot about certain aspects of the development process.

I know people are irritated about the maintenance window but if the work involves hardware and software it can easily take days. I’ve been part of many project teams where an upgrade lasts 3-4 days if things go according to plan.

 

[Wired Parrott]

I'll have to pretty much agree with all of @Davescharf comments. Though I have
not seen GI comparing it to a server farm is not a fair comparison. As Dave stated
though depending on issues found things could go sideways very quickly causing a
much longer delay. Without any inside knowledge there is no way to determine how
everything will shake out. From my understanding with former colleagues server farms
are actually much easier to deal with in this day and age. Now back 30+ years ago an
issue within a large server network was a mighty pain in the arse.
Then again I'm not complaining because of the maintenance.

 

[PhactorPhil]

Well, at least we found something to chat about while we aren't able to visit neighbors or send gifts.

Wholeheartedly agree with @Davescharf and @Canadian Cowboy. Here's a couple stories for ya to laugh at with me ...

During my time as a Service Relationship Manager responsible for SLA compliance, customer satisfaction, and the 2nd escalation point of contact for strategic global customers, I had the joy of being woken up by a phone call at 4:00am the morning of the US Thanksgiving holiday back in 2013 from one of our high-maintenance customers from London who had a dedicated pipe to our data center in Atlanta. The complaint was that service was down and the 1st escalation POC couldn't be reached. After gathering the S1 incident response team - which was a joke by the way - we discover the issue was on their end and they hadn't even completed their troubleshooting checklist before escalating to us. Talk about a wasted 90 minutes of my life

Then there was the 2:00am Sunday morning call where all hell is breaking loose with a customer and even though I transitioned to a new role and no longer was an escalation point of contact, I somehow got the phone call to solve a problem that wasn't even a problem because it was occurring during a maintenance window where the customer received advanced notification of a 4-hour potential downtime

Then there was the time the company upgraded a propriety SaaS product, but failed to load test it in the QA cycle and it crashed at 85% of load and dropped over 5,000 concurrent connected sessions.

And here's the kicker that always makes me laugh today because of its ridiculousness, but I was so angry because few heeded my common sense warnings I almost quit in the middle of an incident. Get this. After Super Storm Sandy back in 2012, my company decided to move its hosted data center from New York City to Atlanta. So we send a maintenance notice telling customers service will be offline from x time Saturday morning to x time Sunday evening. And here's what they did ... they packed a bunch of equipment up in a U-Haul (a rented lorry for my British friends) and drove through the night to Atlanta without properly securing and handling the cargo. They finally finish installation and are surprised it doesn't work. So as customers come online Monday morning and service is still down ... you can imagine the contact center and ELT (executive leadership team) escalations that brought with it. Who risks a customer base by transporting mission-critical hardware several hundred miles in a U-Haul overnight?!?!?!?!?!?

 

[Southoftheborder]

.
I know people are irritated about the maintenance window but if the work involves hardware and software it can easily take days. I’ve been part of many project teams where an upgrade lasts 3-4 days if things go according to plan.

Yes, people are irritated - and rightly so!

You hit the nail on the head 'if things go according to plan'!

GI are a commercial company. making money from their customers (us).

As such, they are providing a service. Yes occasionally updates need to be made, but presumably their plan indicates how long this should take, how long it might take, and a worst case scenario (cancel the updgrade and roll back).

They should tell us the worst case scenario up front, than as the hours/days/weeks progress, update their customers (us) as to progress: eg "we're 4 hours into the upgrade - all going to plan - still on target to complete by 16:00".

It's called communication!

It doesn't matter if it's good or bad, any reputable service delivery company needs to keep their customers informed - otherwise they will vote with their feet!

So GI - what was the plan, and how is it progressing?

 

[Free2Play]

The plan is good, it's progressing good. It should be over later tonight if all goes well.
Airport City has crossed 60 million installs sometime last year, it takes some time to do anything with a database of that size.

 

[Chazzy999]

Airport City has crossed 60 million installs sometime last year

mind boggling

 

[andy8758]

I, and many other customers have been complaining about "server" issues for some time. I welcome GIS taking down the servers to resolve the problem - I am not an IT specialist, yet even I recognise that this work must be given time to be got "right" before we all go back online. If that takes an extra couple of days so be it. And if they ditch the "100 limit" at the same time as getting us back online then I won't need any other "bonus rewards" thanks.

 

[Davescharf]

The plan is good, it's progressing good. It should be over later tonight if all goes well.
Airport City has crossed 60 million installs sometime last year, it takes some time to do anything with a database of that size.

That’s a pretty big MS Access Database

I kid I kid...that is really impressive.

 

[Davescharf]

@PhactorPhil we had a network issue a few months ago that was affecting some of the systems my teams support. It got worse when the redundancy cutover failed.

Once it was over I asked about the root cause. Turns out a small part of the fiber line was above ground and it got severed by a gunshot. I wasn’t sure if to laugh hysterically or cry

 

[Bali Hi]

The plan is good, it's progressing good. It should be over later tonight if all goes well.
Airport City has crossed 60 million installs sometime last year, it takes some time to do anything with a database of that size.

A long time ago, @Dave suggested having a Q&A with GI developer. This was before Free2Play joined the forum, I think.
One of the questions that I wanted to ask how many players Airport City has.

Finally got that answer.
60 million is quite impressive.
For comparison, Pokemon Go, the biggest mobile game to date has 750+ million installs and 65 million active monthly users.

 

[must_dash]

Of the 60 million I wonder how many are active in the last week as opposed to have it sitting on the phone.

 

[Madge59230]

Of the 60 million I wonder how many are active in the last week as opposed to have it sitting on the phone.

And how many are called mitlain.

 

[SS1]

Of the 60 million I wonder how many are active in the last week as opposed to have it sitting on the phone.

Exactly.. 60 million "installs" is a many times multiple of "active" players.

 

[SS1]

I got a good chuckle when this thread opened with a picture of one of Googles data centers and the question of whether anyone grasped the complexity of operations...... The only thing that would be a surprise to most people is how little space is really required to run a game server cluster.

Data center computing density has gone off the hook in the past decade. Where a pop that use to take up say 10 full 42U racks can now easily be done in a single rack and provide many times the power and capability of what was being done in those 10 full racks prior. Now it's not to say that's how everyone is doing it. I'm still shocked to see in the data center old school operators with ridiculous server sprawl. If they would just spend $200k or so on new tech, they would recoup the money in about a year over what they are paying the date center to power and provide space for all of the old, big, low power Dinosaurs they run.

 

[AshPort]

A long time ago, @Dave suggested having a Q&A with GI developer. This was before Free2Play joined the forum, I think.
One of the questions that I wanted to ask how many players Airport City has.

Finally got that answer.
60 million is quite impressive.
For comparison, Pokemon Go, the biggest mobile game to date has 750+ million installs and 65 million active monthly users.

people still pay that game, nooooo

 

[Canadian Cowboy]

I just got a pop up window that this web site has exceeded its resource limit. I guess we'd better cut back on the bitching, guys. LOL