Morning at Apple store (or “No SIM” sign of death)

Today I would like to share something different: My first real service call regarding my iPhone 4 and the help I received at the Apple store. The story begins like every good story with a happy moment (and also has a not too bad ending).

First thing first, the opening scene. Remember late June this year? The first day of iPhone 4 and the long lines in the Apple Store and stores running out of devices?

Well, I was not there. I was not standing in long lines and I did not get my first iPhone on that very first day. I waited…and waited…and waited for whole full week. And than I went to an Apple Store near my home and just walked and picked-up a brand new iPhone 4. No line. No hassle. Just the way I like it…Happy I came home and started to experience phone like I never did before. I am an old time BlackBerry Curve user but could not really use it for more than email and phone calls. With the iPhone I started to surf the web (well only for sites without Adobe things), and started to explore the ocean of available applications (and again, assuming they are approved by Apple) and games (same). It is the first time I really enjoying mobile devices for more than talk and email but for pleasure. Really, really fun!

And now for scene two. The not so fun: My first issue was when trying to use the phone as a phone. Apparently I am one of those people that do not know how to hold the iPhone correctly while talking. I got the disconnects all the time…but $30 fixed that. I went back to the Apple store and bought the bumper on my own expense. Later I got a notification about the refund so that was ok, I guess.

And here we get to the real issue. Issue number two. About a week ago I picked up the phone and there was no reception. Or at least that is what I thought. Looking closer I saw a message saying “No SIM”. I thought to myself – I know that there is a SIM card somewhere but were is it?? I had no idea and my phone was basically dead.

iPhone 4 "No SIM" sign
iPhone 4 "No SIM" sign

I went back to the Apple Store. This time the kid at the shop looked at the phone and said that this is a well known issue (good to know). He pulled some sort of secret key, inserted it to the side of the phone and voilà SIM card appears out of nowhere. He took it out, reset the iPhone, clean the SIM and inserted it back. He hands me the iPhone and said that the device is now working. I looked at it and indeed reception is back. However I am in the computer business and know our long time support tricks, “do a reset and it will be ok”. Our support guys are just buying time and hoping that next time someone else will have to take care of the issue.

I asked the guy what to do if the problem will appear again and he said that in that case I should setup a meeting with a “Genius” and he will surely solve any problem.

I went home and 30 minutes later the problem showed up again. This time I googled the issue and fair enough. Multiple people reported that problem. Many of them went to Apple store and got the same story. Some of them suggest to “do it yourself” and patch the SIM tray (see this article Fixing iPhone 4 “No-Sim” problem). However, some of them got the machine replaced.

"Do it yourself" fix for iPhone 4 No-SIM
"Do it yourself" fix for iPhone 4 No-SIM

With no other alternative I setup a meeting with the “Genius” for the following day and went back to the Apple store. Starting to get annoyed. The “Genius” was polite but started to do the same things… Pick the secret key, take the SIM out and all the rest. I stopped him and told him that we did all of this already and that it does not work. Also, I told him that I googled the issue and others reported that it has to do with a defected iPhone and that they need to replace it. The guy seemed embarrassed and admitted that this is indeed a hardware issue and that the only way to “fix” it is to replace the machine. And that is what he did! He gave me a new iPhone and inserted my old SIM.

Last part, the not so bad ending. I went home. Synced the iPhone with my iTune. iTune asked whether I this is a replacement and should it re-load all the content (including the contact list). I approved ad everything is working since. No real complain about the “Genius” work and about the support, but…

What I have is an unease feeling that they know something I do not. For me Apple seems like this “DR. Jekyll and MR. Hyde”. They build the most amazing products. Fun and stylish and cool and best. We are all too used for Apple to come every few years with yet another life changing device. However, they seems to have this other side. This nasty conglomerate culture. For example, I have no idea if the “No SIM” issue is something that happened in few devices or millions. Same as I did not know if the iPhone disconnects are my iPhone issue or is it a global issue until it was publicly reported by multiple media groups and forced Apple to semi-admit that there is an issue.

Is it just me or is it that Apple became just a little bit too all mighty? – 3 data centers and about 1,000 servers

Previous two posts were talking about Google back-end data centers and the number of servers they are running. In this post I will try to explore a totally different approach to data centers applied by (NYSE:CRM).

In a public quote by CEO from May 2010, the company announced intentions to open a new data center in Japan. The move is done to better support Japanese regulations that require some companies to store customer information inside the physical country boarders of Japan ( to Open Tokyo Data Center). In a side note, they also repeated the fact that the company is operating two US data centers, one in Silicon Valley, and the second in Virginia. And a third data center is located in Singapore to support regional customers there. That is all. Since we are talking about a SaaS giant supporting more than 70,000 customers this is strange. In contrast to Google 30+ estimated data centers this is suggesting very different approach to what is known as internet scale.

Searching for some answers I found the next difference in an article titled The Efficient Cloud from TechCrunch, the company stated (at least true to the March 2009 date) that all of platform was running on only 1,000 servers. And these servers were mirrored so really we are talking about 500 operational servers and additional 500 servers in standby mode.

What is also different between the Google approach and the approach?
Well, the servers themselves. Google is building their own propriety servers, using standard components but the cooling system is done on a container level that runs 1,000 servers and many of the traditional components that you will see in every sever were removed.

So what does does?
Apparently, they started with heavy duty Sun machines, but over the last 3 years have transition to a configuration of strong x86 machines. Surprisingly, they are using regular machines from Dell, the Dell PowerEdge R905 with AMD Opteron processors (see video describing the configuration).

Dell PowerEdge R905Dell PowerEdge R905

Looking at the Dell site for more information on the servers reveal just standard things, technical specifications include:

  • AMD OpteronTM Processors Options with up to Six-Core AMD OpteronTM Processors.
  • Chipset from Broadcom HT-2100 and HT-1100 server
  • Memory up to 256GB2 (32 DIMM slots)

Clearly very different approach but we have to remember that Google and SalesForce core businesses is very different with different operational characteristics and so different architecture makes sense. One example of the different needs is scalability. is serving about 2 million users while Google is serving hundreds of million users if not more. Will be interesting to see if will reevaluate some of their architecture decisions as the number of users and number of customers continue to dramatically increase…

To conclude this blog here is a quick table to summarize some to the differences:

Comparing Google to SalesForce.Com architecture
Comparing Google to SalesForce.Com architecture

Google data centers – how many servers do they have?

Following the previous post, I am trying to figure out the number of Google Servers out there.

Apparently it is not a simple question to answer. Google is kind of secret about this information, regarding the count of servers and data centers they are running. In fact they regard this as some kind of a competitive advantage and allow only very carefully controls glimpses into their back end operations (read more at Google Data Center FAQ). So what is the current estimate of Google servers? The Google platform from Wikipedia article includes a nice summary of multiple estimates (some of them are a little bit older) and helps in understanding current estimates as well as growth trends. Another important aspect is the number of data centers Google is operating. Apparently many of the sites are operated as separate companies contracted by Google to supply the server operations so that bit of information is also not trivial to find. An estimate from April 2008, by the Royal Pingdom titled Map of all Google data center locations, suggests Google was operating at the time 36 data centers. Even newer blogs suggest that the number of data center is even higher and stand as of summer 2010 at least at 40 different locations.
To give an idea of the world wide distribution of Google data centers, here is one map with the known Google data center locations:

Estimated Google world wide data centers (as of 2008)
Estimated Google world wide data centers (as of 2008)

Finally, I think we have what we need in order to produced our own rough estimate of this secrete number (I am sure there are many other methods):
Google official video shows a container based data center that is operational since 2005 and that runs 45,000 servers. If we can take this number as a based line and assume that older data centers have less servers whereas newer data centers have more – a range of 25,000 to 50,000 servers per data center seems reasonable. In addition, based on lists of known data center locations we have an assumption that Google has about 40 sites.
If you do the multiplication we get to a range of between 1 to 2 million servers. Adding to that an assumption that the average data center is somewhere close to the middle of the range, we can guess the magic number as of summer 2010 is in the range of 1.5 million servers!

So where is it going from here? I have started to search for different estimates for the number of Google servers over the years to try to visualize the growth trajectory. Here is what I got and it is easy to do the extrapolation and assume same growth rates how the data center will have to grow.

Google server counts
Google server counts

Google data centers – how do they do it?

Woke up this morning thinking about the magnitude of the Google backbone and asked myself how do they do that? What public information is out there describing their data center. So to answer that I Google a little bit to find what I can dig..

So let’s start with a single data center. Is there any reliable insight into how does a Google data center look like?

Google Data Center Video
This is a 6 minutes video of Google first container based data center. Not the most exciting video but it gives you a feel for what a 45,000 servers data center look like. This data center is based on container, each container is packed with 1,000 servers.

Each Google container is running 1,000 servers
Each Google container is running 1,000 servers

When saying 1,000 servers, we mean Google servers. These are built using standard components that you and me can buy in stores.

An inside view into a single container
Google server container inside view

But unlike the traditional approach to servers that provide a closed pizza box with internal cooling systems, the Google servers are completely naked, no unneeded cards, no unneeded cooling systems and no unneeded box. Instead these are open box, almost naked to the level of a motherboard. The secret is that the cooling, power management and everything else that can be done on the level of the container is done on that level. Bringing the data center into a much higher efficiency compared to standard measures. Picture below shows you how a single server look like. The picture shows an actual Google server. It is not a server that was opened to show the inside but rather this is the complete box that is installed on the container’s rack.

For more information on the Google server read the following from CNET Google uncloaks once-secret server

Google own testimony in Going green at Google suggests that using this server/container architecture Google is getting much more efficient data center, which is both good for the environment and good for Google. They kind of suggest competitors to learn from their experience and build more efficient data centers 🙂

Small efficiency on the server level but much better efficiency at the data center level
Going green at Google

The next set of questions to ask is how many data centers and how many servers does Google operates these days?

I will try to answer these questions next time…