– 3 data centers and about 1,000 servers

Previous two posts were talking about Google back-end data centers and the number of servers they are running. In this post I will try to explore a totally different approach to data centers applied by (NYSE:CRM).

In a public quote by CEO from May 2010, the company announced intentions to open a new data center in Japan. The move is done to better support Japanese regulations that require some companies to store customer information inside the physical country boarders of Japan ( to Open Tokyo Data Center). In a side note, they also repeated the fact that the company is operating two US data centers, one in Silicon Valley, and the second in Virginia. And a third data center is located in Singapore to support regional customers there. That is all. Since we are talking about a SaaS giant supporting more than 70,000 customers this is strange. In contrast to Google 30+ estimated data centers this is suggesting very different approach to what is known as internet scale.

Searching for some answers I found the next difference in an article titled The Efficient Cloud from TechCrunch, the company stated (at least true to the March 2009 date) that all of platform was running on only 1,000 servers. And these servers were mirrored so really we are talking about 500 operational servers and additional 500 servers in standby mode.

What is also different between the Google approach and the approach?
Well, the servers themselves. Google is building their own propriety servers, using standard components but the cooling system is done on a container level that runs 1,000 servers and many of the traditional components that you will see in every sever were removed.

So what does does?
Apparently, they started with heavy duty Sun machines, but over the last 3 years have transition to a configuration of strong x86 machines. Surprisingly, they are using regular machines from Dell, the Dell PowerEdge R905 with AMD Opteron processors (see video describing the configuration).

Dell PowerEdge R905Dell PowerEdge R905

Looking at the Dell site for more information on the servers reveal just standard things, technical specifications include:

  • AMD OpteronTM Processors Options with up to Six-Core AMD OpteronTM Processors.
  • Chipset from Broadcom HT-2100 and HT-1100 server
  • Memory up to 256GB2 (32 DIMM slots)

Clearly very different approach but we have to remember that Google and SalesForce core businesses is very different with different operational characteristics and so different architecture makes sense. One example of the different needs is scalability. is serving about 2 million users while Google is serving hundreds of million users if not more. Will be interesting to see if will reevaluate some of their architecture decisions as the number of users and number of customers continue to dramatically increase…

To conclude this blog here is a quick table to summarize some to the differences:

Comparing Google to SalesForce.Com architecture
Comparing Google to SalesForce.Com architecture

Google data centers – how many servers do they have?

Following the previous post, I am trying to figure out the number of Google Servers out there.

Apparently it is not a simple question to answer. Google is kind of secret about this information, regarding the count of servers and data centers they are running. In fact they regard this as some kind of a competitive advantage and allow only very carefully controls glimpses into their back end operations (read more at Google Data Center FAQ). So what is the current estimate of Google servers? The Google platform from Wikipedia article includes a nice summary of multiple estimates (some of them are a little bit older) and helps in understanding current estimates as well as growth trends. Another important aspect is the number of data centers Google is operating. Apparently many of the sites are operated as separate companies contracted by Google to supply the server operations so that bit of information is also not trivial to find. An estimate from April 2008, by the Royal Pingdom titled Map of all Google data center locations, suggests Google was operating at the time 36 data centers. Even newer blogs suggest that the number of data center is even higher and stand as of summer 2010 at least at 40 different locations.
To give an idea of the world wide distribution of Google data centers, here is one map with the known Google data center locations:

Estimated Google world wide data centers (as of 2008)
Estimated Google world wide data centers (as of 2008)

Finally, I think we have what we need in order to produced our own rough estimate of this secrete number (I am sure there are many other methods):
Google official video shows a container based data center that is operational since 2005 and that runs 45,000 servers. If we can take this number as a based line and assume that older data centers have less servers whereas newer data centers have more – a range of 25,000 to 50,000 servers per data center seems reasonable. In addition, based on lists of known data center locations we have an assumption that Google has about 40 sites.
If you do the multiplication we get to a range of between 1 to 2 million servers. Adding to that an assumption that the average data center is somewhere close to the middle of the range, we can guess the magic number as of summer 2010 is in the range of 1.5 million servers!

So where is it going from here? I have started to search for different estimates for the number of Google servers over the years to try to visualize the growth trajectory. Here is what I got and it is easy to do the extrapolation and assume same growth rates how the data center will have to grow.

Google server counts
Google server counts

Google data centers – how do they do it?

Woke up this morning thinking about the magnitude of the Google backbone and asked myself how do they do that? What public information is out there describing their data center. So to answer that I Google a little bit to find what I can dig..

So let’s start with a single data center. Is there any reliable insight into how does a Google data center look like?

Google Data Center Video
This is a 6 minutes video of Google first container based data center. Not the most exciting video but it gives you a feel for what a 45,000 servers data center look like. This data center is based on container, each container is packed with 1,000 servers.

Each Google container is running 1,000 servers
Each Google container is running 1,000 servers

When saying 1,000 servers, we mean Google servers. These are built using standard components that you and me can buy in stores.

An inside view into a single container
Google server container inside view

But unlike the traditional approach to servers that provide a closed pizza box with internal cooling systems, the Google servers are completely naked, no unneeded cards, no unneeded cooling systems and no unneeded box. Instead these are open box, almost naked to the level of a motherboard. The secret is that the cooling, power management and everything else that can be done on the level of the container is done on that level. Bringing the data center into a much higher efficiency compared to standard measures. Picture below shows you how a single server look like. The picture shows an actual Google server. It is not a server that was opened to show the inside but rather this is the complete box that is installed on the container’s rack.

For more information on the Google server read the following from CNET Google uncloaks once-secret server

Google own testimony in Going green at Google suggests that using this server/container architecture Google is getting much more efficient data center, which is both good for the environment and good for Google. They kind of suggest competitors to learn from their experience and build more efficient data centers 🙂

Small efficiency on the server level but much better efficiency at the data center level
Going green at Google

The next set of questions to ask is how many data centers and how many servers does Google operates these days?

I will try to answer these questions next time…