Data Center Trends – To meet accelerating DC growth density must increase

11 Mar

Did I put enough in the title to make you think “what could this guy possibly be talking about”? I guess that was the point, and strangely enough, the title does have meaning.

There has been a religious war raging in the data center community for several years now on the subject of server cabinet densities. The question is whether high density server cabinets (typically 15-30kW) should be installed versus distributing your compute over a larger area in low density (typically 4-10kW) cabs. The basic argument is that the overhead of designing for high density, doesn’t justify the benefits. I’m here to say that high density is the best answer, but probably not for the same reasons that most have been arguing.

How Does Sustainability Fit Into This Argument?

Sustainability plays into the density argument because of the facilities.  A data center is a complex and expensive facility and building one to cool 25kW server cabinets definitely has added cost vs. the alternative. There’s increased electrical per square foot and of course extra cooling (air conditioning). However, beyond those two things, a data center is still a data center.  In fact, even the airflow management (ducting, etc.) when done effectively doesn’t need to change significantly regardless of density, it’s the ability to cycle the right amount of air to and from the correct locations that matters. The real problem with choosing low density vs. high density is it increases the space required for your servers. You also need more network cable (fiber & Ethernet), more cabinets, switches, electrical copper, ladder rack, patch panels, and, well you get the idea.

So my basic tenet is that in the long run it’s not just wasteful and more expensive to plan, build and run low density environments, its bad practice from a sustainability standpoint.  Even if you discount all the extra materials I discussed in the previous paragraph, you can’t discount the building of another structure or a larger than necessary one. If you accept the fact that the incremental design and materials required to support 25kW cabinets is only a fraction of the cost and construction material of one facility, how could you possibly justify building a whole new facility to support a 5kW design. Even if you could somehow justify the cost of building low density, you can’t justify the environmental and sustainability concerns associated with the construction and management of extra facilities.

There is roughly 1 ton of carbon created by every ton of cement and total global cement production is estimated to create 7% of all carbon output. In case you were wondering, the cement for a large data center equates to tens of thousands of tons.

So, you’re probably starting to get the picture of my sustainability point, and I haven’t even talked about the extra pavement, roofing materials, steel, cars and trucks, paint, and cleaning materials, etc., etc..

OK, How Does Cloud fit into this Story?

Simply put, IT infrastructure, applications and data benefit from having proximity. Unless you’re distributing workloads to balance geographic risks or latency concerns, the closer you can keep your servers and applications to each other the better. So if you’re building data centers that spread servers across 100 cabinets, when they could have put them in 15-25 cabinets, you’re not only wasting space and money, you’re also causing headaches for your infrastructure and application design teams.

The Switch SuperNAP is designed to 1500 Watts a square foot or roughly 25kW per cabinet at a year round average PUE of 1.24. While you don’t have to build environments that require that kind of density, when you do it provides significant cost savings and potential infrastructure performance improvements.

The original objective of building to 1500 Watts a square foot was to support the real application needs of organizations that are maximizing the use of modern infrastructure. Our customers are running hundreds of racks between 15kW and 28kW and several hundred more are running between 6 and 15kW.  Besides the benefits our customers derive from the ability to run modern infrastructure, this density has also allowed us to dramatically reduce our data center construction footprint as compared to the vast majority of other facilities. The result of these low density environments is that we are potentially using up to five times (5X) the amount of space necessary.

To put the previous paragraph into perspective I estimate the total global data center foot print exceeds 700 million square feet and we are adding roughly 10% in new capacity every year.  When you consider that up to 18% of a buildings carbon output is created during the manufacture of materials and the construction phase, that’s an immense figure.

So the next time you hear someone talk about their PUE number, ask them two things;

–          How is your power generated (Fossil, renewables, Clean, etc.)?

  • The energy source is key to determining the actual carbon impact of DC operations, vs. simply measuring the effective use of the power (I.e., PUE)

–          What’s your power density per square foot?

When you can make a decision that helps your company’s bottom line and reduces negative impact on the environment, why wouldn’t you?


The links below are a few additional resources used in the writing of this blog:

  1. Richard D

    March 13, 2012 at 9:00 am

    Not only did you put enough in there to capture a wide audience, you also lay out how the infrastructure supply chain starts at power generation and extends to, mostly what we can control, the point from where end-users interact with applications – the datacenters. (was thinking about networks and extending out to actual end users – just not sure how to capture that)

    It reminds me of the initial hype surrounding the Prius and its increased gas mileage, which on the surface sounded great, until you looked at the entire lifecycle from manufacturing to disposal of wasted batteries. When one looked at “everything” the gains were far more modest if not negligible.

    The industry is pushing hard to get more efficient and it shows in our collective ability to have driven down PUE, domestically, avgs from the original EPA report back from ’07 time frame (I think it was authored not long after Koomey report) which had industry at 2.2 or it is down to 1.8 – a near 20% gain in PUE inefficiencies in a few years – has the auto industry changed that much in last few decades???

    I wouldn’t be surprised, given how much power is used by large infrastructure platforms, if we see innovation driven by this same group in power generation over next decade (see google’s floating datacenter concept for starters!)

    • mthiele

      March 13, 2012 at 5:04 pm

      Richard, thanks for the great comment and I think you’re on to something. This will be a great topic for debate, & discussion.

  2. 7 Green Data Centers Just in Time for Spring « Wikibon Blog

    March 23, 2012 at 6:57 am

    […] is to use water or cool air. Instead of building massive low-density data centers, switching to compact high-density centers makes energy cost and facility construction far more affordable. Not to mention governments around […]

    • mthiele

      March 24, 2012 at 5:48 pm

      We obviously agree that high density is the smarter way to go. However, the use of air to cool effectively at densities above 12kW is the issue. Most data centers in use today would have to completely remove their cooling infrastructure (ducting, HVAC units, etc) in order to replace it with a design that could cool a high density rack (20+ kW), like the Switch SuperNAP. Water can be used, but again, it takes significant upfront planning and is generally only an option for a new facility.

  3. Data Center: Who Knew Saving Cost & the Planet Could be so Easy « SwitchScribe

    September 1, 2015 at 10:19 am

    […] Related Sustainability Blog: To meet accelerating DC growth server cabinet density must increase […]