'Power Capping' the Datacenter

snydeq writes "Datacenter operators seeking increased server density may soon turn to power capping, an emerging technology that limits the amount of electricity a server can consume, InfoWorld reports. The practice, which can be applied at the rack level, ensures that no server draws above a set power level, thereby increasing datacenter capacity within a rack-level power envelope by as much as 20 percent, according to a proof-of-concept study at Baidu, China's largest search company. As with powering down servers during off hours, of course, power capping incurs calculated risk, as those in charge of business-critical applications may be reluctant to set power limits below maximum utilization. Yet given IT's need to contend with the permanent energy crisis, the notion of power capping the datacenter could prove advantageous."

Comments

pointless

[MagicMerlin]

lame...a much better way of handling this is what datacenters are already doing: simply sell you power circuits, say 20 amps each for a set price. if they want to discourage power use, they simply have to raise the price. low tech, and works perfectly well. you can always get a good power strip with a ammeter on it if you want to know what your servers are drawing.

Re:pointless

[Midnight Thunder]

lame...a much better way of handling this is what datacenters are already doing: simply sell you power circuits, say 20 amps each for a set price. if they want to discourage power use, they simply have to raise the price. low tech, and works perfectly well. you can always get a good power strip with a ammeter on it if you want to know what your servers are drawing.

This is certainly an alternative. I think that either way we should see owners of devices examining how much processing power they really need and this may translate into computer companies selling hardware that is either more power efficient or smarter in the way it uses power. For example I can imagine smarter control on the which sub systems or cores are active based on a given server load.

With SSDs and the reduced issue of plater spin up, I can imagine servers that stand-by and wake up if there is any activity on a specified port. I am not thinking WoL, but hardware with a smart network card that wakes up the computer as necessary. At one point I was thinking of an embedded web server (something like http://www.webservusb.com/) that would respond to data and then wake up the main server when it had anything that it could not handle.

On the Linux side of things, what is available for achieving this sort of thing?

Re:pointless

[Wesley Felter]

simply sell you power circuits, say 20 amps each for a set price.

This is a great way to waste money. If you don't use power capping, then you'll be paying for 20A but only using 10A-12A if you're lucky. Power capping allows you to use the full 16A that you're paying for.

Re:pointless

[QuoteMstr]

Power capping allows you to use the full 16A that you're paying for.

Alternatively, a power meter will let you pay for the full 16A you use. Networked power meters are certainly less complex than "power capping". Why not just use those?

Re:pointless

[derGoldstein]

With SSDs and the reduced issue of plater spin up, I can imagine servers that stand-by and wake up if there is any activity on a specified port

SSDs don't actually consume much less power than platter-based drives, but they're far more expensive. The spin-up/down is mitigated if there's good caching on the controller, and I think that's the key here -- caching. You can have a couple of terabytes on a server, but the drives aren't called until something is requested which *isn't* on the controller's buffer.

I should point out though that power consumption of HDs are insignificant compared to the rest of the system. If you want to avoid the spin-up issue, just never spin them down. An idle, spinning HD normally won't consume much over 1W -- that's one *single* watt. How does that compare to the CPU power draw?

Here's that article on Tom's Hardware where they revised their SSD power consumption tests, though it's a year old now.

Re:pointless

[Jurily]

Power consumption is not just about hardware. Maybe people need to cut down on scripted languages and the like. Return to server-client architecture instead of AJAX and you're halfway there.

Re:pointless

[CrackerJackz]

The real 'cool factor' starts to kick in when this stuff is fully automated, the demos that I've seen you can take a rack of dev/test servers and drop the power they are using off hours, then give the power back (cpu clock rate, etc) during the day. We already have something simmilar in our company datacenters: the HP systems (running ESX) we have now ballance (via vMotion) running systems at night, and power down some of the hosts (about half of them on a normal night, it bases it on current load) saving money and power ... no impact to the end user. Its a win-win as far as I'm concerned.

Re:pointless

[QuoteMstr]

I still don't understand why the data center needs to get involved. You can just manage your own power, and using a conventional power meter, the data center can just collect the bill for what you use.

Re:pointless

[tepples]

The spin-up/down is mitigated if there's good caching on the controller, and I think that's the key here -- caching. You can have a couple of terabytes on a server, but the drives aren't called until something is requested which *isn't* on the controller's buffer.

Something not on the controller's buffer would include new data received from the network. Caching works fine for reads, but if you rely on it for writes, you could lose Durability.

How long until..

[Rayeth]

One only wonders how long it will be until every spreadsheet process becomes "business critical" to override restrictions such as this.

Re:How long until..

[mcrbids]

One only wonders how long it will be until every spreadsheet process becomes "business critical" to override restrictions such as this.

When the business involved has to pay a larger and larger bill, that which is considered "critical" is increasingly analyzed as the bottom line gets thinner and thinner. When margins are fat, daily $4 lattes are "critical" to staff morale. Conversely, when times are really tight, staff morale is critical at "well, you ain't fired yet, is yeh?".

Re:How long until..

[benjamindees]

Since the bills are paid by accounting, I'd say approximately one month.

Re:How long until..

[Rich0]

I think the problem is that too often the people who run the servers and the people who benefit from the servers have entirely different budgets and suffer from terrible myopia.

If some IT budget administrators were put in charge of telecom they'd save money by not letting anybody have a phone. Sure, that saves money, but of course completely ignores why employees have phones in the first place.

I can't tell you how many times at work I've seen applications suffering with performance where the application is designed to load balance across an almost arbitrary number of servers but some bureaucrat wants $100k worth of studies done to justify a $10k expenditure (if you're generous and include all the provisioning costs in addition to the hardware). In the meantime the business might be losing $10k per week in productivity easily.

In business everything has a cost, and everything has a return. The problem is in how you measure each. When somebody has an incentive to reduce cost at any cost, you get foolish decisions. The reverse is also true. Ditto for crazy metrics that encourage people to spend more money tracking expenses than on the expenses themselves.

Oversubscription, virtualization, and this latest technology are all great tools for maximizing ROI. However, when misused they reduce the R and damage the business.

Not a crisis

[tsotha]

"Permanent energy crisis"? There's no such thing as a permanent crisis. Yes, energy costs are going up because we're more sensitive to the impact of new capacity. But that hardly constitutes a crisis. The word "crisis" has been practically stripped of meaning - everything these days is a goddamn crisis. When the girlfriend you were about to dump gets pregnant - that's a crisis. A few bucks more on your energy bill - not a crisis.

Re:Not a crisis

[Threni]

Google for "Peak Oil". I think you'll find that we'll all be in very serious trouble very soon.

Re:Not a crisis

[CannonballHead]

Or you'll find that there's no such thing. It depends which link you click.

Re:Not a crisis

[tsotha]

Peak oil isn't a crisis either. We can replace every erg of energy from oil with nuclear if we're motivated to do so. Even if we don't, during my lifetime peak oil will mean as an American I might, at some point, have to pay almost as much for gas as the Europeans are paying now, plus a few commodities will cost more.

Re:Not a crisis

[HTH NE1]

They meant "opportunity".

Re:Not a crisis

[Anonymous Coward]

I think you need to understand a little more about the politics and economics of oil before making a statement like this. A couple things to get you started:
1) Determine the relationship between the US dollar and economy and global oil trade
2) Understand why EROEI is so significant when discussing alternative energies

Re:Not a crisis

[tsotha]

Sigh. I understand all that, but the numbers are hardly insurmountable, or even very uncomfortable. How many nuclear power plants could we have built for the trillion dollars we spent on "stimulus"? Four or five in every state, by my calculations. The idea that everything is just going to fall apart when the price of oil goes up is just silly.

Re:Not a crisis

[Anonymous Coward]

If it were that simple the US would be building nuclear plants instead of waging war in the middle east.

The simple fact is the US's dominance came about because it had both the supply (Texas) and the demand (Cars). That position was leveraged during and after WW2 to put in place the rules of production and global capitalism. He who ruled the oil, ruled the world.
Oil's significance is much, much larger than simply an energy supply for vehicles. As an example, every wholesale crude is bought it is paid for in greenbacks. Those greenbacks are primarily acquired through the purchase of t-bills, etc. So, with every barrel of oil purchased the US is essentially give a nice big loan which has fed the credit economy. That's why the powers that be get jittery every time a country mentions trading oil in the Euro or other currency and, oddly enough, bad things happen to said country.
There may be technical solutions to the problem in the form of alternative energy, but the house of cards that is global capitalism was built on, and depends upon, cheap oil.
I'm not saying you're technically wrong in your assertions, I'm just saying there's a hell of a lot more to factor into the peak oil discussion.

Re:Not a crisis

[rubycodez]

nothing to understand, we have coal sufficient for centuries and the means to make it into every major type of hydrocarbon fuel. No crisis. No peak oil.

Re:Not a crisis

[Anonymous Coward]

It's called Google for Electricity Production. Oil in the US is not used for electricty production which what servers use.

Here's a nice stat from Wikipedia on electricity production in the US.

Re:Not a crisis

[Anonymous Coward]

Hell, you're right technically. Technically. But not politically and economically.
The US may come up with an alternative way to control the world economy and remain the big engine that could. In the meantime, though, the dependence on oil isn't just as a source of energy. It's how the US funds the credit economy, by enforcing the purchase of oil in greenbacks. All those countries aren't lending the US money just because the consumers want it, they're lending it to purchase oil (and other commodities, but if oil goes away it's a free for all anyhow).
So sort out the transportation system and switch to coal. Problem is, the country will bankrupt quicker than a dot-com back in the day. Perhaps someone clever will figure this conundrum out without having to turn into a desperate empire raiding the resources of smaller countries. One can only hope.

Re:Not a crisis

[FooAtWFU]

The Stone Age didn't end because they ran out of stone. The Oil Age might end when we start to run low on oil, but that doesn't mean we won't have plenty of alternatives.

The economic argument for all sorts of magic coming from having oil traded in USD is weak. A barrel of oil is worth whatever the next buyer things a barrel of oil is worth, a dollar is worth whatever the next guy who gets it thinks it's worth. These things are both fungible, they're both pretty liquid. There's a vibrant currency exchange market. If people think the dollar or the barrel-o-crude is not worth what it used to be, the prices are perfectly capable of shifting to match. Look at the last big recession and oil crisis of the 1980s. Look at 2008, for crying out loud. The US dollar may wax and wane, the US economy may shrink 10% in a bad year, but oil dropped from over $100 a barrel to something like $30.

As for the money supply, the Federal Reserve is pretty capable of generating as much or as little of our little fiat currency as they feel like. The national debt (and the price at which people are willing to buy it worldwide) is what's going to be weighing on the US and its economy over the next several decades, much more than any medium-of-exchange games. The government and the private sector compete for loans: when there's more debt, it's more expensive for private firms to borrow and that hurts economic growth - because look! Treasury bonds! They're nice and safe. Why would you invest in a risky old Business in /this/ economy?

Re:Not a crisis

[Anonymous Coward]

Well, ok, then, go ahead and panic then. The rest of us will be busy not preparing for our slightly lower standard of living when we have to switch to energy sources that aren't as easy to get and use as "stick a straw in the ground and burn it"

Nuclear isn't "alternative energy." It's "energy." 80% of France's electricity is nuclear. France. A country bigger than Texas. There is no physical reason why we can't do the same except Politics. And panicking about politics is a surefire way to find your way into things like ceding control of your health care future to people who you can't trust to drive over a tide pool without killing someone.

Re:Not a crisis

[derGoldstein]

nothing to understand, we have coal sufficient for centuries and the means to make it into every major type of hydrocarbon fuel.

It's true that there's enough coal for centuries. However, if we keep using it the way we do now, these centuries will be spent underwater. The power crisis will kick in when sea levels start rising so fast that power will be capped during peak ours, for *everyone*. Either we start adapting now, or we'll pay for it tenfold later.

Re:Not a crisis

[benjamindees]

How could you not believe there's an energy crisis? The average American uses the energy equivalent of 60+ personal slaves. 90% of that is provided by limited fossil fuels that are having an irreversible impact on global climate.

We have zero viable plans to replace any of it any time soon.

And in case you haven't been paying attention to the current breeding-age generation, a pregnant girlfriend is a ticket to 20 years of government cheese. It's not a crisis at all.

Re:Not a crisis

[tsotha]

I must admit I'm not sure how to convert units of "personal slaves" into kilowatt-hours. I must have gotten my degree too early, or, I guess, too late. But assuming I understand the gist of your comment, and further assuming scientists have accurately assessed the impact of fossil fuels on the earth's climate (which would exclude coal as a long-term power source), I don't see why nuclear power can't produce all the power we'll need for hundreds of years. And that's assuming we can never get reasonable efficiencies out of the various solar power schemes (PV, biofuel, concentrators, space-based), which is a dubious assumption.

Of course we'll have to start reprocessing spent fuel, and we'll need switch to thorium eventually, but you and I won't outlive the uranium supply.

Re:Not a crisis

[Anonymous Coward]

Totally off-topic, but why is your entire post italicized? Italics in modern forums generally means you're quoting someone, which you aren't. Also, why start off something with "sigh." seems pretty condescending, imo.

Re:Not a crisis

[Sique]

It's not that easy. The US FED is the only legitimate source of USD. So whenever someone needs USD to buy crude oil, he has somehow to get USD, and that means that he either takes a loan with the US FED, or he tries to sell something to someone, who has already USD, which means, that in the end he has to do business with the U.S. directly or indirectly.
Because the US FED has a monopoly of the USD, it is the only institution that can directly manipulate the price of the USD. So no, the price of the USD is not that of a commodity.

Re:Not a crisis

[Threni]

We don't have coal of the current quality, with the current ease of access for centuries with current trends in consumption and population growth.

This is quite an interesting, and worrying, read:
http://www.chrismartenson.com/crashcourse

Re:Not a crisis

[OhPlz]

Sixty slaves? Sounds like you've found a solution to our nation's unemployment crisis! I bet using people as energy sources would also help correct our obesity crisis, which would in turn lessen the effects of our current health care crisis. Returning people to work would help the financial crisis. These laborers wouldn't need much schooling, so there goes the education crisis. My goodness, it's brilliant!

Of course all these crisis could be the equivalent of the Y2k crisis.. much ado about nothing.

Re:Not a crisis

[greed]

In Ontario, we're having an electricity surplus. With a manufacturing "slow-down" and a cool summer, the grid is peaking about 7 gigawatts below typical load for this time of year.

Which means it's running almost entirely on nuclear and hydro, which are effectively "free" by comparison to coal, oil and gas.

So wholesale customers are getting rates below 1 cent per kWh. Large industrial customers may even be getting rebates to use power.

At least one ~700 MW reactor has been shut down to reduce base generating capacity.

What crisis is that again?

Re:Not a crisis

[tlhIngan]

Peak oil isn't a crisis either. We can replace every erg of energy from oil with nuclear if we're motivated to do so. Even if we don't, during my lifetime peak oil will mean as an American I might, at some point, have to pay almost as much for gas as the Europeans are paying now, plus a few commodities will cost more.

I think peak oil is real, but we won't get any warning. It'll just happen that we run out of oil.

Think about it for a little while - we've already seen that when gas prices jump, people change their habits, and as more supplies come online, gas prices fall, and people revert back to their old habits, somewhat (not everyone who adapted reverts).

It's the same with alternative energy sources.

It's a very carefully balanced supply-demand curve. When gas prices rise, people change habits, but not many. Likewise, money gets spent in alternative energy research. Oil-rich countries see the opportunity to make money, and pump out more oil (and go on expeditions finding more oil), which has the effect of lowering gas prices again, and alternative energy research stalls because it can't compete with cheap gas. The oil producing countries start cutting output (e.g., oilsands) because they margin isn't there because now there's more oil on the market and lower prices.

The end result is oil prices will stay within a certain range - in fact, I'd guess we don't know if we've hit peak oil yet (mostly because we don't know where all the resources are, the only deal is that it gets harder and harder to extract), and probably won't. We'll draw more oil out when prices are high, and draw less when prices are low. No, the only real warning would be when prices jump, then people discover that there really is no more oil, and prices jump again, people discover even difficult to extract sources are all tapped out, and prices will skyrocket. All this will probably take place over a few years at most, far from a gradual rise of prices as traditional peak oil theory holds.

There may be signs that these cycles are starting to fall apart, but I'm guessing they're too subtle to actually generate any useful warnings. It'll be one of those "we'll know it when we hit it". Part of it comes from those who adapt and don't revert, lowering consumption again which puts prices back to where they were prior to the spike.

Re:Not a crisis

[Anonymous Coward]

Peak oil is not when we run out of oil, it is when we discover new oil at a pace slower than we use it. However, we're talking about proven reserves, which depends on how much you are willing to spend to get the oil out of the ground. At 30$ per barrel we have already passed peak oil. At $130 a barrel, it probably won't come for a few decades, much farther into the future if conversion of coal to oil can be included at the right price.

Re:Not a crisis

[rubycodez]

uh no, future won't be spent underwater, despite whatever Al Gore's sensationalist bullshit movie said.

by the way, the ocean levels have been rising since the last ice age. Man might be making them rise a wee bit faster, so really all these place you see on new evacuating because of higher ocean level are doing so because they were right at ocean level. Might as well move now since they would have had to move later anyway.

Stupid

[EmperorOfCanada]

This might be the stupidest thing I have heard in a while. If my servers are eating a bunch of power then they are doing a bunch of work. If some bozo green policy turns my server down during a peak time then my business would suffer. I would then move my servers to another provider beginning the next day. Pricing my server's power might be one way get me to buy more efficient servers but that is as close some green crap that I would accept.

Re:Stupid

[Wesley Felter]

You should set the power cap slightly higher than the server's typical power usage, so the server rarely or never slows down. Also, in corporate IT there is no other provider, so the alternative to power capping is usually to not buy any more servers.

Just a friendlier name for...

[clarkcox3]

Ins't this just putting a friendlier name on "overselling"?

• We can pack in 20% more boxes because we don't really have the power to meet demand should all of them spike at once, but that doesn't usually happen.
• We can sell in 20% more airline seats because we don't really have the room to meet demand should all of the customers actually show up, but that doesn't usually happen.
• We can claim unlimited bandwidth, because we don't really have the capacity to meat demand should all of our customers actually download 24/7, but that doesn't usually happen.
• etc. etc. etc.

Re:Just a friendlier name for...

[Wesley Felter]

Yes, power capping allows safe oversubscription of power. Honestly, these days if a business isn't overselling they're leaving money on the table.

Re:Just a friendlier name for...

[frosty_tsm]

It depends what they are overselling and to what degree. Overselling a plane screws whoever is left behind (large impact over small set of customers). Overselling bandwidth slows someone's download or game (small impact over many customers).

Overselling a rack and causing servers to a) fail or b) corrupt data costs in hundreds of thousands or more pretty quickly in damages and legal fees. It's a much wiser business decision to just increase the power capacity (not that some suits will think this is the greatest thing since cloud computing).

Re:Just a friendlier name for...

[Wesley Felter]

The point of power capping is that it doesn't cause any kind of failure; servers just slow down when they reach the cap.

Re:Just a friendlier name for...

[EdIII]

Honestly, these days if a business isn't overselling they're leaving money on the table.

Yes, and it is that kind of amoral and unethical crap that is giving the shaft to the consumer day after day after day.

Just because you can do something, and it will be profitable, does not mean that you should do it.

Overselling anything should be expressly prohibited by law in the strongest terms possible. Honestly, it really pisses me off. At the very least, the amount of overselling should be disclosed to customers with agreed upon limits. I should be INFORMED.

If I am paying your company X amount of dollars and you don't deliver your Y amount of product, I will vehemently seek all remedies available. You can bet your ass on that. It's an unacceptable risk. Additionally, you are not actually delivering Y amount of product at all. What you are really stating is that there will be a Z amount of probability that I will receive Y amount of product for X amount of dollars. Well, contracts don't usually reflect that do they? Not any I have seen.

Overselling is a dirty secret that is kept in the shadows for good reason.

Now if I KNOW that you are overselling X amount then I can at least make an informed decision as to whether or not I want to risk using your services. This is at least well known with residential ISP's. Even the most unsophisticated consumer knows they will never see the number they are being sold. They just accept it. Well, businesses do not. If I have 50 Mb/s promised to a full rack and I can't achieve that amount with testing and use, I will complain loudly until I do.

Yes, power capping allows safe oversubscription of power.

I don't see how it allows oversubscription at all. Most full racks have several 20 amps circuits that are delivered.... 20 amps. Rule of thumb says you limit your usage to 15 amps at any one time to reduce the risk of server usage overloading the circuit and causing failure.

What this technology allows is for you to use the entire 20 amps more efficiently. I can put more servers in that cabinet since I can be assured that all of the servers together will not exceed 20 AMPS. Assuming that the technology is really that good and can control the power usage of every component, hard drives included.

I don't see how this allows the data center to deliver me less then 20 amps to my circuits. If they did that and caused my equipment to go down .... it would get really ugly. Really ugly. Really frickin UGLY. I have seen and heard about just those situations where power was not delivered and redundancy failed. Meetings and conferences and rather large compensations from the data center to the customer.

It is far more logical that this would appeal to the data center since it would simply allow more servers to be populated in a rack and more efficient usage of available power, not that the data center is going to deliver less than what they are promising.

Re:Just a friendlier name for...

[Anonymous Coward]

Yes, and it is that kind of amoral and unethical crap that is giving the shaft to the consumer day after day after day.

You're right! Gyms, for example, shouldn't sell memberships to more people than they can fit in the gym at the same time. Otherwise, if every one of their members decided to go to the gym at once, they'd have to turn some away!

(Seriously - overselling is just another risk. Taking no risks is bad. Taking too much risk is bad.)

Re:Just a friendlier name for...

[bky1701]

Overselling is you forcing your customer to take a risk, not you taking a risk. Your only risk is indirect.

Power management

[westlake]

Tell me why it doesn't make sense to buy power at off-peak rates and store it locally to meet peak demands.

Re:Power management

[timmarhy]

how exactly are you planning to store such a massive amount of power and avoid losing energy in the transfer?

Re:Power management

[Seth024]

Storing power is not as easy as it looks. If you have a good idea let us know. Huge batteries are expensive and lose their charge over time. You could use the power to pump water higher and then use that potential energy later. There are so many convertions of energy that make it not effective enough.

Re:Power management

[whoever57]

Tell me why it doesn't make sense to buy power at off-peak rates and store it locally to meet peak demands.

This has been possible for consumers in the UK for decades. In the UK, they have "storage heaters". These are heaters that store heat at night, at off-peak rates (the meters measure peak-time and off-peak usage) and then release the heat during the day when it is needed. Many people also use delay-timers on other devices such as dishwashers to take advantage of off-peak rates. It amazes me that this concept has failed to reach the USA, although it is a lot more difficult to store "cool" than it is to store heat, so perhaps the possible gains are smaller across much of the USA.

Re:Power management

[HTH NE1]

how exactly are you planning to store such a massive amount of power and avoid losing energy in the transfer?

I was going to make that point, so instead I'll make the counterpoint: the value of the energy lost would have to be within the peak/off-peak cost differential.

Re:Power management

[brusk]

Or more precisely, the differential PLUS the cost of the storage system (which includes buying more land, having more staff, more insurance, etc.).

Re:Power management

[brusk]

It amazes me that this concept has failed to reach the USA, although it is a lot more difficult to store "cool" than it is to store heat, so perhaps the possible gains are smaller across much of the USA.

The problem isn't with household usage patterns but with the fact that the utilities generally don't have a time-tiered pricing system. My electricity meter is of the old-fashioned dumb sort, as is most people's, so there's no incentive to use power at off-peak times. That's changing, but slowly. If smart meters were ubiquitous, I think people would respond (and there are many places in the US that need more heating than the UK!).

Re:Power management

[miller60]

It does make sense. That's exactly what i/o Data Centers is doing in Phoenix. They're installing a thermal storage system at their huge Phoenix ONE data center. The building chillers will cool a solution of water and 28 percent glycol. The thermal storage tank contains Cryogel ice balls, which freeze when the system is charging at night, and then cool the glycol solution during the day. The glycol solution is then pumped through a heat exchanger, which chills water in a separate loop used in the data center.

Re:Power management

[DerekLyons]

Because there doesn't currently exist a cheap method of storing huge amounts of electric power to release at demand.

Re:Power management

[SBrach]

This is done in other data centers in phoenix also.

is this news?

[Anonymous Coward]

this has been done for quite awhile by several providers -- is this really news to anyone? SAVVIS has been limiting by cage for at least a year in the chicago area.

Not saving energy

[russotto]

If I'm reading the article right, this doesn't save any energy at all (and might increase total energy consumption). It's a way of spreading out power use over time, so you can get more servers without increasing peak power capacity. It makes sense for some loads (why run at 2Ghz for 2 minutes of every 30 when you can save power by running at 500Mhz for 8 minutes). But for interactive loads it won't be good.

Our Policy

[teknopurge]

I can't see any legitimate provider capping power usage. We have 20A running to a client rack by default - if they need more circuits we charge them per circuit. The only place I can see people wanting to make an argument for capping power usage is if a provider has oversold their power infrastructure and is starting to feel the pinch because they're not charging enough. Same goes for bandwidth: if you want to price things cheaper and cheaper to attract customer, I believe it';s unethical to then raise rates after-the-fact because of poor planning/forecasting.

Re:Our Policy

[Wesley Felter]

Power capping is intended to be used by the server owner; e.g. in a colo that would be the customer, not the provider. You give the customer a circuit and they use capping to fit as many servers as possible on it.

Re:Our Policy

[TubeSteak]

The testing revealed that the optimal workload was reached at a CPU utilization of around 50 to 60 percent with peak power at about 300W per server.

Wouldn't it be cheaper to start with a CPU that runs 50%~60% slower and uses 50%~60% less electricity?

Re:Our Policy

[Mad Merlin]

The testing revealed that the optimal workload was reached at a CPU utilization of around 50 to 60 percent with peak power at about 300W per server.

Wouldn't it be cheaper to start with a CPU that runs 50%~60% slower and uses 50%~60% less electricity?

Yes, except it doesn't work that way. A CPU running at 1 GHz might use 10-20% less power than the same CPU running at 2 GHz.

Re:Our Policy

[Cassini2]

I think you have it backwards. A CPU running at 1 GHz would use 10-20% of the power of a CPU running at 2+ GHz. The modern processor's power consumption is on a cubic power consumption curve. To increase frequency, you have to increase voltage, which gives a 2nd order 1/2 C * V^2 energy loss (per cycle). Additionally, the power consumption is the frequency times the capacitance, so the overall power formula is: 1/2 f * C * V ^ 2 or roughly O(f^3).

Someone more into the physics may be able to give you a better formula. The reason why CPU clock frequencies are not increasing anymore is that increasing frequency dramatically increases power consumption. The semiconductor people really have there backs against some serious technical walls. I wouldn't be expecting CPU power consumption problems to go away anytime soon.

Re:Our Policy

[Mad Merlin]

Then why do entire lines of CPUs have the same TDP? For example, the Core 2 Duo is 65W across the board from 1.8 GHz to 3 GHz: http://en.wikipedia.org/wiki/List_of_Intel_Core_2_microprocessors

Re:Our Policy

[Cassini2]

Then why do entire lines of CPUs have the same TDP? For example, the Core 2 Duo is 65W across the board from 1.8 GHz to 3 GHz

The entire line has exactly the same TDP because they are exactly the same chip. Intel automatically tests each one as its manufactured, and puts it into the appropriate speed bin. After a while, Intel's manufacturing technology improves, and they actually make too many of the fast chips. At that point, every speed bin gets the same chip. That is why, when the process technology improves, the best overclocking can be done on the slowest chips (provided you can unlock the clock multipliers, and the thermal packaging is identical.) All the chips have the same maximum speed, it is just the slowest ones start at the lowest clock speed.

It used to be, companies would actually produce CMOS chips with speed/voltage/power curves. That was a long time ago. Nowadays, Intel just calls everything 65W in a given line. Nevertheless, the early Asus EeePCs featured an underclocked Celeron processor to reduce power consumption. I recently purchased an AMD desktop processor rated at 45 (W). All the 45 W AMD processors had the lower clock frequencies than their faster AMD siblings. The effect is real. The manufacturers just don't always advertise it anymore.

Also, note that modern semiconductor manufacturing is very complex. AMD actively trades off speed, voltage, power, and fabrication technology to tweak its products for different market segments. This kind of optimization means each chip is specialized. The difference between 45 (W) and 140 (W) is more than just clock speed and voltage. If you increase the voltage on the 45 (W) AMD chip, you might break something. All of these fabrication optimizations make it very difficult to make firm clock speed vs. power dissipation comments. However, as a general trend, it is valid.

Frequency is roughly a cubic effect now. I can purchase either a 45 (W) 2.6 GHz AMD processor, or a 3.2 GHz 125 W AMD processor. I can't purchase a 4 GHz AMD processor, even at 140 (W) plus power dissipations. You can try rerunning the analysis with different models, but every analysis will say that the modern CPU designers have significant power dissipation issues when trying to increase clock frequency to 4 GHz.

Cutting back only gets ya so far.. :)

[tetrahedrassface]

Scaling down the use or throttling the usage during off peak times could be a good thing, but how much energy would it really save in the long run? It has been noted that data centers energy usage could double by 2011. Would steps like this really make a dent in that trend? I would grant that it might lower the curve, however it won't stop the trend of growing datacenter energy usage. I don't subscribe to the "permanent energy crisis" argument. There *is* a "permanent political crisis", meaning that things like nuclear, wind, solar, biomass, oil and coal are all wrapped up in a NIMBY attitude that has gotten us to this point. I understand trying to cut usage, however electricity usage is going to increase for near term and quite possibly into the long term. What we need is more generation from a wide mix of sources to power our datacenters, homes, businesses, and yes... our lives.

The funny thing is this where cost vs. efficiency start working and sooner or later when the maths are right for companies and individuals bottom line higher efficiency will begin to be adopted. But it has to make sense for the bottom line, or it won't fly.

Re:Cutting back only gets ya so far.. :)

[cdrguru]

Datacenter power usage isn't going to double in two years. It can't. There isn't that sort of power capacity in the US today.

Sure, we might build more plants. But even a common nearly-off-the-shelf coal plant is going to require five years to bring online. And as you point out, we don't have five years.

If it is going to be a decision between turning off power to homes at peak times or preventing new businesses from being established, we are probably going to turn off power to homes. At least in the short term. Because any other choice ends up just costing everyone more money - as unemployement grows and the working are taxed more and more to support the non-working.

And conservation isn't going to save us. Turning stuff off might.

Ew.

[Anonymous Coward]

That's all I have to say. Very messy way to 'solve' the 'problem'.

(I know it's 'hard to understand' for some, but people use quotes wrong on purpose to 'get the idea across', it really doesn't matter and they don't need a grammar lecture.)

Re:Ew.

[Anonymous Coward]

Can I quote you on that?

Mechanical batteries

[copponex]

Beacon Power is an American corporation specializing in flywheel based energy storage headquartered in Tyngsboro, Massachusetts. Beacon designs and develops products aimed at utility frequency regulation for power grid operations. The storage systems are designed to help utilities match supply with varying demand by storing excess power in arrays of 2,800-pound (1,300 kg) flywheels at off-peak times for use during peak demand.

http://en.wikipedia.org/wiki/Beacon_Power

The people who keep saying we can't find ways to be more efficient should stop wasting oxygen.

Re:Mechanical batteries

[ihavnoid]

2,800lb flywheels, plus the related cirtuits, plus maintainence of the flywheel ain't cheap.

Plus, proper power capping can reduce infrastructure cost, since the power distribution/conversion whatsoever system in the datacenter needs to be designed according to the maximum load, which means they need a lot of headroom on their power supply. By doing so, they can reduce the maximum load.

The only problem... is how they will reduce power while minimizing the interruption to the workload on peak times. Not an easy job, but I don't think it would be impossible. We should try, at least.

Re:Mechanical batteries

[afidel]

Those are insanely expensive and only store enough power for a few MW load for a handful of minutes. In theory they are better than a UPS in the long run due to not needing to replace batteries every few years but they aren't going to shift much load from offpeak to peak.

Re:Mechanical batteries

[merreborn]

Wow. Beacon Power's flywheels are actually pretty amazing:

The basic unit of Beacon Power approach is the Smart Energy 25, that is basically an enormous steel vacuum bottle holding a 2,800-pound cylinder made of carbon and fiberglass composite that is levitated by magnets.

Beacon's flywheels take in electricity and use a motor to spin the cylinder so fast that the surface hits Mach 2. The spinning cylinder stores most of the electricity's energy for as long as needed (thanks to the near frictionless vacuum and levitation, it spins for a long time). ....with so few moving parts that Beacon Power says they'll last 20 years with no scheduled maintenance.

That's really pretty fucking remarkable. (source)

Re:Mechanical batteries

[Lehk228]

and how much for the insurance in case one of those kinetic bombs decides to go off?

Re:Mechanical batteries

[MartinSchou]

and how much for the insurance in case one of those kinetic bombs decides to go off?

Not much, considering that the ISS runs flywheels naked (no containment).

From a fairly old article on flywheels:

Eric Sonnichsen, who founded Test Devices in 1972, points out that a wheel created from carbon fibers is safer than a steel wheel, because even if a few fibers break, the wheel won't come apart. On the other hand, if a flywheel does disintegrate, says Sonnichsen, "it's more like potentially lethal lumps of coal coming at you, traveling at kilometers per second."

His company helped to develop containment vessels to mitigate this worst-case scenario. Sonnichsen's team has overaccelerated flywheels and dumped them off their bearings; in each case, the wheel skidded to a stop harmlessly inside its container. They finally figured out how to blow a speeding flywheel apart by firing a bullet into it. "I tend to be the Cassandra of the high-speed spin world," he says. "But at this point I am satisfied with the centrifugal safety of flywheels. In fact, they are much less hazardous than other storage methods we have now. A can of gasoline can be dangerous. Even a car battery can blow up, if you reverse polarity."

In space, though, flywheels won't have the same containment structures, because the shielding would weigh too much. A sudden failure could be disastrous. So wheels must demonstrate through cycle testing and spin testing that mechanical components will last many times longer than expected use. The wheels are then derated to - that is, run at - 50 percent of maximum speed.

Re:Mechanical batteries

[TheLink]

Those look really expensive. How much do they cost?

If they cost more than they'll save you over their lifetime, they're not more efficient.

Another way to overcharge smaller users

[khodsden]

This happens to be why my quarter rack space has only 2 1U computers in it. It was supposed to be a quarter rack (10U), but I was told I had only 7U of space. Okay, not a problem, I can put in 7 1U systems, 14 if I purchase the half sized systems. Then I was told I have only 2A, oh, and here's a switch that'll turn it off if you go over. Which means my quarter rack has two 1U servers in it.

Worse, even the full rack is allowed only 15A before you have to buy a secondary power conduit to the rack at this particular colo.

I suspect it's more a way for the facility to make money than it is to reduce energy usage. When I visited the facility last to move boxes, 4 racks were being emptied and a good 60% of them were completely empty anyway, so the facility may not be long in this economy.
 

Re:Another way to overcharge smaller users

[TooMuchToDo]

A rack is only allowed 15A (in your case) or 16A usually out of 20A, because NEC code states you can only drive a circuit at 80% of it's max capacity (16A on a 20A circuit, 24A on a 30A circuit, etc). Colo customers should be taking this into account when they buy power. If you want to buy 10U of space, and expect to fill it with 10 1U servers, be prepared to pay an appropriate price, as it's not just space, but power, and cooling to cool all that power going into those serves. Every watt of heat you make is a watt of power needed to cool everything.

Re:Another way to overcharge smaller users

[BitZtream]

Except that the circuits are already designed with that in mind, which means when you get a 20 amp breaker for instance, the wiring is already enough to handle 25 amp.

Actually theres far more that goes into it than your simplistic version, but I might as well keep it overly simple as well.

Re:Another way to overcharge smaller users

[Anonymous Coward]

Get a freaking proper data center.

My half rack has 10A attached to it, with the permission to get a redundant 2nd 10A circuit.

My old quarter rack ran 7A permission.

High performance racks I can rent (a tad more expensive) go up to 80 A per rack.

Go to a professional data center. 2A for a quarter rack is not a data center. It is a joke.

I am mostly for high density computing - no sense in wasting space. Better have one high end system and use.... virtualization ;)

Re:Another way to overcharge smaller users

[TooMuchToDo]

I'm glad people like you don't do electrical for a living. There's a reason NEC code exists, and it's to stop stupid people. Being cheap with electrical is what burns your house/datacenter/etc. down.

Re:Another way to overcharge smaller users

[n4djs]

Ed Nisley of Circuit Cellar shared an interesting statistic a couple of years ago - that it costs approximately $2 per watt per year. This counts both the supplying the power to the computer, and the power to the fans and chillers to take it out...Needless to say, I don't put friend's servers in my basement....

Nuclear

[The Second Horseman]

Americans (and Europeans outside of France) are going to get over their nuclear power allergy really, really quickly once their lifestyles start to suffer more than a certain amount. Especially when counties like Iran and India start using it heavily and manage to undercut our economies with cheaper energy.

Flat out, if the carbon / climate change problem is as bad as is portrayed these days, and we think peak oil is a looming problem, nuclear is the ONLY rational response to the problem over the next few decades. It allows you to quickly lower carbon emissions while working on other technologies. Sure, there are downsides, but this is an emergency, right? Sometimes the lesser of two evils is your only real choice.

It looks like the UK may end up being the first power fiasco - last I heard, they have a few nuclear plants scheduled for shutdown in the next decade with no real plan to replace the generating capacity. Wind farms and solar sure in hell aren't going to do it.

Re:Nuclear

[Anonymous Coward]

There's a few more problems with nuclear than the NIMBY factor:
- Last time I checked, the return on investment for one of those puppies was in the area of 20 years. No business makes an investment like that, and apparently neither do governments.
- There's little to no global political or economic power returned by investing in nuclear energy. Oil allowed the US to dominate global economy and politics.
The intelligent thing to have done with that big wad of change the US threw about was to build as many power plants as it could and prop up alternative energies both on the supply and demand side. Sadly, I don't think we'll be heading that direction until it's too late.

Re:Nuclear

[cdrguru]

Sorry, it is probably too late for that. It would likely take 10 years to build a nuclear plant of any size. We are going to be seeing huge power shortages in the US pretty soon, and conservation isn't going to make a bit of difference. There isn't any point in shaving a couple of megawatts off the load when the shortage is a couple of gigawatts.

Building a coal plant today would require at least five years, if we started construction tomorrow. We aren't. I don't believe there are any new major power plants even through the basic permit process. There is some talk of some nuclear plants that have just started the permitting process. Ten years from now we might see some benefit.

I believe we aren't going to be in much of a position to take advantage of that. If everyone knows that electric power is unstable and unreliable, they will change their behavior to match. Ten years from now we will not be expecting constant reliable power and any changes will not help.

Already happening

[dasmoo]

My Datacentre power caps me. I can't install any more hardware into a rack I'm paying good money for. Because they don't have enough power.
Don't go with Primus for anything.

Metered datacenter?

[Anonymous Coward]

Are there any datacenters that meter energy per customer? This would lower the base price per RU and encourage energy conservation without forcing anything. This would be just like apartments that meter energy per unit.

This would entail having two things: power-on over LAN capability, and the ability to check your "electric meter" online--and maybe even see how many watts each outlet is using. IIRC, APC allows all of this.

Re:Metered datacenter?

[TooMuchToDo]

APC doesn't allow per-outlet monitoring, only per PDU monitoring. UCSD or UC Berkley is working on small per-outlet devices to both monitor and control loads (i.e. servers) that are to cost less than $20/piece. Assume they are controllable via HTTP, and you can do some fine power control.

Re:Metered datacenter?

[afidel]

Basically every colo datacenter is metered. Standard is 2x20A circuits (16A max draw) per rack. If you want 30A or quad 20A circuits in your rack you generally pay a hefty overage charge. At our DR colo provider they require metered PDU's. I have this capping capability with my HP servers which means I can fill a rack right up to the edge but make sure I don't overload the circuit by keeping my peak usage closer to average. You also need to use staggered startup on your servers if you want to play it that close to your power cap.

Can I get it on my laptop?

[zippthorne]

I'm sure we can all think of circumstances where we care more about how long a machine stays on and somewhat functional than how fast it is.

Like movie watching: a few dropped frames are a small price to pay for finishing the film. If it's bothersome, maybe the brightness can be lowered during actiony bits.

Or typing while the thing is actually in your lap: not burning your groin is far more important than a hastily typed "find -exec dostuff {};" operation in the background finishing quickly.

Reduce power - serve fewer ads.

[Animats]

I'll believe this when I see more sites that start dropping unnecessary ads and tracking when under heavy load. (Slashdot does some of that; under heavy load, most users get a canned home page. When the system is less busy, the "customization" machinery is used.)

The real killer is overdoing "customization". Customization makes serving pages far more expensive. Consider Google's problems with "Michael Jackson" searches. Google used to answer the most common queries from a cache in the first server to which the client connected, without actually going to the big search engine in back. That makes it very cheap to answer huge numbers of similar queries. But "customization" breaks that; now every search has to pull up the user's profile and rework the output.

oops...

[derGoldstein]

Just as I posted that I remembered that idle power draw on 3.5" drives isn't 1W, it's ~4W... I was thinking about 2.5" drives. This is compared to the 40W to 60W that a typical CPU will consume.

Deactiving subsystems

[Midnight Thunder]

Just as I posted that I remembered that idle power draw on 3.5" drives isn't 1W, it's ~4W... I was thinking about 2.5" drives. This is compared to the 40W to 60W that a typical CPU will consume.

This is why I was suggesting a smart system that can deactivate cores or subsystems until they are actually needed. Most servers aren't designed with energy consumption considerations, so they still draw a fair amount of power even there are five users visiting the web site. If they could be made to use the least resources possible to get the job done, then it would probably end up saving a fair amount of money long term.

Re:Deactiving subsystems

[Fulcrum of Evil]

the new AMD chips can do this, but apparently it isn't quite as responsive as they'd like. Perhaps exposing hooks to allow the OS to control this would work better.

Re:Deactiving subsystems

[derGoldstein]

If they could be made to use the least resources possible to get the job done, then it would probably end up saving a fair amount of money long term.

Finally found the term they used: "power gating". Here's an Ars article about it.

The relevant bit:
Traditionally, Intel has been able shut down an unused core by cutting its active power, but even though it's in a sleep state, that core is still dissipating plenty of power because of leakage current. Intel's power gating technique involves a new transistor design, and it lets Intel cut the leakage current, as well, so that the sleeping core's power dissipation drops to near zero.

AMD also has some term they use for this, but I can't find it.

The question is what happens when you really *do* need the processing power? It seems to me that they should focus on lowering the power draw during peak usage, which I understand they may do by designing many-multi-core CPUs (16+) that run at relatively low frequencies.

Hydroelectric power

[kriston]

I have to note that the big datacenter concentration of both Google and Microsoft in The Dalles, OR, is powered by copious, carbon-negative hydroelectric power. Discuss.

Re:Hydroelectric power

[benjamindees]

What makes you think it's carbon negative?

Datacenters need reliable power and water. They will likely be able to outbid other users for the foreseeable future. They don't want to have to move their datacenter if a coal or nuclear power plant is scrapped.

Re:Hydroelectric power

[BitZtream]

Carbon-negative? WTF are you talking about. Hydroelectric power doesnt' consume carbon nor emit it, its neutral as far as carbon is concerned.

And...

[benjamindees]

Hydroelectric is used for peak power production, which coincides with the cooling power profile of a typical datacenter.

Power capping is unclear

[Orion Blastar]

on how it saves power. For almost any business you will need your servers running 24/7 esp if they are hosting web servers for your business. Running a power capping technology that shuts down the servers hosting your web site doesn't make much sense because when it is shut down nobody can access it. The whole point of running a web server is to provide 24/7 services even if your business is closed and everyone went home. But for some businesses they have people working all shifts and there is never a down time for hospitals, law firms, fire houses, police stations, etc. So why would you want to power cap, say a 911 Computer Automated Dispatch server that has to run 24/7 and can never be down?

Yeah I know you can shut down parts of the server not in use to save 20% energy etc, like the monitor and graphics card, etc, but you cannot shut down the hard drive or network controller if it is constantly being used. I guess you can even shut down cache memory, but it would make the server slower in a way.

It actually makes better sense to design servers that use less electricity than to rely on a power capping technology to do it for you. When are we going to see "green servers" that not only reduce the electricity used, but also include batteries to store electricity on them for use later on when electricity becomes expensive and the server needs to use less power from the A/C source and pulls power from the battery instead.

Re:Power capping is unclear

[Wesley Felter]

Power capping slows down the server, but it continues running.

Re:Power capping is unclear

[Orion Blastar]

Just what a company needs, a slower server. Nice bottleneck there.

Re:Power capping is unclear

[BillyGee]

Web servers aren't of course the only type of servers running out there. You know the whole big shut-off-the-light initiatives all over the world in office buildings? Well this is much of the same. You may not need terminal servers going 24/7, or even some databases that are only used during business hours for internal operations. Or you likely have development, test, live environments - why not shut down dev and test during hours where no one uses them? Or for larger operations, smart monitoring may shut down / start up cluster nodes as necessary, as demand falls or rises.

As other posters have mentioned, it all comes down to economics. If it there is no cost benefit to the server owner then there's no incentive other than goodwill to do any of the above. Currently, I don't know of any provider who separates the electric bill from the rest of your bill - as in if you keep your servers shut off the entire month, you pay nothing, if you draw huge power, well then you get walloped.

In cases of larger operations / data center owners who have their own dedicated data center, then other than the electric bill itself, the huge benefit would be the (un)necessity of upgrading the power infrastructure. Or in cases of capped power in an area (yes, there are some where extra power cannot be had from the utility because their own infrastructure would have to be upgraded and the resulting passed down cost would be unfeasible), the simple fact that you can still add more servers is also incentive enough.

Isn't that inefficient?

[robzy]

Correct me if I'm wrong, but wouldn't power-per-calculation actually be hurt by that? If you do this, you might end up with 50 racks drawing 50kW (for instance). Yet you'd actually be able to do the same number of calculations, with _less_ power, if you had 40 racks drawing 45kW. Rob.

Green Datacenter Addition?

[woutersimons_com]

In our datacenters we have already started to employ special airconditioning units, raised operating temperatures, redesigned floor layouts, etc. All of this to reduce power used, but not because we do not have enough power, because it is commercially attractive to say and show how "Green" our datacenters are. I suppose this can be seen as another way to prove how green you are, especially if you host services and storage and not dedicated systems.

i dont think we seem to understand

[nimbius]

energy crisis or not, reducing the amount of energy we use in the data center is a good thing all around. if not for global warming, then for business expense, and if not for that, then the potential to make life as a noc power and cooling specialist alot easier (we hope.)

Makes sense in some situations

[Hydrian]

I could see it making sense in certain situations. Cloud computing is a good example. If you some of your OS images are parsed across a few machines with only 20 percent usage and if you could devise a way to seamlessly migrate them to other machines so there is more OS image density per machine you could shut down some of the unnecessary capacity. You would obviously leave a few machines open for hot machines if there was a usage spike, but the rest could be shut down and be powered up when the hot spares start to used.

What energy crisis?

[Guspaz]

Sure, power costs in Quebec have gone up over time, but not THAT much. If the US refuses to build sufficient capacity, just move servers to Montreal or some other Canadian city with cheap renewable power.

Same at my last DC

[phorm]

Not a DC where I worked, but where we had rack space at. Each cabinet has to start 15-20A, and you could request up to three strips (total 45-60A) per rack.

They mentioned at one point that they were actually reaching the capacity for power on our floor, so we'd have to be careful about power as ordering more strips wouldn't be an option unless another rack elsewhere let some go.

No capping needed.