Slashdot Mirror
Generator Delays May Slow Data Center Projects
miller60 writes "The data center building boom is causing backlogs for new generator orders, with some companies reporting delivery delays of up to a year for new 2,000kw units, which are the current standard for mission-critical facilities. Generator availability is 'the No. 1 thing that will drive your construction schedules,' according to Equinix, which is building centers in three major markets. 'This will be a big issue for the next wave of data center builds,' says another industry executive. Used generators and smaller units tend to be more available than the 2 megawatt units, but companies targeting the enterprise sector may be wary of relaying on used units or smaller generators than those powering competing facilities."
HOLY MOLY, that's a lot of power! If you had 250 watt power supplies, all running at maximum, you'd be able to power 8,000 power supplies simultaneously! You could run a small town on a generator that large! Or to put it another way, you could use it to power a Diesel Locomotive capable of pulling dozens of fully loaded cars.
Wow, just wow. That's just an incredible amount of power to be putting in a datacenter. Is it even possible for these centers to run off the grid?
The part that I don't undestand is this: Why do they need just one generator? If you're having difficulties obtaining a 2Mw unit, wouldn't it make sense to get two smaller units? You'd waste a bit of extra space, but you'd have redundancy that a single genearator couldn't offer.
Javascript + Nintendo DSi = DSiCade
Easy, just buy 400 5kw generators instead of 1 2000kw generator.
You may have to hire more people to start them, though. And change the oil.
Lose Weight and Feel Great with Isagenix
If you've built out your data center but the generators have not yet arrived, a quick and inexpensive solution is to buy several thousand hamsters and hamster wheels. These will keep your data center humming along in the event of an outage. If PETA starts protesting, sysadmins with larger wheels can be substituted. Most of them will find this activity a welcome break from the inane busywork in their everday lives.
You need to remember, its not just the computers & servers. Its the lights, the heating/AC, all other power needs. We have a generator at my work and it has to power EVERYTHING. 100 desktops 34 thin clients, 19 laptops, 34 servers, the A/C for the data center, the elevator, the emergency lighting, fire detection system, phones... you get the idea. We have a 30 KVA battery backup for the data center in case the generator doesn't kick on the right way.
Take out a $10,000 simolian loan and place a natural gas power plant in a corner. It costs $9,000 simolians and produces "moderate pollution", so residential and commercial zones won't develop well nearby. But your city won't grow quickly without a lot of power.
Developers: We can use your help.
I didn't see any delays on a project I am working on. We have 4 Gens in the 2MW range plus 3 gas/diesel gens in the 30MW range. Of course, this isn't for a datacenter, but I wonder if the generators would be the same.
You can keep your generator(s) running indefinitely. Certainly longer than any predictable power outage, but if you're running on batteries you're against the clock. What are you going to do when they start to run down - nip over to the 7-11 and buy all the AAs they've got?
init 11 - for when you need that edge.
You would think that companies that make larger generators (I'm talking > 100kW) would understand that demand goes up during hurricane season, and things of that nature.
We only encountered a one month delay on delivery of our 125kW natural gas fired generator. Our delay was mostly because of hurricane Katrina having struck the gulf coast. We had to pull some serious string but since we wouldn't be moving in until November of 2005, it didn't really impact us.
need 20 or 30 2MW generators! [...]I know there are several 60-75MW units under construction in the US now
Comming soon on eBay : 100MW generators
starting bid : 10$
condition : mint
reason : our startup went belly up.
When the next speculation bubble burst, there're sure going to be a lot of diesel mastodonts left every were...
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
Step 1: Take a price from this map.
Step 2: Multiple that price times 2,000.
So for California, 2,000kWh would cost $240 per hour to run. That's $5,760/day, $40,320/week, and a whopping $2,096,640/year!
Of course, for diesel your prices may be higher. As of right now, diesel is approximately $2.669 per gallon in California. To compute the costs, you'd need to know how efficient the generator is. This page claims "approaching 40%", so we can use that for a guesstimate. At about 146,520,000 joules per gallon of diesel, we can compute a need of 122.85 gallons per hour. At the going rate, it would cost ~$327.89/hour to run a 2MW generator.
Javascript + Nintendo DSi = DSiCade
I was having dinner with an IT guy from Skywest Airlines the other night and he told me in the last big power outage in St. George, Utah where they're based (and I live) their battery backups ran out quickly and there was a hardware problem between their generators and their IT department.
The end result is that all of their servers and network equipment went out for hours, and they had to cancel a whole lot of flights costing the company well over $1,000,000.
You can buy a lot of crappy gas generators at the Lowes across the street for $1M. I think I would have sent two guys there and two guys to the gas station to keep the essentials online.
There is a lot to be said for redundancy. Redundancy is very important. You can't talk about redundancy enough. Seriously, it's better to be twice as redundant than only half as redundant. And three times as redundant is even better than twice!
Lose Weight and Feel Great with Isagenix
In a former life, I worked on large (800KVA) uninteruptible power supplies as a field service engineer. Normally, for highly mission critical data centers, as described in the article, the data center is powered by 2 different, independent power grids (if possible.) For example, there would be one feed from power company A, and another feed from power company B. These would be connected by using an automatic transfer switch (ATS) which would immediately switch to the other power feed if the main failed. The output of this ATS would then be connected to another transfer switch which was connected to the emergency generator, which starts automatically in the event of a power failure.
On the downstream side of this second ATS (the one that switched between the redundant power company feeds and the generator feed), there would be the redundant UPS network, which would in turn feed a different type of transfer switch before heading to the actual critical load in the data center. Each of these multiple redundant UPS's were provided backup power with large strings of batteries. A typical 800KVA UPS would have 3-4 strings of 40 batteries each. Each of these batteries would cost around $100-250, and weigh about 100 lbs. The data centers I worked on, which were nowhere near as large as some of the MS or Google data centers described in the article, had between 4-16 independent, redundant UPS's.
How long would this huge amount of batteries keep the UPS up at full load in the event of a power outage?
About 15 minutes, if you were lucky and had batteries at the top end of their charge cycle.
The whole purpose to having the UPS redundancy and massive battery backup is to give the emergency generators enough time to start and come online.
When I was working at MGE, a typical figure we were told was that a minute of downtime for some of these data centers can cost the datacenter well over a million dollars in lost revenue, and immediate termination of the CIO (or whatever equivalent) of the company. I'm not sure how accurate this was, but it seems reasonable considering how much money goes through some of these places in a day.
Our datacenter has about 24,000 sq feet of raised floor (not huge by datacenter standards) and we have 2 x 2MegaWatt Generators (as well as redundant utility feeds). To say that 2MW is too much power in this case shows your lack of understanding of what a "MAJOR data center with plenty of redundancy" means.
Prior to working here, I was an engineer at one of the main datacenters for a big Texas based IT provider (think Ross Perot) and their datacenter had over 100,000 sq feet of raised floor. At last count, they were up to 9 750KW generators and badly in need of more.
While we're on it, cooling towers do not provide chilled water, they provide CONDENSER WATER, and I promise you that they would not be able to satisfy your cooling needs in a MAJOR datacenter for 2 hours in the event of a chiller failure.
I can appreciate your being surprised at the power/cooling requirements of a datacenter, but don't let your experience at 1 "datacenter" fool you into thinking you know about all datacenters, as like most things, they are not all created equal. In truth, what is one man's datacenter is another man's "server room".
I would bet that a datacenter probably uses as much electricity as a small skyscraper. Because of the high equipment densities, a 1-story datacenter filled with racks probably has HVAC requirements that are like a multi-story office building filled with cubes. IT doesn't have the same lighting requirements, but that's not nearly the draw that heating and cooling are.
Actually, I bet that in many situations, if you just pulled the plug on a 'center, very bad things might happen to the equipment, aside from the obvious ones like data loss. Even without the machines actually producing any more heat (because they're off), without cooling air being forced through them, in very high density racks I wonder if the residual heat might not build up to rather high temperatures and become a problem; damaging hard drives or other temperature-sensitive parts.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
"Modern Generators" as you call them do not happily sync together under load. If you have PLC Controlled Switchgear associated with them, then you will be able to sync them, and even then they'll never be perfect. If one generator lags and becomes a "slave" it also becomes a load to the "master" generator, thus reducing your available power to the critical load. It's definitely not as simple as just dropping a few generators in a row and wiring them to the same buss, I assure you.
Also, a 1MW generator is pretty large, as is the switchgear and any fuel storage tanks associated with it. As far as the UPS's, true, they will take care of brownouts and frequency errors, but you want the power coming in (either utility or generator) to be as clean as possible so that you're not beating the shit out of your batteries every few minutes to take care of spikes. Batteries do have a shelf life and they're not cheap.
I see a lot of posts that 2MW is a lot of power. Perhaps.
There are many facilities out there with 10-30 2MW generators. I personally worked on a facility that had 18. 12 were for "critical load", that is, things fed via UPS's. This included all of the servers and networking equipment, the NOC, emergency lighting, fire supression systems and soforth. The other 6 were for "emergency load", these were things that did not need UPS protection (they could go down), just not for long. Virtually all of this load was air conditioning. So when the power went out the A/C's went off and came back 30-60 second later on generator, while the servers and all stayed up the same time.
While big, it's far from the largest facility out there.
Now, why would you buy one generator? Well, many buildings use bus variations (the whole N+1, or N+2 thing). So you build your data center for all 18 generators (as above), but install 6 and 3, or half the capacity. You now wait for the building to fill to 25-30%, and then start adding more generators, one at a time. However, they now have to be matched to the other generators.
Now, why are data center generators special? Well, to switch from one AC source to another the two loads must be approximately in phase (there is some tolerance, but it's small). So in order to be able to switch between generators, switch from UPS to generator, and all that other stuff you need additional circuitry to keep the generator just so. While the engine block and generator were the same as say a diesel locomotive, there were some additional sensors, lots of additional computer control, and some additional quick start features.
Most sites want their backup generators to be "ready to load" in 15-30 seconds. Indeed, several manufacturers make 10 second ready to load units. However that requires things like engine oil and coolant heaters that basically keep the entire motor warm 24x7x365. These are not found on industrial generators or locomotives.
So, they really are special, high tech tight tolerance units designed to work in a system. When you connect to WoW or your other large game you're probably one one of a thousand servers run by that company, who is one of but hundreds of companies in the larger colo facilities. It's not uncommon to find 50,000 servers under a single roof. So you need lots, and lots, of 2MW generators.
Your product is interesting but there is a difference. The reason the cost per KW of engines over about 500KW starts to rise is that they are designed for very long life. Unlike gas engines, where when you get outside the cylinder size range of about 50-500cc things start to go downhill, Diesels scale to enormous cylinder sizes but, because the mean piston speed needs to be constant, power goes up as the 2/3 root of cylinder volume. Mass scales more closely to volume, so mass rises faster than output. However, the benefit is that the wear life of a larger cylinder is much greater, because it can tolerate larger amounts of linear wear before blowby becomes excessive.
The implications are that while the cost per installed KW of your multi-engine plant is about the same as that of a single 2MW unit (because of the additional switching and control gear) it will not have the same service life and its lifetime cost per KW is higher.
Correctly and honestly, you describe your generators as backup. But larger units in the 2MW and over range can be used as primary generators. That's the difference. As somebody else has observed elsewhere, large units can be remote controlled by electricity utilities for handling load peaks. This means that their asset utilisation can be much higher than standby generators in well designed stationary applications. The lifetime cost per KWH of a backup generator can be very high because its first cost is amortised over low running hours.
Fact is, 2MW isn't a big Diesel. It's portable power (as used in trains and boats.) That's why supply and demand is likely to vary according to major events like earthquakes and wars.
Pining for the fjords
Where I used to work, they had 6 diesel locomotive engines, all in a row in a big room. When I went on the tour, they told us we could only stay in there for a minute, because of the possibility of them firing up at any time (the sound would be deafening). Each of the engines sat on an isolated "pad" which reduced vibrations. They told us that was necessary to prevent the harmonics for damaging the building. It was really a beautiful thing :)
In the summer, they would sometimes run them during peak periods. I was told that they generated enough power to run the datacenter (which is one of the largest in the midwest), plus sell enough power back onto the grid to earn a sizeable kickback.
Technology is cool.
http://cubemonkey.net/quotes -- fortune-mod quote generator