Slashdot Mirror
UPS Setup For a Small/Mid-Size Company?
An anonymous reader writes "We're a small company employing ~30 people and we are becoming increasingly reliant on virtual servers. Unfortunately, the hosts they are on don't have redundant power supplies because we simply don't have the capacity. We currently have one UPS per rack, which gives us about two minutes. This may have been enough time when they were put in — they've been there for some time — but it isn't really enough time to shut everything down in the event of a failure. Domain Controllers alone may take up to 15 minutes. So I'm looking at upgrading the UPSs to ones that would preferably give us around 15 minutes of breathing space and send an email or text alert when a failure is detected. Something that could trigger shutdowns automatically would also be nice. Of course cost is a key factor too. so given all of the above, what does Slashdot recommend?"
Get a generator that can power things from natural gas (or other available resource).
So when the power goes out, it will be seconds before the generator kicks on and the UPS are just there to keep power available until the generator is ready.
Mod me down, my New Earth Global Warmingist friends!
This is a much better solution. Plus, it can melt your face!
When you're afraid to download music illegally in your own home, then the terrorists have won!
Not knowing the load required on the UPS makes it very hard to tell what kind of UPS you need. You need to know how many watts are used in the rack to be able to plan some proper UPS capacity.
apcupsd can be networked between machines and can trigger auto shutdowns of all of them, including VM guests.
Some virtual machine system can also suspend all VMs on shutdown which could be a better alternative then shutting them down. Again, without knowing which VM system you use it's hard to get into details.
You haven't provided enough information. To answer your questions we'd need to know how many racks, how many watts or Vamps per rack, or even the type of servers you're running. On top of that you mention that cost is an issue, but you don't mention a budget.
Without having that info imagine the following scenarios:
1. You have 1-2 racks with 4-5 piece's of equipment each
Get a Large APC (or comparible unit for each rack)
2. You have 1-2 racks halfway populated
Get an expandable hardwired rackmounted APC
3. You have 1-2 (or more) racks fully populated
Get a large hardwired dedicated UPS
Of course none of this considers anything beyond just bringing the systems down gracefully. If you want something more than that you might want to consider an outsourced datacenter or a generator.
We have had good experiences with the HP R5500 XR. You may require a smaller and cheaper model like the R3000 or R1500 depending on your servers.
These UPS are fully supported by NUT.
The months are just too short. I can count the number of days on one hand.
I have 2 3000 watt APC SmartUPSes per rack. They have both Serial and USB notification. Since each rack has about 25 servers, I get around 25 to 40 minutes of runtime for each server. So I have a small PC for each rack that monitors those 2 devices. It connects by serial to the upses, and runs CentOS. Then I have APCUPSD installed and configured in multi-ups mode. On each server, I simply install APCUPSD (There is a windows version), and tell it which UPS it is on. I also configure the appropriate shutdown parameters (20 minutes of battery left for non-critical servers, 15 for DC, and 5 for other critical servers. I also hooked each UPS monitor into Nagios and Munin, so I can track each one's power output and time remaining. So far, it's worked great over 2 "brownouts", and 1 total power failure (a test where I simply tripped the appropriate breakers).
The rational behind having dedicated UPS monitors, is that I don't really care if the loose power while running, so I have them set to never shut down from UPS activity. Then, I simply implemented a script that on power restore issues a netboot command to each server under its control (configured with puppet for Linux, AD for Windows). That way, the whole system (all servers) automatically shut down, and turn themselves back on even if they never really lost power... So far, it's worked flawlessly (and with nagios, I get a text message on my cellphone within a minute or two of a UPS switching to battery (we have 2 dedicated internet connections that are on different power sources and different UPSs.
I hope this helps!
If a man isn't willing to take some risk for his opinions, either his opinions are no good or he's no good
Its time to break out the calculators and do some math. There are two main factors at work here, UPS load capacity and battery run time. I run a series of research clusters at a university, so only the core systems (landing pads, schedulers, auth, disk arrays) are on UPS and all the compute nodes just die at a power hit.
Retrofitting a datacenter for whole center UPS is a very daunting and expensive task, so odds are good you'll be replacing the current rack mounts with beefier units, either pedestal sized units next to their racks or rack mounted units.
When buying UPS gear for work, I aim to hit either 67% capacity with the planned load, or the smallest VA rating that takes 208V single phase, as long as its at least 1/3 under utilized for future expansion. That covers the VA rating. As for battery run time, most of the larger units accept external battery packs to increase the run time. I've never used them, since a 5KVA unit with my load gives me 20 minutes of run time, and if the power isn't back on by then, odds are good its not coming back any time soon.
Another option for extending UPS run time is to prioritize services/VMs. With the appropriate monitoring software on each host, you can configure each host to shutdown when the UPS estimates X minutes of battery time remaining or there have been Y minutes on battery, or both. Less load, more run time for the really important stuff. Almost every UPS I've used (APC, Tripp-lite, Powerware) comes with off the shelf software or there are opensource solutions (apcupsd, nut) for monitoring the UPS over serial, USB, or SNMP (Options vary with mfg and model). My shutdown schedule is: after 5 minutes on battery, power down the compute cluster landing pads. With 10 minutes remaining, power down the file servers with the archival data on them. With 6 minutes remaining, power down the primary file servers. With 2 minutes remaining, power down the auth box/network monitor/iLom control host (This is the only one that can't get powered on/monitored remotely).
I use a Su-Kam inverter at home. It powers a whole room, has a clean sine-wave output (unlike traditional UPSes), and its switchover delay is small enough that the SMPS in computers handle the switchover to battery power properly.
It uses two large lead-acid multi-cell batteries (~car batteries) for storing charge. The last time there was a major power cut, it powered my computer systems for 10 hours (yes you read that right... 10 hours.)
I was laughing at the old APC UPS which did 10 minutes before I had to power down.
This is India btw.. power cuts are common.
Banu
Meh, a 5500 VA UPS can drive a rack full of low-end 2U servers.
I had a $3000 UPS to keep a big Sun alive. After a couple of years they're $99 on ebay with dead batteries. But they have some of the cleanest pure sine wave power you ever saw. Best. Inverters. Ever. Capish?
The batteries last 5 years then you replace them, period. They're "Sealed lead acid" or "SLA", in plastic cases with two tabs. They come in various sizes. Get the same ones. Be careful where you get them. $8 batteries off ebay tend to be $40 in the wrong store, for example, one is the same as a chair lift, and the medical devices store that have them in stock want $40 ea. Of course shipping LEAD acid batteries ain't cheap.
The batteries for the UPC2200, just as an example, are $150 new for the pair plus shipping. $99 for the chassis (plus shipping and they're ungodly heavy without the batteries) and you have a $3000 UPS.
That'll keep a small server running for a while if you give them one each. But you'd have to be a bit of a dick to have a dozen of these running a dozen servers, what you want is a one ton 12V battery, the kind your phone CO might use, a huge ass inverter and some panic circuit to cut power over to battery when the line goes down. That's the proper way to do it. Once a year they come out and recharge your battery for a small fee. These batteries cost a grand or two but last a long time. Refurbs are fine.
The other nice thing about big batteries is if you get wind or solar stuff added on the to the
building you can just wire that power in to the battery with no charge controller. Cause, uh, there's no fear your solar panels are gonna overcharge a ONE TON battery.
Need Mercedes parts ?
It's no act. I am happy to put up photographs of my setup if you want. It's been working well for me the last year, so I don't have issues recommending it. Apart from being a customer, I have no connections to any inverter/battery company. You OTOH are an anonymous coward. Here is my website. Go find more about the shill there.
Banu
If it is a Linux (or BSD) box then he can use APCUPSD. It can send emails, has network feature to let other machines know the status of the UPS, and even has a web interface for monitoring the UPS.
Not properly maintained propane storage/transfer sites have be known to explode. For instance: the 2008 Toronto explosions, the 2006 explosion, and the Feyzin disaster. Propane is highly vulnerable to a BLEVE - Boiling Liquid Expanding Vapour Explosion. Many more smaller propane exposions have occured, they just did not make the news.
I've seen what happens when Propane explodes. I would think twice about using it for an emergency backup fuel source.
http://www.apcupsd.com/
Apcupsd can be used for power mangement and controlling most of APC's UPS models on Unix and Windows machines. Apcupsd works with most of APC's Smart-UPS models as well as most simple signalling models such a Back-UPS, and BackUPS-Office. During a power failure, apcupsd will inform the users about the power failure and that a shutdown may occur. If power is not restored, a system shutdown will follow when the battery is exhausted, a timeout (seconds) expires, or runtime expires based on internal APC calculations determined by power consumption rates. Apcupsd is licensed under the GPL version 2.
The developers system
http://matou.sibbald.com/cgi-bin/multimon.cgi
I'd go on a Vegan diet but the delivery time from Vega is too long. --brownkitty
"I can smell the curry from here."
Shmuck detected.
Large lead-acid batteries (or Optima batteries if servicing is a concern) are cheap compared to most UPS batteries, inverters are old, proven technology, and systems using them are easy to set up.
"This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
An UPS is not an alternative to redundant PSUs, it just seems that way until your PSU fails.
I've got a server room with about 150 servers (physical and virtual) every physical server got redundant PSU and the whole room runs on a PowerWare 9305 30 kVA UPS.
...and yes I should have redundant UPSes too, I just dont have the room for another one.
"Troll?"
It wasn't original or funny and he spelled "hamster" wrong, but "troll?" Who woke up with mod points and a dick in their ass?
If he were trolling it would have read more like "Or if you have a bunch of hampsters around for whatever reason, all you'd have to do is punch a Jew in the nose and fellate Hussein Obama while using Windows."
At home, I use 1 APC UPS per PC, except for the laptop. These cheap units give about 30 minutes for the PC and its immediate peripherals (displays, external disks). The PCs run Ubuntu, and automatically recognized the UPS units when they were connected via USB ports. Another UPS powers the fiber switch, headless server, and router. Yet another keeps the network printer and an ethernet switch powered. Total of 4 APC UPS units and less than 350euro investment.
We need this, because we're in the countryside, where power glitches are depressingly common - but often last just a second or two. If the outage lasts until the UPS goes below 50%, then the systems shut down gracefully. By then, of course we would have saved any work in progress. Keeping the whole LAN in operation during this time is essential.
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
DirecNIC (an ISP located in New Oreleans) managed to keep their operations going for the entire duration of the Hurricane Katrina disaster and long after that with practically no downtime thanks to their backup generators. They didn't just shut things down - they kept right on going. If you're in an area where power loss or some other sort of disaster is a real threat then this is absolutely the way to go. Bonus, you can pretty much keep the UPSes you have because they just need to keep the systems online long enough for the generators to switch on (as the parent said).
Random Thoughts From A Diseased Mind (Not For Dummies)
Here they are: inverter1.jpg, inverter2.jpg
I'm sorry it's pretty dusty, but this has been exposed to the elements for the past year. It has to be kept outside because of the lead-acid batteries, which need to be ventilated. The stand is an old TV stand reused to host this. The inverter is on top. The batteries are at the bottom.
The little yellow alien looking caps that you see filters that the acidic fumes from the batteries. Each cap tops a cell. The little stick on top indicates the liquid level inside the cell. After about a month, the levels go down in some of the caps and I call the local shop to come and top-up distilled water. Basically the water evaporates whereas the acid is still there, so they fill in distilled water. I could do this myself, but the local shop does it for about $1 so I just let the experts handle it :).
Banu