Slashdot Mirror
What To Do During A Power Outage?
Chanc_Gorkon asks: "What do you do before and after a power outage? Does your security people call the IT people? Do your systems come back automagically? I mean yeah a PC server may not have a problem, but what if the server comes up before the backbone is available? What if there are certain things that must be done manually due to security policies or because that's the only way it can be done (our UPS has to be manually brought up after an outage as well as our S/390). Do you review how you did after the disaster or do you just say thank God something serious didn't get broken and go on with things? What do you do?" I think many places actually have backup generators for situations like this. Many other shops live and breathe by the UPS. What are your procedures for handling heavy elecrical storms, and what are the best ways to handle the occasional dissaster?
Our policy was that all local vital functions were on UPS. This included the switcher, the master control tape machines, the NT boxen that handled the automation, the Unix boxen that served the advertisements, the microwave antenna that sent the signal over to the transmitter, one of the small satellite dishes, and probably a couple of things I'm not thinking of. Plus the building had backup power for emergency lights for about 6 hours (after that the flashlights came out...that only happened once during the 2 years before the big generator was installed).
When I started working there, the station itself had a small generator for vital functions that would come on after ~15 minutes of failed power. A couple years in, we installed a bigger generator that could handle most of the normal functions also. It was, I believe, an 800HP diesel generator with a fuel tank big enough for about 30 hours of operation with no power.
The transmitter (being a UHF station, which requires TONS of power) couldn't run off of a UPS because I don't think they make them big enough, but it had a generator that would automatically start every time there was a brownout and stay on until it was needed, or until the power returned to normal and stayed that way for about half an hour. However, whenever a big storm was coming the generator at the transmitter would be turned on by the engineer on duty and someone would drive out to be prepared in case the automatic switchover to generator power didn't go smoothly. (Those big transmitter tubes are pretty quirky...it doesn't take much to throw them offline. A big voltage spike can cause them to overheat and shutdown. And sometimes they drop offline just for the fun of it.)
The generator at the transmitter was a 1600HP diesel with enough fuel for 4 days (!) of uninterrupted operation with two tubes online (normally, we ran two video tubes and one aural tube at full power...when at half power, one video tube was dropped offline and the aural tube was run at half power). This fuel tank exuberance was, in the words of the Chief Engineer, "In case someone can't get out to the transmitter for a while...nuclear war and plagues of frogs do happen. We'd like to stay on the air."
Oh, yeah, the transmitter was also connected to two power grids via direct lines (not shared with any other buildings and contracted by the two TV stations that were on the tower). We had a direct phone number for a 24 hour on call power company technician, that could be called when power failed. The power company tends to pay attention when a company with multimillion dollar a year power bills calls.
This probably isn't representative of all TV stations, the station I worked at was in Houston...the 5th largest TV market in the US. But I'm sure most have similar plans and equipment, if not the same excess.
Just thought it might be of interest. I don't really know how network folks handle such big jobs. We've got UPS power for our vital machines and quality surge suppressors on the rest, and that answers our inhouse power needs OK. ;-)
A funny, and interesting tidbit for those who stuck it out through this whole post: When a TV station must drop off the air for a few minutes at an unscheduled time (like maybe a tube is failing and needs to be switched out) it will be timed so that no commercials are missed. Now you know who the TV stations are looking out for. ;-)