Slashdot Mirror
Multiple Sites Down In SF Power Outage
corewtfux writes with word of a major outage apparently centered on 365 Main, a datacenter on the edge of San Francisco's Financial District. Valleywag initially claimed that a drunken person had gotten in and damaged 40 racks, but an update from Technorati's Dave Sifry says the problem is a widespread power outage. Sites affected include Technorati, Netflix (these display nice "We're Dead" pages), Typepad, LiveJournal, Sun.com, and Craigslist (these just time out).
I can verify that it affected much of the Financial District here in SF. We had the power go out 3 times. Seems to be back now. Haven't heard any explanation yet.
They probably just didn't kick in. Had the same problem at Internap in Seattle a few years ago. Power was cut to the building and the UPSs failed to switch over.
Does this mean backup generators have failed or is the fault somewhere outside the datacenter? Time to start shopping.
Quack, quack.
It's interesting that so many major sites would go down in a local power outage? Are they all sharing one data center in SF? If so, why don't they have co-locations in other cities?
~ I am logged on, therefore I am.
This has got to be some type of joke: RedEnvelope Reports Two Years of Continuous Uptime at 365 Main's San Francisco's Datacenter.
It was released today....
ÕÕ
No kidding. years ago in my former job on traffic systems we had a great UPS with a generator on site and the ability keep it fueled up indefinitely. A security contractor came in on the weekend to install something and tried to wire up a new circuit hot. He slipped with a screwdriver and shorted the white phase to the chasis of the breaker panel. I don't think the tip of the driver actually touched ground, but the burn mark is still there to show how close he got.
The resuting current spike blew the 100A fuses (heavy metal strips) both going in to and out of the UPS. With the UPS effectively broken the generator set failed to start and the system gracefully shut down 40 minutes after the incident. Thats not bad. The batteries were only specified to work long enough for the genny to settle at 50Hz.
In the process of blowing the fuses a spike got back into the power supply of one of our DEC Alphas and took out the power supply. The system was redundant at the software level so I didn't notice immediately.
The UPS guy came out and didn't have enough fuses to replace the blown one, but we found that with a bit of brute force and filing attacks some others could be made to fit.
Please type the word in this image: problems
http://michaelsmith.id.au
Data sheet for 365 Main:
The company's San Francisco facility includes two complete back-up systems for electrical power to protect against a power loss. In the unlikely event of a cut to a primary power feed, the state-of-the-art electrical system instantly switches to live back-up generators, avoiding costly downtime for tenants and keeping the data center continuously running.
They use a Hytec Continuous Power System, which is a motor, generator, flywheel, clutch, and Diesel engine all on the same shaft. They don't use batteries.
With this type of equipment, if for some reason you lose power and the generator doesn't start before the flywheel runs down, you're dead. There's no way to start the thing without external power. Unless you buy the optional Black Start feature, which has an extra battery pack for starting the Diesel. "Usually the black start facility will not be often needed but it won't hurt to consider installing one. Just imagine if you were unable to start up your UPS system because the mains supply is not available.". Did 365 Main buy that option?
For some of these sites they are a lot more central than you might realize. If they failed to build their systems with a secondary site in mind it can be near impossible for the "CTO" types to pony up the dollars for it later. The biggest issue I have seen that affects this is storage. Either they aren't using suitable SAN technologies, or they didn't put enough money behind the storage initiative to set up secondary site replication. I agree with you though. This is a problem that has been solved. Perhaps netflix thought - wth - if we go out for a few hours and people can choose their movies that's just tough luck.
Sun.com going down is a good example of someone totally screwing up. They have absolutely NO excuse. The others - maybe they can get away with it and we won't care. If Sun can't keep their own site up, how can I expect them to keep mine up?
I really feel for all the folks who have to deal with this outage; it's no fun at all!
A client of mine had a number of servers in a Sterling, Virginia data center managed by Verio/NTT. It's a good data center and seems to be well-run.
Last September, the data center experienced two complete power failures in the span of three days. To their immense credit, data center management was straight with customers about what had happened. For those who might be interested, their statements about the problem appear here.
My point? Make sure you know how to bring your systems back up from a completely cold start, and that you find a way to test this periodically. While we work to ensure that this sort of situation occurs rarely, the fact remains that these sorts of failures DO occur, and they're not as uncommon as the sales and marketing folks would like you to believe.
Phil
...until the commercial power fails and doesn't come back for days.
:o(
The only places I've actually seen the insane levels of backup that some would like is in some telco central offices. The one I was associated with the longest had eight-hour-plus battery backup and 8 days of fuel for the diesels. Some of our really remote microwave sites had 24 hour battery and 30 day diesel.
Of course one of those sites failed high up in a mountain range in a mid-winter storm (Tieton, 1978) when the commercial power failed, and the starter battery for the diesel froze. When one of the techs finally got there (after burying his Sno-Cat and walking the last couple miles), he had to chip ice off the steel door to get inside, where he was able to get the diesel started with a little "rewire" of one of the backup battery sets. Oh, his two-way radio also failed during his hike, since it was outside his snowsuit, and the lack of communication caused the company to start two more Sno-Cats and a helicopter in that direction.
The site was out for nearly six hours, IIRC.
Even the BEST designs are subject to failure.
--
Tomas
HP has a nice overview of building systems which can failover between widely distributed nodes called Designing Disaster Tolerant High Availability Clusters. It's a bit old, and is focused on ServiceGuard, but is still interesting.
I would say incompetence... Craigslist has been plauged by incompetence since they started and small problems turn into big problems and make their site completely unusable. Their decision to use ambiguous messages like "This posting has been Published" in their anti-spam fight has made their system unreliable. One only has to take a look at the help forum for indication that their admins really do not care about the reliability of the system and questions about the constant downtime and unreliable nature of the postings are answered with vague condescending responses from staff members. Postings say they are Published and in fact they never show up on the site. This has been going on for months now with no end in sight. I would say they need a few good systems engineers to fix what's going on, however, you would almost conclude that they enjoy and even relish the moments when their site is completely unreliable or offline for days at a time. It makes one wish of a day when a competent site with competent administration would come along to replace this type of environment.
true 'dat. Makes one wonder why not just relocate everything to Alaska or somewhere else that's cold as hell.
Speaking of energy costs associated with heat dissipation, I've alway been curious of a method that could produce energy from wasted heat- as does a solar panel from the sun.
Wrap that supercharged V8 in some energy producing heatwrap, instant hybrid and more horsepower. Run your processor's cooling fan off the energy produced from the excessive heat.
Someone please tell me I'm talking out of my ass, or worse, just gave away the next big idea that could have made me billions.
Waiting in line for checkin at 365 Main:
http://tastic.brillig.org/~jwb/dorks.jpg
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"