Slashdot Mirror

For years, I co-located at the top-rated 365 Main data center in San Francisco, CA until they had a power failure a few years ago. Despite having 5x redundant power that was regularly tested, it apparently wasn't tested against a *brown out*. So when Pacific Gas and Electric had a brownout, it failed to trigger 2 of the 5 redundant generators. Unfortunately, the system was designed so that any *one* of the redundant generators could fail and there wouldn't be any problem.

So power was in a brownout condition, the voltage dropped from the usual 120 volts or so down to 90. Many power supplies have brownout detectors and will shut off. Many did, until the total system load dropped to the point where normal power was restored. All of this happened within a few seconds, and the brownout was fixed in just a few minutes. But at the end of it all, there was perhaps 20% of all the systems in the building shut down. The "24x7 hot hands" were beyond swamped. Techies all around the San Francisco area were pulled from whatever they were doing to converge on downtown SF. And me, 4 hours drive away, managed to restore our public-facing services on the one server (of four) I had that survived the voltage spikes before driving in. (Alas, my servers had the "higher end" power supplies with brownout detection)

And so it was a long chain of almost success of well-tested, high-quality equipment that failed all in sequence because real life didn't happen to behave like the frequently performed tests did.

When I did finally arrive, the normally quiet, meticulously clean facility was a shambles. Littered with bits of network cable, boxes of freshly-purchased computer equipment, pizza boxes, and other refuge were to be found in every corner. The aisles were crowded with techies performing disk checks and chattering tersely on cell phones. It was other-worldly.

All of my systems came up normally; simply pushing the power switch and letting the fsck run did the trick, we were fully back up and all tests performed (and the system configuration returned to normal) in about an hour.

Upon reflection, I realized that even though I had some down time, I was really in a pretty good position:

1) I had backup hosting elsewhere, with a backup from the previous night. I could have switched over, but decided not to because we had current data on one system and we figured it was better not to have anybody lose any data than to have everybody lose the morning's work.

2) I had good quality equipment; the fact that none of my equipment was damaged from the event may have been partly due to the brownout detection in the power supplies of my servers.

3) At no point did I have any less than two backups off site in two different location, so I had multiple, recent data snapshots off site. As long as the daisy chains of failure can be, it would be freakishly rare to have all of these points go down at once.

4) Even with 75% of my hosting capacity taken offline, we were able to maintain uptime throughout all this because our configuration has full redundancy within our cluster - everything is stored in at least 2 places onsite.

Moral of the story? Never, EVER have all your eggs in one basket.

Of COURSE there are people onsite. Most likely they have anywhere from a dozen to a hundred people onsite. But what's that going to do for you in the case of a large-scale problem?

The otherwise top rated 365 Main facility in San Francisco went down a few years ago. They had all the shizz, multipoint redundant power, multiple data feeds, earthquake-resistant building, the works. Yet, their equipment wasn't well equipped to handle what actually took them down - a recurring brown-out. It confused their equipment, which failed to "see" the situation as one requiring emergency power, causing the whole building to go dark.

So there you are, with perhaps 25 staff a 4-story building with tens of thousands of servers, the power is out, nobody can figure out why, and the phone lines are so loaded it's worthless. Even when the power comes back on, it's not like you are going to get "hot hands" in anything less than a week!

Hey, even with all the best planning, disasters like this DO happen! I had to spend 2 wracking days driving to S.F. (several hours drive) to witness a disaster zone. HUNDREDS of techs just like myself carefully nursing their servers back to health, running disk checks, talking in tense tones on cell phones, etc.

But what pissed me off (and why I don't host with them anymore) was the overly terse statement that was obviously carefully reviewed to make it damned hard to sue them. Was I ever going to sue them? Probably not, maybe just ask for a break on that month's hosting or something. I mean, I just want the damned stuff to work, and I appreciate that even in the best of situations, things *can* go wrong.

So now I host with Herakles data center which is just as nice as the S.F. facility, except that it's closer, and it's even noticably cheaper. Redundant power, redundant network feeds, just like 365 main. (Better: they had redundancy all the way into my cage, 365 Main just had redundancy to the cage's main power feed)

And, after a year or two of hosting with Herakles, they had a "brown-out" situation, where one of their main Cisco routers went partially dark, working well enough that their redundant router didn't kick in right away, leaving some routes up and others down while they tried to figure out what was going on.

When all was said and done, they simply sent out a statement of "Here's what happened, it violates some of your TOS agreements, and here's a claim form". It was so nice, and so open, that out of sheer goodwill, I didn't bother to fill out a claim form, and can't praise them highly enough!

Craigslist runs at the 365 main colo.

365 Main has placed a statement about their Hitec UPS failures on their web site. Highlights:

• Generator 1 detected a problem in its start sequence and shut itself down within 8-10 seconds.
• After initial failure, Generator 1 attempted to pass its 732 kW load to Back-up 1, which also detected a problem in its start sequence.
• After Generator 1 and Back-up 1 failed to carry the 732 kW, the load was transferred to Back-up 2 which correctly accepted the load as designed.
• Generator 3 started up and ran for 30 seconds before it too detected a problem in the start sequence and passed an additional 780 kW to Back-up 2 as designed.
• Generator 4 started up and ran for 2 seconds before detecting a problem in the start sequence, passing its 900 kW load on to Back-up 2. This 900kW brought the total load on Back-up 2 to over 2.4 MW, ultimately overloading the 2.1 MW Back-up 2 unit, causing it to fail. Generator 4 was manually started and brought back into operations at 2:22 p.m. Generator 4 was switched to utility operations at 7:05 a.m. on 7/25 to address an exhaust leak...
• Generators 2, 5, 6, 7 and 8 all operated as designed and carried their respective loads appropriately.

So they apparently had startup failures in four out of ten Hitec units. Their basic architecture is that they have eight main UPS systems (each is a motor/generator/flywheel/Diesel combo), each driving a separate section of the colo, and two spares, Backup 1 and Backup 2, which can be switched to drive any section. No big battery banks; it's all flywheels and Diesels. At least eight systems must be running to keep the full data center up.

365 Main has had Hitec experts flown in from Holland, where the UPS was made. Today, Hitec top management arrived: "A longstanding member of the Hitec Board of Directors is arriving later tonight and will be onsite tomorrow (Sunday) to participate in all investigation activities."

http://365main.com/press_releases/pr_5_30_07_green .html

Going green! Maybe they were using eco-conscious fuel, or they had started making power saving "eco friendly" decisions and modifications. Nasty smoky diesels! Can't have that in our new green image!

Heh!

The press release "RedEnvelope Reports Two Years of Continuous Uptime at 365 Main's San Francisco Data Center", which was on the 365 Main web site earlier today, has disappeared from there.

But they sent the press release to PR Newswire, and you can still read it there.

Well, according to their self-congratulatory press release, issued earlier today, they were allegedly at 100% uptime for the past two years.

The irony of issuing a press release like that, and then to be hit with a power outage and apparent simultaneous failure of all backup systems later that day, is beyond measure.

I don't know about God, but it's enough to make me believe in karma. ;-)

I biked past the place twice a day for years- they rehabbed and prepped the building up as a datacenter just in time for the dot.com to crater. It was left cold for a few years, but then there were a spat of articles in the local press, talking about the cheap hosting deals being offered, and of the incredible redundancy built into the the place in case of disaster. They've promised a lot, over the years, and whatever they cause may be, it really looks like they failed to deliver.

I warned these idiots eight months ago during my review that the datacenter had outgrown its generator capacity.

According to the San Francisco Facility Data Sheet, 365Main has "Ten 2.1-megawatt Hitec Continuous Power Systems ("CPS") designed to N+1 redundancy", "Three 20,000 gallon double-lined fuel tanks", and non-interruptible power-circuits up the wazoo.

They should be able to survive a multi-day power outage. At conferences, 365Main sales people talk about how the datacenter could remain self-sufficient if a dirty-bomb exploded in the neighborhood--- meaning no fuel or water trucks for days, few staff may enter or leave the facility, etc.

Data sheet for 365 Main:

The company's San Francisco facility includes two complete back-up systems for electrical power to protect against a power loss. In the unlikely event of a cut to a primary power feed, the state-of-the-art electrical system instantly switches to live back-up generators, avoiding costly downtime for tenants and keeping the data center continuously running.

They use a Hytec Continuous Power System, which is a motor, generator, flywheel, clutch, and Diesel engine all on the same shaft. They don't use batteries.

With this type of equipment, if for some reason you lose power and the generator doesn't start before the flywheel runs down, you're dead. There's no way to start the thing without external power. Unless you buy the optional Black Start feature, which has an extra battery pack for starting the Diesel. "Usually the black start facility will not be often needed but it won't hurt to consider installing one. Just imagine if you were unable to start up your UPS system because the mains supply is not available.". Did 365 Main buy that option?

This has got to be some type of joke: RedEnvelope Reports Two Years of Continuous Uptime at 365 Main's San Francisco's Datacenter.

It was released today....

They do have backup power systems in place. Ten 2.1 MW "Continuous Power Systems" according to this document. I wonder how close they were to guaranteeing 99.99 percent uptime this year...

5 minutes a year is nearing "five nines" for reliability (and you don't want to rely on the power supply being your only source of downtime in that situation.) I'm not sure if their customers have "99.999% uptime guaranteed" in their contract, but if so, I'm sure they did have their tanks in working order. Some old press releases of theirs are touting 100% uptime.

I realize that a press release from 2004 is hardly relevant, but this is slashdot... so here is a choice paragraph:
By surpassing the five-nines milestone, 365 Main further establishes itself as the go-to facility with the necessary investments to ride through any worst-case scenario. Equipped with ten 2.1 Megawatt continuous power systems, an N+2 or greater facility-wide redundancy, an award-winning base isolation system, and 60,000 gallons of fuel on-site, 365 Main's structural resiliency is unmatched by any other data facility in the region. 365 Main's customers continued to perform at 100% during one of the Bay Area's largest power outages of recent history. 365 Main's customer data remained online, accessible, and secure.

Press Release on Red Envelope having 2 years of uptime at 365 Main - San Francisco from today: http://365main.com/press_releases/pr_7_24_07_red_e nvelope.html

I just don't get it. I mean, I really, REALLY don't get it?!?!?!

Places like 365 Main offer top-notch server hosting for dirt-cheap prices. I have a half-rack there with 6, quad-core Opteron clustered LAMP servers in place now. Reliability is excellent, bandwidth availability is fabulous (we have a Gb interface to the Internet) and the price is just astonishingly cheap - although we are an "Internet Company", we spend more on phone calls than we do on hosting and related fees. Never mind hotels and travel/flight expenses!

We've gotten between 4 and 5 nines of uptime over the last 4 years just by using quality software, (Linux software stack) good quality (but generic) hardware, and a good quality, outsourced hosting environment, at a price you simply couldn't believe.

I've seen plenty of other companies roll their own datacenter at 3-10x the cost, with greatly reduced reliability, embarassing outages, and lots of internal friction over maintenance overhead. Me? I want it cheap, fast, and reliable. It really IS a case of cheaper is actually the best!

I can see the need for an "internal" datacenter if your needs are largely local and your bandwidth usage is great - think enterprise application availability, or large volume local network storage. But if your company's core deliverable is public facing, I can't imagine any advantage to having your own datacenter until you company income passes the "100 million/year" mark.

And yes, I'm the CTO of a million-dollar per year Internet services/software company, growing at about 50% annually. (We'll probably be close to the 2 million-dollar-per-year mark by Christmas)

Uh, any respectable colo will have those - along with stockpiling enough fuel to run those generators. Even in downtown san francisco high end colocation centers have to have those abilities.

$64, 10.7GB disc, 264GB, $0 domain, shell, MySQL, cron, RoR, cvs. [dreamhost.com]

Uh, for $49 I can get 80GB on a dedicated server from various places -- with the UPS & power capabilities I described. (though I admit it's awesome you guys have RoR pre-instaleld)