Slashdot Mirror
Power Problems Force Seattle To Throttle City Data Center For Days
Nerval's Lobster writes with an except from sister site SlashDataCenter: "On Aug. 23, Mayor Mike McGinn of Seattle informed residents that the city would partially shut down its municipal data center for five days including the Labor Day weekend. As a result, city residents will be unable to pay bills, apply for business licenses, or take advantage of other online services. In a Webcast press conference, McGinn isolated the issue as a failure in one of the electrical 'buses' that supplies power to the data center. Because that piece of equipment began overheating, the city had to begin taking servers and applications offline to prevent overloading the system. The maintenance will cost the city $2.1 million of its maintenance budget. A second power bus will remain operational, supplying enough electricity to power redundant systems for critical life and fire safety systems, including 911 services and fire dispatch. The city's Web sites should also be up and running in some capacity."
That should help the situation.
Certainly there are some things government can do more effectively in-house, and Seattle should be lauded for the effort. However, the effort failed: It clearly can't do it more effectively (when was the last time a well-run private datacenter was offline for five days barring a natural disaster?) and Seattle would be wise to consider decommissioning their municipal data center, and other governments would be wise to heed this example before launching on their own datacenter projects.
While looking at the prospect of moving to Seattle, I've read repeatedly read that the city is in political gridlock and seems totally unable to get any meaningful long term infrastructure additions put in place despite wide support for them. It seems to me this is the case in many cities out there, but can anyone say what it's like in their city?
I suspect that the first ones to finally get something useful done (rather than just repaving a few highways) will be the ones to reap a lot of growth when this recession finally begins to fade away.
-
FTFY karma whore.
Interesting that this is not on the front page of the Seattle Times. In fact, I can't find it at Washington's biggest paper at all.
If you want news from today, you have to come back tomorrow.
If you lived in podunk nowhere then no probably not, if emergency services continue to operate it wouldn't be a big issue. But for such a large municipality to go dark for 5 days...would definitely be impactful locally and possibly regionally/nationally to a smaller degree. Emergency services are very important but the business of government (no matter how i feel about it from time to time) needs to continue and serve it's people...I am sure (at least i hope) that they looked into portable power generation, but it seems that this is a poor solution. just my 2 pennies.
Chief Thinker www.devotedskeptic.com
If bills don't get paid, there better not be any late fees imposed. The banks could make millions on this.
“He’s not deformed, he’s just drunk!”
iCarly will be pissed.
This wouldn't have happened if they ran things "in the cloud".
Sounds like Seattle's 911 system is quite fragile.
"If any question why we died, Tell them because our fathers lied."
If power problems are downing the city's datacenter for a holiday weekend, couldn't they just rent a few $100/mo servers and run the city apps on them for the downtime and make the problems transparent to the end user? No one-place site is ever safe for important apps, we call that a Single Point of Failure around here.
I'm LostCluster but I lost my password to that user. Hey Slashdot, how about helping me get it back!
Seattle? The home of Amazon? Why on earth don't they just move their datacenter to Amazon Web Services? They could probably do it for less than the $2.1 million they're spending on this single part!
A slashdotter who didn't build his own computer is like a Jedi who didn't build his own lightsaber.
What I'm trying to figure out is why 911 and emergency services didn't have a separate offsite backup. I mean, how much more mission critical can you get than that? Everytime I see one of these articles I think to myself: Why are they mentioning this if there wasn't some risk of failure? And the answer is... because quite obviously, there was some risk.
I don't want my cause of death to be "Your call could not be completed as dialed. Please check the number and try your call again later..."
#fuckbeta #iamslashdot #dicemustdie
Long weekend for me.
Nice, so they're running their mission critical operations on reserve systems. Hope nothing too important happens while they're getting bombed by a /. post.
Why don't they just fail over the critical life and fire safety systems to the backup datacenter, and keep normal services up at the primary datacenter while they do the work? They do have a second site, right? Surely no one would host a system deemed "critical" and "life safety" at a single site?
Why is this the citys problem...
Sounds like the power company should suck it up and deal with it.
Unless i'm completely misunderstanding what they're talking about.
Maybe its all 'inside' city property... In which case..... Why the fuck was it designed in such a non fail resistant way? that just seems.... stupid.
Oh wait. Seattle... nevermind. carry on. 2.1 million. poof. good job city!
overheating power buses / wires are a fire risk and that comes from them being under sized for the load.
See the towering inferno to see where that can get you.
Seattle is 95% Democrat and has been for decades.
The remaining 5% consists of 'magic' voters, libertarians and socialists.
Most of whom cannot define the terms.
--
If you're guessing: I moved away from there.
No brain, no pain.
The datacenter is on the 26th floor of the municipal tower and the overheating bus runs up to that floor. The power company in question is municipally owned, either way it would be the city's problem.
McGinn had quite a few facts wrong in the press conference. The equipment is working fine now and the overheating only caused a minor amount of downtime. The major issue though was the backup generator never kicked in because as it turns out, the electric starter for the diesel generator is connected to the same bus. Labor Day weekend was then choosen to fix this majorly obvious design deficiency.
McGinn is a well-known Gaia,fag AKA mud-hut-GREEN from the nations prime monopoly protected city of effete Nimbyites. He is famous for wanting street-clogging bike-riders to generate city-power by microwave transmission. No electricity sez hizhonor no problemo .. pass the $150 salmon dinner please.
If you've got enough compute resources to have your own data center, it's probably cheaper for you to have your own data center instead of paying someone else to do it for you. So then, you build your own data center, and you decide on compromising on certain things like power bus redundancy, because in any given data center environment there are a million things that can fail, but you have to prioritize the systems you make redundant by looking at their failure probability and expected failure impact. You can't make everything redundant. That would be foolish because you'd spend so much money on redundancy that you'd have no money left for functionality. You probably want redundancy for routers, switches, servers, storage, etc because that's all stuff that's likely to fail, but I'd bet most single tenant datacenters probably don't have power grid redundancy because that's really expensive, and not as likely to fail. You would probably be better served by staying on a single power grid and putting the money to bring another power connection in on a generator instead.
The point I'm trying to make is that there is a level at which you have to say something is "redundant enough". I think that call that the point of diminishing returns. I would say an overheating power bus is probably an acceptable failure because I would've considered that as something that has a pretty low failure risk, so I wouldn't have spent the money to have two of them.
Remember I'm talking about a single tenant, privately owned datacenter for a small entity here. I.E. a municipality that probably has somewhere between 100-500 servers. Naturally if you're a company that is in the business of doing business online, or a huge company, then this isn't the right path for you. At the end of the day, a municipality offers online services as a convenience for its residents. When the DC blows up, you can still write a check and drop it in the mailbox to pay your water bill.
When you figure out your critical services, you separate them and define another SLA which applies only to them. And from the article, it sounds like that is exactly what they did - they kept the critical life safety systems running and took down the convenience systems for an acceptable period of time. So what's all the fuss about?
The $2.1 million figure is to reconfigure/update the data center for future operations; it is NOT the cost to repair the building's electrical bus.