Slashdot Mirror
Water Main Break Floods Dallas Data Center
miller60 writes "IT systems in Dallas County were offline for three days last week after a water main break flooded the basement of the Dallas County Records Building, which houses the UPS systems and other electrical equipment supporting a data center in the building. The county does not have a backup data center, despite warnings that it faced the risk of service disruption without one."
There should always be duplication of critical components of a system with the intention of increasing reliability of the system, usually in the case of a backup or fail-safe.
Some of my favourite people are from th US; Vonnegut, Chomsky, Bill Hicks.
Sometimes it's cheaper to deal with it when it happens than to take precautions.
The county does not have a backup data center
Traditionally, facilities with that level of management oversight don't even have backups. No not backup centers or facilities or hardware, I'm talking about backup tapes. Am I right or wrong?
I know rotating hard drives don't like immersion. Are SSDs any better, or do they die from galvanic corrosion? It might be an interesting race, which will survive longest.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Not in the cloud
At least it wasn't a sewer line break, that would be real shitty.
The county does not have a backup data center, despite warnings that it faced the risk of service disruption without one.
That alone spells disaster. If I had a nickel for everything that could go wrong in a data center, I could buy a new data center.
Next question. Has the backup and restore process been tested?
Any sufficiently unpopular but cohesive argument is indistinguishable from trolling.
About a year ago...
At the time, we had all our WLAN connections carried through Bell Canada VIA Frame circuits. I guess many of these circuits went through a facility in Edmonton. This facility was being rennovated, and some poor worker drilled through a pipe that they thought was empty... As it turns out, that pipe was filled with pressurized water, and so the water started spraying everywhere/everything and ended up taking down all our frame services north of Edmonton (about 30 sites). It took about 2 (very stressful) days for Bell to route our frame circuits through another data center.
It sucked, but I really feel bad for the poor guy that drilled through the wrong pipe.
Copyright 2010. All rights reserved. This comment may not be copied in any way including, but not limited to caching.
Roof -> generators Above water level (1st floor and up) -> Electrical stuff Basement -> rats and other Texas style vermin ??? profit
Whose bright idea was it to put the UPS and backup systems in the portion of the building that is first to be flooded, and the most devastated in just about any natural disaster, AND the least accessible afterward? Sounds like something a government would do....
Tequila: It's not just for breakfast anymore!
In a state as blessed as Texas, they were told that God would provide protection against acts of God. I imagine many of the faithful are confused, especially when Jesus day is only a few days away.
Maybe they didn't execute enough retarded people this year?
http://en.wikipedia.org/wiki/Jesus_Day
Maybe they were more worried about tornadoes than floods there.
Of course the police/para/military mindset is bunker/underground everything critical as much as possible as well. I know my county just built a new jail/police station a few years ago and more than half that sucker is underground, you can see it from the road so it was easy to see while it was being constructed. It's built more like some fort than what you might think of as a traditional jail or copshop facility. Well sort of, it is half underground now, what they did was build it on level ground, the first level, then they bermed that over and built the next level you can still see now on top of it.
Data center floods, Katrina, and the BP oil spill the Gulf. All have one thing in common - a government that was not prepared.
I chuckle when I hear my more conservative friends complain about the level of competence or disaster response times we find in government. These are the very same people who want less government and lower taxes.
You can not have a small, cheap, government that is staffed by geniuses and prepared for every possible problem. Smart people cost money, resources cost money.
While this flood was an avoidable occurrence, do taxpayers really want to pay for redundant EVERYTHING at the local, state, and federal government? Probably not.
-ted
In what world do you live in that the government is ultimately responsible for a massive oil spill that was caused by a corporation?
While the government may be responsible for the fact that the regulations weren't followed every step of the way, the company that is behind the actions on the rig are ultimately fully responsible for not following those rules and regulations. I'm glad that the government can help in any way possible, but BP should foot the bill.
It's entirely possible that the reason they're complaining is that they want a smaller government with less things they're responsible for, so they'll have fewer things they're capable of screwing up, and want the things that government does taken over by private industry. Having a larger government does not necessarily mean having a more prepared government - it generally means having more layers of bureaucracy and more people who's jobs are not directly tied to their performance.
While I don't promote handing over the control of our jails over to private industry, you could certainly hand over the server management to people who actually know what they're doing and have a vested interest in making sure that there's a stable, solid datacenter behind the servers. You could get rid of some government sysadmins, and it'd be cheaper than buying a brand new datacenter as well.
Well sirs I have this here deed that says I own the Cowboys Stadium. What do you mean it looks like I typed it up? You got any proof I don't own this here stadium? OK Then. lol can see it now.
"Computers are a lot like Air Conditioners" "They both work great until you start opening Windows"
So, when that joke went over your head, did it make a WHOOSH!, or was it more of a buzzing sound?
Information theory is life. The rest is just the KL divergence.
From the movie Contact: "First rule in government spending: why build one when you can have two at twice the price?" Load balanced and replicated no less. Basement + critical systems = bad idea. Not only for flooding potential, but bad sewer systems as well. Yes been there done that. Curious, is there a systems engineer that could make a good argument for building data center infrastructure in a basement. Two points already for shielding from severe weather; anything else?
Jesus, is that you?!?!
Some of the replies are saying the basement is a fine place to put these things, I can appreciate that. I'm wondering if this is less a problem of basement and more a problem of sewer. I don't know about Dallas, but I live in Houston, and Houston has the most terrible sewers I've ever seen. Storm drains are located very far away from each other, and the amount of space underground for carrying water seems to be miniscule. Once a year or so, we get just a day's worth of heavy storms which result in flash flooding - which seems like it would have all just washed into the drains in any other major city I've ever been in. Maybe the greater issue here is whether or not those basements were properly protected by drainage to keep the waters out.
Libertarians somehow believe that private businesses should be stronger than governments but weaker than individuals.
money != productivity"
Confidentiality? Check! ... Can I borrow a pen?
Integrity? Check!
Availability?
With reasonable men I will reason; with humane men I will plead; but to tyrants I will give no quarter. -- William Lloyd
Tell that to Bobby Jindal.
It doesn't hurt to be nice.
That's where all the Kennedy Assassination records were stored!
Their they're doing there hair.
Having a larger government does not necessarily mean having a more prepared government - it generally means having more layers of bureaucracy and more people who's jobs are not directly tied to their performance.
And this is different from larger industry how?
It doesn't hurt to be nice.
I was at a $BigSwitchMakerResearchSite and they located their servers on the second floor of a 3 story building. They had a flood in the area in the past which wiped out their first floor data center. Since this place was also in tornado country some one made a sane and sensible decision to locate all the important servers on the second floor away from the outer walls.
Take heed all you junior sysapes.
putting the 'B' in LGBTQ+
What your conservative friends are objecting to is a massively bloated government that appears to be completely inept at EVERYTHING regardless of how much money we give them.
If they can't be prepared with the tens of billions that they spend NOW what makes you so sure that they would do any better with additional billions?
That is a very false false dichotomy... and a very poor argument.
Your basic argument is that people who favor small government expect the government to do nothing.
It's like talking about small government, and someone says: don't you like safe food inspections?
Yes, I 'like' those things. I also don't mind the government doing them. That's why I believe in small government. Not no government.
Just take a look at the government's spending. The things 'small government' folks want the federal to do would cost next to nothing.
Our biggest costs are healthcare, military...
So go ahead and cut those things down to focus on small government. It will free up hundreds and hundreds of billions of dollars.
Then, I'd be more than happy if the government spent what it costs to have proper data centers.
While the government may be responsible for the fact that the regulations weren't followed every step of the way, the company that is behind the actions on the rig are ultimately fully responsible for not following those rules and regulations. I'm glad that the government can help in any way possible, but BP should foot the bill.
Wanna bet? Seriously. If you really believe that (and a brief review of the history will clearly show how "responsible" the oil industry has been when it comes to their messes), I'll even offer generous odds.
First, this is Dallas County, not Dallas city.
Second, they knew about the potential for failure and were working on setting up a backup data center. TxDOT denied them rights of way to lay fiber along the highway into a facility in Tarrant county, so they were looking at other potential sites in Garland. Unfortunately this happened before they got it all resolved.
TxDOT might have had good reasons for denying the request, I don't know, but I would wager that the backup site would be a lot further along if they had been able to run that fiber. Sometimes you know there is a problem, management agrees, and you even have a budget to fix it... but someone else (another department, another company, a government agency, etc) stands in the way.
Natural != (nontoxic || beneficial)
They nice people from a regional http://en.wikipedia.org/wiki/Fusion_center will have all the data backed up for any state database. eg http://www.fusionsystem.us/
Staffed by IT experts eg the NSA, they all have clearance and just love to take good care of all of your regional data 24/7.
Been state and federally funded they will have fancy off site backups in real time.
Many snapshots of the data will be flowing around the USA at any time.
Domestic spying is now "Benign Information Gathering"
Curious, is there a systems engineer that could make a good argument for building data center infrastructure in a basement. Two points already for shielding from severe weather; anything else?
Well, I'm not a systems engineer (studying to become a software engineer) and frankly have no idea what I'm talking about here. I do have a few guesses, though. (Feel free to educate me if I'm completely off)
Fires. If a fire breaks out in a building, I would guess that basement is the part most likely to survive. Collapses. If a building collapses due to some structural flaw, a bomb, whatever, I guess that stuff in the basement would have the highest chance to survive. Security. A basement might be easier to secure against burglars, etc. than the ground floor, for example? Cabling. I guess that a lot of cables (be they network, electricity or whatever) run underground. If that is the case, your access to them might be more secure in the basement than higher up in the building (for example, some of them might accidentally be damaged if lower floors are being renovated... or something.) Cost. As somebody else commented, room in the basement is likely to be more affordable. Easier to seal. I've only ever been to one small datacenter but the fire extinguishing system there worked by filling the place with gas that would suck out all the oxygen (I was told that if a fire would break out, I should run. Fast.) so I guess that the enviroment should be somewhat sealed for that to work efficiently? I think that basement is good for that.
Ok big shot. I'll bite.
What happens when BP cannot fix the problem? We take all their assets and then.... we have all ze money?
We'll still have a whole gulf full of oil. Might as well fence off Florida to Mexico and just have a big oil swimming pool.
> ...they're trying to get 270% raises and to double the length of their terms.
We can't know if this is a good thing or a bad thing without a lot more information. If they're one-year terms, for example, that could be a problem because it intensifies the election cycle and the slave-to-polls problem--or it could be a good thing, if they're so corrupt that they routinely make horrible decisions as soon as they get into office, and you need to keep them in check.
Likewise with the 270% raises. Congress and the Supreme Court both operate with staffs that are too small do do what they should be doing. I don't know why the court does it--maybe just because it always has. But the Congress does it because it would be politically unpopular to increase their staff or office space. But the job Congressional staffs do has exploded in the last twenty years or so, (looking at the volume of constituent concerns they examine and attempt to reply to), while staff has stayed constant. The result is far less attention to each voter, more automation, more mistakes, and less responsiveness.
In the context you're discussing, a 270% raise may be politically unpopular, but it may make sense if the City Council is heavily underpaid, as compared to other reasonable Government salaries, given the work that they do.
-- IANAL, this isn't legal advice, and definitely isn't legal advice for you. Also, Squee!
http://en.wikipedia.org/wiki/Infinite_monkey_theorem
I would bet a few geniuses would work for less than an infinite amount of monkeys would...
Your point would be more meaningful if disaster preparation were a significant portion of the federal budget that them silly conservatives complain about, but it's not.
There are certain limited number of roles the government is suited to. Faithfully fulfilling those roles is what this conservative wants.
There are a vast number of tasks the federal government takes on that it is structurally incapable of handling in any intelligent way. You can't turn a screw with a hammer, yet we spend hundreds of billions of taxpayer dollars trying to just about do that. This conservative wants to see those roles, and those expenditures, eliminated.
Responding to certain types of disasters is something the federal government could competently and legitimately address. I would be all for funding such things.
Alcohol, Tobacco and Firearms should be the name of a store, not a government agency.
Well, they say a rising tide lifts all boats. I guess the same isn't true for racked servers.
I prefer rogues to imbeciles because they sometimes take a rest.
Yes, I 'like' those things. I also don't mind the government doing them. That's why I believe in small government. Not no government.
And when you add up all the small things and hand them over to the government, you get a big government. The problem is people want a government of their preferences. "Small" to you includes food inspections, but probably not the FDA's weaponized anthrax research. Others might think that a small government wouldn't include food inspections, but encouraging people harmed by bad food to sue. Others will think that lawsuits (a government action enforced at the end of a gun) should be capped and people should be responsible for themselves. And some think that a "small" government would include FDA research into weaponized anthrax in order to protect our livestock from a biological attack. So get 10 people who believe in "small" government to agree to what that means, and I'll think you have a point. Otherwise, "small" means to me "big but only big in what I personally like" because that's what it ends up being.
Just take a look at the government's spending. The things 'small government' folks want the federal to do would cost next to nothing. Our biggest costs are healthcare, military...
Well, the "small government" Tea Party starting with the goal of being a small government party doesn't list reduction of the military on their list of to dos. In fact, though a massive chunk of our budget is military, only the liberal Democrats ever cut it. The Republicans grow it every chance they get. And the Republicans claim to be conservative, and the "real" conservatives like the Tea Party don't put reduction of the military on their top 10 list. That, and the military is one of the few powers the feds use that they actually have explicitly listed.
Learn to love Alaska
I think a better solution would be for Congress to do far less. I'm not some small government Republican, but I do think they have a point at times. The fact that you could spend 10 lifetimes and still not read all the laws that have been passed in this country should be an indicator that something is wrong with the system.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
The Canadians required that the relief well be drilled simultaneously with the primary well when they allowed drilling in the arctic, the fact that the US didn't have similar regulations in place is a direct contributor to the current situation of waiting 4 months for the problem to be stopped.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Not really.
Everyone is going to have a different opinion on what 'big government' is.
Just like everyone has an opinion on what 'pornography' is.
But I guarantee you you will get no argument on 'what is porn' if you show someone real hardcore porn on any of the porn sites on the internet.
Show someone art or maybe nudity in film... and you will have debate.
Yes, I don't mind the government doing regulation. Some might.
I also don't mind a military to defend the country. Some might.
But show any small government person a government that monopolizes and provides healthcare and education... and they will tell 'IT'S BIG GOVERNMENT'
All the little things don't really add up to much when you compare it to the spending big government requires in terms of entitlements, wars of choice...
The article seems to state that the delays were fairly crippling to the justice system. It didn't sound like a "we'll just record everything on paper for now" situation. Rather, people couldn't access records until the systems came back up as most paper records had been tossed in the transition to electronic filing. Honestly, if it's just a power system, that's all the more reason for a back-up system. The logistics of storing a few generators off-site that you can truck over and set up in a few hours is trivial. A 3-day shutdown of a data center is unacceptable when the servers and data are all untouched.
But show any small government person a government that monopolizes and provides healthcare and education... and they will tell 'IT'S BIG GOVERNMENT'
And some will be happy with education. The studies show that federal education (usually targeted for head start programs and such) usually saves money. Someone wanting to spend less government money will pay for education to save 10 times than in welfare, law enforcement, and prison costs. I haven't seen the breakout without welfare in there, but it is still cheaper to spend a little money on headstart than the enforcement costs later if you don't spend the money. So if the goal is the smallest government, you should be supporting the federal education expenses, not complaining about them.
What I find is that many "small government" people are looking to cut funding of services that serve the poor, regardless of whether they are actually saving money. Given that minorities are disproportionately represented in poor, it's been suggested that it's racial in basis, but it could just be the "they should just make more money/work harder" thought on being poor that they get what they deserve.
Learn to love Alaska
And yet whenever they get into power they deliberately run the government into the ground and install corrupt incompetents wherever they can, bloating the bits that make them money massively.
As soon as they're out of power, they point to the failures they introduced and caused and claim that government is bad and needs to be cut. Well, yes, government is bad and incompetent if you go out of your way to make it so to score political points and get cushy jobs for your friends and massive corporate welfare (in the form of tax breaks, subsidies, bullshit contracts and removing inconvenient safety regulations) for your donors.
Both sides fuck up, but one goes out of its way to ruin the government so it can claim that government is bad.
(at which point the conservative friends say that the republicans don't represent them at all, they're true conservatives, and then vote for them anyway, because they're lying, dishonest scum)
See! Proof there is no god or God.
Yes, we have too many laws and there are many things Congress should do differently, and some things that should be done at a state level so that states can operate as "laboratories of democracy" (i.e. evolutionary learning of the best approaches to societal problems). But the increase in workloads doesn't come from that, it comes from the fact that the number of constituents asking for help and writing to explain their views has gone way up, as has the number of constituents in each district.
-- IANAL, this isn't legal advice, and definitely isn't legal advice for you. Also, Squee!
From TFA: "The 90-year-old water main ruptured..."
90 years old because taxpayers (read: well-heeled conservatives) never want to pay for maintaining and replacing infrastructure until after the disaster occurs. No doubt they will somehow get federal funds to help defray the costs - all the while cursing the federales' very existence.
My company is currently having a "independant consultant" look at our IT infrastructure and he is dead set on us moving our production systems out of our building and into a datacenter. He keeps giving us the "what if there was a disaster and the fire marshall won't let you have access to your building". Uhm, ok, what if there is a flood in the datacenter and there is no power...
Are you sure you're a conservative? Most conservatives argue that the safety nets and "social contract" proposed by the New Deal, which meant that the government could and would step in and help people out of a natural disaster, were wrong. Most libertarians and conservatives believe that the government has no business helping people in a natural disaster. If people didn't buy private insurance from a private company, why should the government help them out?
"When the president does it, that means it's not illegal." - Richard M. Nixon