Slashdot Mirror


Water Main Break Floods Dallas Data Center

miller60 writes "IT systems in Dallas County were offline for three days last week after a water main break flooded the basement of the Dallas County Records Building, which houses the UPS systems and other electrical equipment supporting a data center in the building. The county does not have a backup data center, despite warnings that it faced the risk of service disruption without one."

50 of 230 comments (clear)

  1. Silly rabbit. by migla · · Score: 4, Insightful

    There should always be duplication of critical components of a system with the intention of increasing reliability of the system, usually in the case of a backup or fail-safe.

    --
    Some of my favourite people are from th US; Vonnegut, Chomsky, Bill Hicks.
    1. Re:Silly rabbit. by jo_ham · · Score: 5, Funny

      This is Texas - God is their backup solution.

      Every night they pray for no hardware failures.

    2. Re:Silly rabbit. by vlm · · Score: 4, Interesting

      There should always be duplication of critical components of a system with the intention of increasing reliability of the system,

      Yeah that might be the intent, but it only works if the combined reliability is higher than individual reliability. Transfer switches I'm looking at you! I have worked at numerous facilities with data centers, and inevitably the transfer switch is less reliable than either wall AC power, or the diesels (youch!). Yes I know exactly what I'm saying, that at every facility I've worked at, power reliability would have been higher without the transfer switch and the generators. But its politically incorrect as the rare wall AC power failure would be unacceptable unless we spent money on switches and gens. As long as you spend money on switches and gens, any low level of reliability is acceptable.

      Your mileage may vary, maybe coastie cities have less reliable power. Don't know.

      --
      "Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
    3. Re:Silly rabbit. by migla · · Score: 2, Insightful

      >And the funding comes from where?

      If Dallas having a data center is not that critical, then never mind. Otherwise, from taxes, where civilized civilization usually comes from. But actually I just put that first post there to spoil it for prospective first posters. It was a copy/paste from the start of the wikipedia article on redundancy(engineering).

      --
      Some of my favourite people are from th US; Vonnegut, Chomsky, Bill Hicks.
    4. Re:Silly rabbit. by Mordok-DestroyerOfWo · · Score: 5, Funny

      God is their backup solution.

      If this is true, shouldn't they have been prepared for a flood?

      --
      "Never let your sense of morals prevent you from doing what is right" - Salvor Hardin
    5. Re:Silly rabbit. by Anonymous Coward · · Score: 2, Insightful

      And the funding comes from where?

      A redundant source of income that is.

      Sadly, no matter how you design a system there is always a single point of failure. Just depends on how much you want to spend and what you want to risk going south.

    6. Re:Silly rabbit. by gyrogeerloose · · Score: 3, Funny

      Their mistake was preparing for the burning bush, from the oil spill, instead of the flood.

      They should have known better, having already passed off their burning Bush to the rest of the United States back in 2000.

      --
      This ain't rocket surgery.
    7. Re:Silly rabbit. by DigiShaman · · Score: 2, Funny

      Supposedly, they have an employee named Noah responsible for off-site backups. Ask him.

      --
      Life is not for the lazy.
    8. Re:Silly rabbit. by Anonymous Coward · · Score: 5, Interesting

      In Texas, we are proud of our faith.

      Which is incredibly funny, especially considering that pride is considered one of the deadly sins (along with lust, greed, gluttony, and several others that Texans are well known for). Apparently too many of you are too busy thumping your chests about how you are such good Christians to even stop to consider what it really means to be a good Christian.

    9. Re:Silly rabbit. by headhot · · Score: 2, Interesting

      Transfer switches are the bane of my existence. I work for a major MSO, and every site we have lost has been to a transfer switch problem. Equipment with 2 ps is the best solution, so you have have 2 sets of transfer switches, UPS, Generator, and Mains.

    10. Re:Silly rabbit. by jo_ham · · Score: 4, Interesting

      It is quite possible to take Christians seriously and still make jokes about them. Some of the most devout Christians I know make jokes about their faith and God all the time.

      One of the defining human characteristics is being able to laugh at yourself.

      I make similar jokes about Apple users, Linux users, Football fans, people who drink Mountain Dew, atheists, rock fans, sci fi nerds, people who watch reality TV...

      Humour is part of human character. You appear to be missing yours. Perhaps you should pray for it to return.

    11. Re:Silly rabbit. by Crudely_Indecent · · Score: 3, Informative

      This is Dallas, where the City Council is doing such a good job, they're trying to get 270% raises and to double the length of their terms.

      --


      "Lame" - Galaxar
    12. Re:Silly rabbit. by bertoelcon · · Score: 2, Funny

      You weren't supposed to send him back though.

      --
      Anything can be found funny, from a certain point of view.
    13. Re:Silly rabbit. by theshowmecanuck · · Score: 2, Funny

      They didn't care. Most of the data referred to Thomas Jefferson.

      --
      -- I ignore anonymous replies to my comments and postings.
    14. Re:Silly rabbit. by couchslug · · Score: 4, Funny

      "This is Texas - God is their backup solution."

      So much for that. I prayed to Allah for a flood.

      --
      "This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
    15. Re:Silly rabbit. by zippthorne · · Score: 2, Funny

      Word is he was out on his boat...

      --
      Can you be Even More Awesome?!
    16. Re:Silly rabbit. by AK+Marc · · Score: 3, Informative

      I know you meant to be funny, but the City of Dallas and the County of Dallas are distinct and unrelated governmental entities. You are complaining about the city, but the county is the one affected by the flood. Don't tarnish the bad name of the city with the bad name of the county.

    17. Re:Silly rabbit. by victorhooi · · Score: 4, Interesting

      heya,

      Haha, silly little boy...*grins*.

      Ok, firstly, the seven deadly sins is actually a Catholic device, and even then it's just a categorisation thing, it's not Biblical, from what I can tell. The majority of Texans are, from memory, Evangelical Christians? As a Christian, a sin is a sin is a sin - they're all bad, and we all do them, Christian or otherwise. In the eyes of God, they're actually all "equally bad", if that makes any sense. I'm not going to go into a lengthy discourse on why, but ask any Christian, and they'll be happy to help answer your question.

      Read the bible - there is only one unforgivable sin - and that is to blaspheme the Holy Spirit, such as the Pharisees did. Everything else is by definition, forgivable via Jesus. Also, put it this way, if you're worried you've committed the unforgivable sin, then by definition you haven't (I'm paraphrasing Larry Richards here).

      And in terms of proud of their faith...err, there's nothing wrong with that. In fact, according to the Bible, your God is actually one of the few things you're meant to be proud of, that and being saved from sin by Jesus, and all that. The Bible repeatedly tells you to be proud about your God, the living God etc. etc.

      So your feeble attempt to accuse Christians of not knowing their faith just fell flat on it's face.

      Now, as a Christian, if you were to accuse us of other things, I'd be happy to entertain you, and indulge in self-reflection. However, please use an actual valid point if you're going to try to do that. We've done some questionable things in the past, and I'm sure we'll continue to stumble, and pick ourselves up, going forward. The one thing that differentiates us if that we're saved via Jesus, and that's an external thing.

      So I don't think Texans (or Christians) are proud, in the sense that you might use the word pride (e.g. proud of what you've achieved), they're simply proud that God has saved them.

      Cheers,
      Victor

    18. Re:Silly rabbit. by Leebert · · Score: 5, Insightful

      There should always be duplication of critical components of a system with the intention of increasing reliability of the system, usually in the case of a backup or fail-safe.

      Let me try to reply a bit more constructively than some of the others here.

      It is never a foregone conclusion that you will always have duplication of critical components of a system, if you are doing proper risk management.

      Essentially, the art of risk management is figuring out how far to go with mitigations of various risks.

      To illustrate with an excessively simplistic example (Assume a perfect vacuum and a frictionless environment):

      Let's say you sell something online, you sell W products/hour, and if you miss a sale, that's it, you're not getting it back.

      So that means that you lose the profit on W products every hour, let's call that X.

      Next, you look at the potential hazards, and calculate how often you expect to have each hazard occur per year. For example, to be simple, let's pretend your only hazard is that you expect the basement to flood once every 20 years, causing a complete outage of your data center. This means your Annualized Rate of Occurrence (ARO) is 0.05 basement floods/year.

      Further suppose that you expect a downtime from a basement flooding to last, say, 24 hours. That means your Single Loss Expectancy is your profits per hour, X, times 24 hours, let's call that Y.

      From Y and 0.05, we can calculate the Annualized Loss Expectancy, that is, the cost of a single occurrence times the probability of occurrence in any given year. So let's let Z be the ALE of (Y * 0.05).

      If the annualized cost of having an alternate data center to mitigate only the risk of flooding exceeds Z, the Annualized Loss Expectancy, you do not invest in an alternate data center, because it makes no business sense. You just take the loss when it happens, because it's cheaper than dealing preventing it.

      Of course, it's *never* quite this simple, and sometimes the SLE is essentially infinite (such as when loss of life could occur) and thus you spare no expense in mitigating the risk. Sometimes, you can't easily quantify the cost, because it isn't always money, it could be, for example, reputation.

      But it is *never* a foregone conclusion that you should automatically spend money mitigating risk without first thinking about if the mitigation costs more than the risk itself.

    19. Re:Silly rabbit. by neonKow · · Score: 2, Funny

      They were. The data they lost was obviously wicked data that needed to be cleansed from this earth.

    20. Re:Silly rabbit. by neonKow · · Score: 4, Insightful

      Seriously, have you ever actually read any respectable theology? No, probably not, which is why you promote these misinterpretations and equivocations. Go read some of the top theologians of our day, and then tell me how stupid you think it all sounds. You are not qualified to engage in reasoned discussion on this issue until you have.

      And this, boys and girls, would be what sinful pride looks like.

    21. Re:Silly rabbit. by Abstrackt · · Score: 2, Insightful

      Seriously, have you ever actually read any respectable theology? No, probably not, which is why you promote these misinterpretations and equivocations. Go read some of the top theologians of our day, and then tell me how stupid you think it all sounds. You are not qualified to engage in reasoned discussion on this issue until you have.

      I've read "respectable theology" and I'd like to submit the following quote for you to ponder: "the greatest single cause of atheism in the world today are Christians who acknowledge Jesus with their lips and walk out the door and deny Him by their lifestyle."

      Churchgoing or considering myself of any particular religion really isn't my thing, I have tried it at least, but I've always believed that the surest sign of someone secure in their faith is that they can defend it without attacking others. If you consider yourself a good christian you should really evaluate your participation in today's discussion because you are seriously damaging the reputation of people who are better because of their religion.

      --
      They say a little knowledge is a dangerous thing, but it's not one half so bad as a lot of ignorance. - Terry Pratchett
  2. Shit happens by Anonymous Coward · · Score: 2, Interesting

    Sometimes it's cheaper to deal with it when it happens than to take precautions.

    1. Re:Shit happens by KevinKnSC · · Score: 2, Insightful

      I don't disagree with you, but I strongly suspect this will be one of those times that it really would have been worth it to take precautions.

    2. Re:Shit happens by vlm · · Score: 2, Interesting

      The criminal justice system of Dallas - maybe not in the same category.

      No. Real high availability is like the AC power to the respirator in a hospital surgical room.

      The criminal justice system of Dallas was closed exactly one week ago today for the holiday. Giving the parole officers, judges, juries, attorneys off is no big deal. The inmates, unless they had a trial scheduled today, frankly probably won't know the difference.

      --
      "Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
    3. Re:Shit happens by Monkeedude1212 · · Score: 2, Interesting

      I have not seen a case with a Data Center where this has EVER been the case.

      Essentially, look at it this way:

      Everything gets destroyed. You have to buy new ones. Then you have to start all over. It took you 3 days to get operational again, and now you've got to some-how start over from scratch.

      Or what they could have done:

      Everything gets destroyed. You already bought a spare system just in case, and had it stored off-site. For about a negligable amount, you went through the effort of backing up the data once a month, to either a spare hard drive, or to the old computer your IT admin has in his basement that he never uses. It'll take you 3 days to get operational again, but at least you'll have that data somewhere and you don't have to play catchup.

      Or if they wanted to go crazy

      Everything gets destroyed. They had a spare system set up with full redundancy in case this kind of disaster. The cost was high when setting it up, but you have essentially no down-time. Ideally, the 3 days you saved will cover the cost of the damaged equipment. Or if the data is particularily sensative, that is the cost you pay to keep it safe.

    4. Re:Shit happens by kiwimate · · Score: 2, Interesting

      Exactly. "Principles of Corporate Finance", Brealey/Myers/Allen.

      Major public companies typically buy insurance against large potential losses...

      BP has challenged this conventional wisdom...

      BP...took a hard look at its insurance strategy...BP decided not to insure against most losses over $10 million. For these larger, more specialized risks BP felt that insurance companies had less ability to assess risk and were less well placed to advise on safety measures. As a result, BP concluded, insurance against large risks was not competitively priced.

      How much extra risk did BP assume by its decision not to insure against major losses? BP estimated that large losses of above $500 million could be expected to occur once in 30 years. But BP is a huge company with equity worth about $200 billion... BP concluded that this was a risk worth taking. In other words, it concluded that for large, low-probability risks the stock market was a more efficient risk-absorber than the insurance industry.

      Now we get to see how well their hedging will work out.

  3. Sewer line by Anonymous Coward · · Score: 4, Funny

    At least it wasn't a sewer line break, that would be real shitty.

  4. Something like that happened to us... by snowraver1 · · Score: 2, Insightful

    About a year ago...

    At the time, we had all our WLAN connections carried through Bell Canada VIA Frame circuits. I guess many of these circuits went through a facility in Edmonton. This facility was being rennovated, and some poor worker drilled through a pipe that they thought was empty... As it turns out, that pipe was filled with pressurized water, and so the water started spraying everywhere/everything and ended up taking down all our frame services north of Edmonton (about 30 sites). It took about 2 (very stressful) days for Bell to route our frame circuits through another data center.

    It sucked, but I really feel bad for the poor guy that drilled through the wrong pipe.

    --
    Copyright 2010. All rights reserved. This comment may not be copied in any way including, but not limited to caching.
    1. Re:Something like that happened to us... by vlm · · Score: 2, Interesting

      At the time, we had all our WLAN connections carried through Bell Canada VIA Frame circuits.

      Was it ever possible to buy a frame relay switch that doesn't have an automatically rerouting ATM backbone as the underlaying technology? As far as I know, the answer is no.

      ended up taking down all our frame services north of Edmonton

      Somebody's got a single point of failure in Edmonton. Huge design mistake, not inherent technological limitation.

      I worked for a carrier for many years that was properly designed. PVCs would drop over the dead trunk and reroute over the live trunk transparently, assuming you were using a real packet based protocol like TCP/IP and not SNA/SDLC where a dropped packet means reset the SNA communications controller...

      It sucked, but I really feel bad for the poor guy that drilled through the wrong pipe.

      The dude with the drill might be in ice cold Edmonton water. But the dude whom designed in the single point of failure in Edmonton is probably in really hot water.

      --
      "Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
    2. Re:Something like that happened to us... by molo · · Score: 2, Informative

      I think you mean WAN.

      -molo

      --
      Using your sig line to advertise for friends is lame.
  5. Re:Tested Backups? by Fallen+Kell · · Score: 2, Insightful

    Probably not....

    --
    We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
  6. Who's idea... by Pharmboy · · Score: 4, Insightful

    Whose bright idea was it to put the UPS and backup systems in the portion of the building that is first to be flooded, and the most devastated in just about any natural disaster, AND the least accessible afterward? Sounds like something a government would do....

    --
    Tequila: It's not just for breakfast anymore!
    1. Re:Who's idea... by NewWorldDan · · Score: 4, Insightful

      .... and it's also the part of the building that's easier to cool and isn't in demand for office space. A lot of businesses put their data centers in basements. I've seen a few places that built dedicated buildings for the data center, but usually, cost dictates that they stick it where they can.

      Frankly, while it will be a pain in the butt for 2 weeks, they'll get through this just fine. If they had a redundant data center, people would be whining about the waste of money and so on. There's no right answer here.

    2. Re:Who's idea... by fuzzyfuzzyfungus · · Score: 3, Interesting

      It's a terrible plan; but putting the heavy stuff in the basement makes life easier(unless your building was purpose-built, or you have serious renovation funds, putting thousands of pounds of UPSes on one of the upper floors isn't always one of your choices). And, more generally, there seems to be this perverse part of human nature that clings to the atavistic belief that stuff you don't have to look at isn't an issue. We put the ugly, heavy, parts of the system in the basement, we bury dangerous chemical wastes(where it is virtually impossible to inspect them for leakage, and all leakage goes directly to the water table) rather than putting them on the second floor(where discovering leaks is as easy as walking through the first floor)...

    3. Re:Who's idea... by Nemesisghost · · Score: 3, Interesting

      Actually most buildings in Texas are built where the basement is the most structurally sound part of the building. This is because the biggest 'natural' disaster are tornadoes. This is especially true of older buildings(like most government buildings). There's only one state run building I know of that was built any other way. It's one of the state backup centers in West Central Texas. They spent 2-3x the amount of a comparable building so that the top floor could withstand hurricane force winds. They didn't do it because it was easy or economical, they did it because if you want to remain safe during a major storm, you head to the basement.

  7. Texas doesn't need backup. by copponex · · Score: 4, Insightful

    In a state as blessed as Texas, they were told that God would provide protection against acts of God. I imagine many of the faithful are confused, especially when Jesus day is only a few days away.

    Maybe they didn't execute enough retarded people this year?

    http://en.wikipedia.org/wiki/Jesus_Day

    1. Re:Texas doesn't need backup. by Anonymous Coward · · Score: 2, Funny

      Maybe they didn't execute enough retarded people this year?

      They'll get around to you, don't worry.

    2. Re:Texas doesn't need backup. by PopeRatzo · · Score: 4, Funny

      While Governor of Texas, Former President George W. Bush, signed a bill into law proclaiming June 10, 2000 to be Jesus Day

      Wow.

      This year, they decided to re-write textbooks to eliminate the "liberal bias".

      You know what? Maybe the Dallas datacenter going out without a backup is a blessing in disquise.

      You think we could get Texas to secede if we asked them real nice? We can give them Arizona and a state to be named later, as long as they promise to never tell anyone that they were once part of the United States.

      --
      You are welcome on my lawn.
    3. Re:Texas doesn't need backup. by copponex · · Score: 2, Insightful

      By the way, what's wrong with re-writing the textbooks to eliminate the hardcore pro-materialist bias?

      Pro-materialist bias? Is that what you ingrates are calling science and history now?

    4. Re:Texas doesn't need backup. by PopeRatzo · · Score: 4, Insightful

      That wouldn't work -- existing historical records show that Texas has been part of the United States

      We could re-write the textbooks to eliminate the bias toward truth.

      It would only be fitting.

      --
      You are welcome on my lawn.
  8. Prepared government or small government pick one by zerofoo · · Score: 4, Insightful

    Data center floods, Katrina, and the BP oil spill the Gulf. All have one thing in common - a government that was not prepared.

    I chuckle when I hear my more conservative friends complain about the level of competence or disaster response times we find in government. These are the very same people who want less government and lower taxes.

    You can not have a small, cheap, government that is staffed by geniuses and prepared for every possible problem. Smart people cost money, resources cost money.

    While this flood was an avoidable occurrence, do taxpayers really want to pay for redundant EVERYTHING at the local, state, and federal government? Probably not.

    -ted

  9. Re:Tested Backups? by NevarMore · · Score: 2, Funny

    It has now.

  10. Re:Prepared government or small government pick on by Anonymous Coward · · Score: 2, Informative

    In what world do you live in that the government is ultimately responsible for a massive oil spill that was caused by a corporation?

    While the government may be responsible for the fact that the regulations weren't followed every step of the way, the company that is behind the actions on the rig are ultimately fully responsible for not following those rules and regulations. I'm glad that the government can help in any way possible, but BP should foot the bill.

  11. Someone forgot the rules... by U8MyData · · Score: 3, Insightful

    From the movie Contact: "First rule in government spending: why build one when you can have two at twice the price?" Load balanced and replicated no less. Basement + critical systems = bad idea. Not only for flooding potential, but bad sewer systems as well. Yes been there done that. Curious, is there a systems engineer that could make a good argument for building data center infrastructure in a basement. Two points already for shielding from severe weather; anything else?

    1. Re:Someone forgot the rules... by Renraku · · Score: 4, Insightful

      Climate control is easier in the basement. You can build big fuckoff heat exchangers that go under ground level and surface however far from the building you want them to surface.

      Simpler wiring plans because you don't have to run big industrial power cables up to the top floor and the data lines don't have to go far to get to the basement.

      All that being said, below-ground server rooms should have some method to be able to seal themselves off from the rest of the world in case of flooding. Perhaps the elevator or hallway door can form a decent seal, whereas everything else is already as sealed as it can be. Perhaps sealing everything also cuts power so nothing overheats.

      --
      Job? I don't have time to get a job! Who will sit around and bitch about being broke and unemployed then?
    2. Re:Someone forgot the rules... by Locke2005 · · Score: 2, Insightful

      Climate control is easier in the basement. Assuming you are using ground source heat exchange, yes. Otherwise, it's about as far as possible from the A/C heat exchangers on the roof. Which do you think Dallas was using?

      --
      I've abandoned my search for truth; now I'm just looking for some useful delusions.
  12. Re:Prepared government or small government pick on by scamper_22 · · Score: 2, Interesting

    That is a very false false dichotomy... and a very poor argument.

    Your basic argument is that people who favor small government expect the government to do nothing.
    It's like talking about small government, and someone says: don't you like safe food inspections?
    Yes, I 'like' those things. I also don't mind the government doing them. That's why I believe in small government. Not no government.

    Just take a look at the government's spending. The things 'small government' folks want the federal to do would cost next to nothing.
    Our biggest costs are healthcare, military...

    So go ahead and cut those things down to focus on small government. It will free up hundreds and hundreds of billions of dollars.
    Then, I'd be more than happy if the government spent what it costs to have proper data centers.

  13. Some clarifications by rabtech · · Score: 5, Insightful

    First, this is Dallas County, not Dallas city.

    Second, they knew about the potential for failure and were working on setting up a backup data center. TxDOT denied them rights of way to lay fiber along the highway into a facility in Tarrant county, so they were looking at other potential sites in Garland. Unfortunately this happened before they got it all resolved.

    TxDOT might have had good reasons for denying the request, I don't know, but I would wager that the backup site would be a lot further along if they had been able to run that fiber. Sometimes you know there is a problem, management agrees, and you even have a budget to fix it... but someone else (another department, another company, a government agency, etc) stands in the way.

    --
    Natural != (nontoxic || beneficial)
  14. Root Cause by ThatsNotPudding · · Score: 2, Insightful

    From TFA: "The 90-year-old water main ruptured..."

    90 years old because taxpayers (read: well-heeled conservatives) never want to pay for maintaining and replacing infrastructure until after the disaster occurs. No doubt they will somehow get federal funds to help defray the costs - all the while cursing the federales' very existence.