Slashdot Mirror
Water Main Break Floods Dallas Data Center
miller60 writes "IT systems in Dallas County were offline for three days last week after a water main break flooded the basement of the Dallas County Records Building, which houses the UPS systems and other electrical equipment supporting a data center in the building. The county does not have a backup data center, despite warnings that it faced the risk of service disruption without one."
This is Texas - God is their backup solution.
Every night they pray for no hardware failures.
God is their backup solution.
If this is true, shouldn't they have been prepared for a flood?
"Never let your sense of morals prevent you from doing what is right" - Salvor Hardin
In Texas, we are proud of our faith.
Which is incredibly funny, especially considering that pride is considered one of the deadly sins (along with lust, greed, gluttony, and several others that Texans are well known for). Apparently too many of you are too busy thumping your chests about how you are such good Christians to even stop to consider what it really means to be a good Christian.
First, this is Dallas County, not Dallas city.
Second, they knew about the potential for failure and were working on setting up a backup data center. TxDOT denied them rights of way to lay fiber along the highway into a facility in Tarrant county, so they were looking at other potential sites in Garland. Unfortunately this happened before they got it all resolved.
TxDOT might have had good reasons for denying the request, I don't know, but I would wager that the backup site would be a lot further along if they had been able to run that fiber. Sometimes you know there is a problem, management agrees, and you even have a budget to fix it... but someone else (another department, another company, a government agency, etc) stands in the way.
Natural != (nontoxic || beneficial)
There should always be duplication of critical components of a system with the intention of increasing reliability of the system, usually in the case of a backup or fail-safe.
Let me try to reply a bit more constructively than some of the others here.
It is never a foregone conclusion that you will always have duplication of critical components of a system, if you are doing proper risk management.
Essentially, the art of risk management is figuring out how far to go with mitigations of various risks.
To illustrate with an excessively simplistic example (Assume a perfect vacuum and a frictionless environment):
Let's say you sell something online, you sell W products/hour, and if you miss a sale, that's it, you're not getting it back.
So that means that you lose the profit on W products every hour, let's call that X.
Next, you look at the potential hazards, and calculate how often you expect to have each hazard occur per year. For example, to be simple, let's pretend your only hazard is that you expect the basement to flood once every 20 years, causing a complete outage of your data center. This means your Annualized Rate of Occurrence (ARO) is 0.05 basement floods/year.
Further suppose that you expect a downtime from a basement flooding to last, say, 24 hours. That means your Single Loss Expectancy is your profits per hour, X, times 24 hours, let's call that Y.
From Y and 0.05, we can calculate the Annualized Loss Expectancy, that is, the cost of a single occurrence times the probability of occurrence in any given year. So let's let Z be the ALE of (Y * 0.05).
If the annualized cost of having an alternate data center to mitigate only the risk of flooding exceeds Z, the Annualized Loss Expectancy, you do not invest in an alternate data center, because it makes no business sense. You just take the loss when it happens, because it's cheaper than dealing preventing it.
Of course, it's *never* quite this simple, and sometimes the SLE is essentially infinite (such as when loss of life could occur) and thus you spare no expense in mitigating the risk. Sometimes, you can't easily quantify the cost, because it isn't always money, it could be, for example, reputation.
But it is *never* a foregone conclusion that you should automatically spend money mitigating risk without first thinking about if the mitigation costs more than the risk itself.