University of Twente NOC Destroyed

JanJoost writes "Around 08.00 CET today the University of Twente Network Operations Center, which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org, caught fire. The UT, which hosted the HAL in august last year is completely unreachable and is not likely to come back up any time soon. The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings. More information can be found at the Telegraaf, Planet Internet and Twentsche Courant. Pictures can be found here and here. It's a shame to see a great infrastructure go down in flames like this."

Comments

now the engineers come out...

[ravidew]

..to see how this could be prevented in the future. How much fire protection do NOCs owned by the big boys (Verio, WorldCom) have? Offsite backups, too, I hope?

Re:now the engineers come out...

[DunbarTheInept]

When you can't get oxygen from the surrounding air, the worst thing in the world to do is take a breath. You will survive much longer by fighting the urge to take a breath, and instead just hold the air already in your lungs in place until you can get out. You don't convert all the oxygen in your lungs into carbon dioxide with each breath - far from it. You just convert a portion of it, so the air you expell just has a smaller percentage of oxygen than the air you breathe in. But there's still quite a bit there that never got converted. (That's why mouth-to-mouth breathing can help someone - the air you breathe out still has enough oxygen in it to be a lot better than no air at all.)

But if you breathe that air out and breathe oxygen-less air in, you will pass out very fast. Most people have the misconception that you can do without oxygen for a minute or two before dying. That's not true - your body needs to consume fresh oxygen at a continuing rate just to function at all, it's just that your lungs can HOLD a small supply of oxygen to supply this need for a minute or two. Get rid of that oxygen by breathing it out and replacing it with oxygenless air, and you're going to pass out in just a few seconds, and be dead shortly thereafter.

And the worst part is you won't FEEL like anything is wrong. Your body is unable to measure the level of oxygen in your lungs. Instead your body senses the level of carbon dioxide in your lungs. As the by-product of normal breathing, when carbon dioxide has built up enough, that indicates you've converted a lot of oxygen and it's time for another breath. This is what triggers the automatic involuntary breathing that takes over when you stop thinking about it. This is also what triggers the panic feeling that you get when you know you need air. Your lymph nodes detect too much carbon dioxide and start sending the panic signal to your mind. What this all means is that if your body isn't exchanging oxygen for carbon dioxide, your body doesn't even realize it's asphixiating. If there's no oxygen in your lungs to start with, then there won't be any carbon dioxide building up in the lungs, and you will feel no sensation of needing a breath at all. You'll feel just fine for a few seconds and then *poof* you're gone as the blood going to your brain runs out of oxygen and your brain activity just plain stops.

So if you're ever in a halon gas system when it goes off - DO NOT BREATHE. Just hold whatever breath happens to already in your lungs and get out. The instinct is to hold your breath by first inhaling your lungs full and THEN holding it, but that's the worst thing you could do, as explained above. The tricky part is remembering to override that instinct.

A good reminder..

[Martigan80]

To never keep back-ups in the same physical location.

Re:A good reminder..

[Zocalo]

Or even general location for that matter. A friend of mine did disaster recovery work for IBM after the Trade Towers attack. They had their data center in Tower 1 and their backup center in Tower 2. After six weeks of what was essentially scrabbling through rubble they managed to recover a single spindle. The company concerned became another statistic, and part of an important lesson in DR implementation; safety increases with distance.

Priorities

[brianvan]

Was anyone killed?

If not, was anyone hurt?

If not, do they have insurance?

If they do... well, I'm sure someone just lost their masterpiece pr0n directory, but otherwise, things like this happen. (ask Hemos) You have to make it through such things. In this case, it was a commercial (educational) building and no one is homeless, so it's less of a tragedy than usual. Let's hope that they rebuild with something better and newer.

That said, I get the feeling that those plumes of smoke really are millions of dollars floating away in the wind...

PGP Keyserver root was hosted by SURFNet

[Erik_]

I believe, the open and distributed network of Keyserver.net (distibuted network of PGP keyservers) was hosted by SURFNet. This network is a distributed network holding PGP and OpenPGP keys. The loss the to UT NOC could have an impact on the updating of key-rings across the keyserver.net network.

Re:So much for server areas never burning down

[Plutor]

The fact that "The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings" leads me to believe that the fire didn't start in the server area. Lots of server rooms were destroyed on September 11, for example, but it wasn't the fault of the room's design, or the presence or lack of fire suppression systems. If the whole building is burning down, fire suppression in one room is only going to work until the floor and ceiling collapse.

BACKUP!!!

[Beliskner]

Now we can find out how secure and hardened Debian really is. You are as good as your latest backup.
BACKUPS BACKUPS BACKUPS Off-site! I've had enough of people who are talking about RAID-5 because 5TB tape drive arrays are too slow. Always keep your BACKUPS!

About as good as it gets with only two sites...

[Kjella]

I mean seriously, each tower collapsed because it was hit by its own plane. If one tower had been in NY, one in California and both were still hit by a plane each, the result would be exactly the same.

The lesson should be: Primary back-up is a very good start, but secondary/tertiary back-up is the thing if it's that critical.

Kjella

Re:Disaster Recovery plans

[wiredog]

The Gartner Group put some stuff up after 9/11. Most of it is common sense.

Do full backups weekly, store copies offsite. Incremental backups daily, copies offsite also. If you can afford it (or can't afford any downtime), have emergency backup hardware (enough for minimal operations) in an offsite storage facility. Old hardware that would otherwise be thrown out is good for this (remember, it's for an emergency). Have a supplier who can get replacement hardware to you in a hurry (so you can get off of those old 90 MHz Pentium servers).

The most vital part of the plan, after backups, is good insurance. If the building burns to the ground Monday morning, you want to be able to call the insurer Monday Noon, and have the check in hand Tuesday morning at the latest.

These recommendations do not cover disasters such as 767s flying into the building and killing all the sysops. Earthquakes dropping the building on the same. Etc. The people are the most important part of any company and, if too many of them are lost at once, the company probably is lost too.

Unless you have really good (and expensive)insurance which can provide enough funds for you to hire new people, get them trained, and keep the company solvent while you do so.

Re:Vunerability

[coupland]

It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup.

Uhhh, yes there is... I suspect you either know nothing about IT or are fresh out of college. DRP (disaster recovery planning) factors in things such as criticality of data, cost, and acceptable downtime. A university payroll system may need to be back up within 12 hours of a major incident, so in addition to tape backups you might have a failover site. Contrary to your simplistic post, even the richest corporations rarely have failover sites of their own. They simply contract out to a DRP vendor who have these types of machines lying dormant in a glass room, waiting to cut over. On the other hand a university FTP site is probably classified as low risk, low impact. So you would rely on off-site backup tapes and perhaps only restore when you've arranged for an alternate site and taken delivery of new servers. You don't pay millions of dollars to have two glass rooms just so you can have uninterrupted FTP service...

Re:Vunerability

[pellaeon]

Try doing that on a university-wide multi-gigabit capable network on the budget of the average Dutch university. Our universities aren't like M$ in cash, you know. I know, I'm an admin at one myself.

I just hope they're well insured....poor colleagues...

On the upside: they may get a squeaky-clean start when this blows over :-)

They Needed Low-Tech Fire Protection

[0x69]

Our ISP bought an old legal office building for their HQ and colo facility. The place was built with file rooms to safeguard tons of irreplaceable paper documents - imagine thick concrete walls & ceilings, with heavy steel fire doors, rated to preserve the contents through an EVERYTHING-else-burned-to-the-ground fire.

Critical stuff is spread between the file rooms, with metal conduit, etc. protecting the few small holes they added for wiring.

Steel & reinforced concrete aren't quite obsolete.