University of Twente NOC Destroyed

JanJoost writes "Around 08.00 CET today the University of Twente Network Operations Center, which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org, caught fire. The UT, which hosted the HAL in august last year is completely unreachable and is not likely to come back up any time soon. The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings. More information can be found at the Telegraaf, Planet Internet and Twentsche Courant. Pictures can be found here and here. It's a shame to see a great infrastructure go down in flames like this."

Comments

More info

[Strike]

From debian-devel, here's a slightly (only slightly) more informative blurb

Re:More info

[Stonehead]

Hey, I posted that to debian-devel. Scary to see it get Slashdot headlines, since this posting from Wichert Akkerman himself is more 'official'.
He's now probably busy setting up klecker.debian.org as the next security.debian.org host. Don't get yourself trojaned, please people, don't panic and just wait for the official Debian announcement that everything has been fixed again. Or play around with inofficial mirrors like these, and there are more. But I feel a bit stupid myself, because - unlike Wichert - I have done nothing myself except forwarding the news and act like a karma whore.

Well Damn...

[jhines0042]

... I hope nobody was hurt.

After that, I wish them luck getting back online.

Photo's

[fearlezz]

More images can be found on http://www.bsdfreaks.nl/files/brand.htm

now the engineers come out...

[ravidew]

..to see how this could be prevented in the future. How much fire protection do NOCs owned by the big boys (Verio, WorldCom) have? Offsite backups, too, I hope?

Re:now the engineers come out...

[Yo Grark]

After 911 all the big techfirms created DRA's (disaster recovery area's) in case anything major happened.

I had no less than 1000 companies call asking how to handle licensing and DRA's.

I'd be surprised if they didn't have one, it's pretty much standard practice comercially....

Yo Grark

- Canadian Bred with American Buttering

Re:now the engineers come out...

[rleyton]

Whilst I'm no expert in fire supression systems, I have studied a few data centres in my time as a Senior Sys Admin for a number of companies. Most (decent) NOC's have a fire *supression* system based on gas or (more likely these days) some form of liquid (gas has the nasty side-effect of potentially killing humans).

These are for small fires that can be contained within a data centre, ie. a computer catching fire or emitting smoke. Really good systems are very localised (racks or cage specific). A big fire just isn't going to be stopped by such a system.

I haven't read too much into this particular incident (ie. not at all), but my initial thought was that something more serious must have happened (well, duh!), perhaps a fire outside of the main suppression system (outside of the raised floor area?). Or perhaps the paint on the walls/carpet wasn't fire resistant and just took hold very quickly.

Or a large initial fire (gas leak?) that just didn't die down when the supression system kicked in. Maybe the type of fire (again, gas? oil?) didn't die down because Data Center supression systems presumably focus on electrical fires.

A well, just my 2p's worth.

Re:now the engineers come out...

[Martin Blank]

Most NOCs I've been in have an oxygen-deprivation gas that's dispersed in the case of a fire (after a series of highly visible and audible warnings). The one I'm in right now seems to have gone the inexpensive (and arguably safer) way of a two-stage dry-pipe water sprinkler system where the pipes are usually dry (empty). In case of a smoke detection above a certain level, the pipes are "charged" (filled with water), and if the heat gets above a certain level, the sprinklers go off. Basically, the decision was made that if there's a fire, the equipment will probably need replacement anyway, so why not use sprinklers?

Personally, I'm not too sure of this route. I can understand it, from an environmental and human-safety perspective (the gasses eat the ozone layer, and you *really* don't want to inhale stuff that ties up oxygen at those rates), but if it ever happens here, it's going to be an awfully hurried mess to get everything back up and running in a reasonable timespan, even if the fire is only a small one that doesn't destroy the building.

Re:now the engineers come out...

[Nos.]

We have a similar two stage system as mentioned above. When the installers came by a year later for a test, they must have "forgotten" how they hooked it up. Before the sprinklers actually start spraying water, the breaker that runs the power to that room is automatically flipped. The UPS we purchased for the servers is also hooked up so it doesn't do a safe power down, it just powers down.

Well, they came in to do the test, everything but actually spraying water, and managed to flip the breaker. It was interesting to note, that only a few of the machines actually powered off. Some are on smaller single machine UPSs, so those stayed running. The most expensive piece of equipment in there, a Nortel Telephony Switch which handles 6 PRI lines (almost 150 ISDN lines) is on battery backup, and kept running.

All in all very little powered down. In the even of a real fire, we would have had sparks flying everywhere. What did management do when I mentioned this to them? That's right, nothing.

Re:now the engineers come out...

[AlphaInsight]

Actually FM-200 and Inergen are the current replacements for Halon. Non Ozone-depleting and won't kill you if you're trapped in a NOC. Just lowers the avaliable O2 in the air. You'll get out of breath easily, but as long as you stay calm you won't have to worry.

Re:now the engineers come out...

[CaptainZapp]

NOC's have a fire *supression* system based on gas or (more likely these days) some form of liquid (gas has the nasty side-effect of potentially killing humans).

When I worked for DEC they actually had sprinkler systems in their data centers (I couldn't verify first hand that every data center had sprinkler, but I was told so).

The reasoning was, that even if they was one big, bad, evil, wicked water damage in a data center, that the company considered this to be better then one killed employee by gas extuinguishing systems.

That was in the early 90ties

Re:now the engineers come out...

[strictnein]

You'll get out of breath easily, but as long as you stay calm you won't have to worry

And, as we all know, there's nothing more calming than being in a room that's on fire and having some funky gas poured into the room that makes it harder to breath

Re:now the engineers come out...

[DunbarTheInept]

When you can't get oxygen from the surrounding air, the worst thing in the world to do is take a breath. You will survive much longer by fighting the urge to take a breath, and instead just hold the air already in your lungs in place until you can get out. You don't convert all the oxygen in your lungs into carbon dioxide with each breath - far from it. You just convert a portion of it, so the air you expell just has a smaller percentage of oxygen than the air you breathe in. But there's still quite a bit there that never got converted. (That's why mouth-to-mouth breathing can help someone - the air you breathe out still has enough oxygen in it to be a lot better than no air at all.)

But if you breathe that air out and breathe oxygen-less air in, you will pass out very fast. Most people have the misconception that you can do without oxygen for a minute or two before dying. That's not true - your body needs to consume fresh oxygen at a continuing rate just to function at all, it's just that your lungs can HOLD a small supply of oxygen to supply this need for a minute or two. Get rid of that oxygen by breathing it out and replacing it with oxygenless air, and you're going to pass out in just a few seconds, and be dead shortly thereafter.

And the worst part is you won't FEEL like anything is wrong. Your body is unable to measure the level of oxygen in your lungs. Instead your body senses the level of carbon dioxide in your lungs. As the by-product of normal breathing, when carbon dioxide has built up enough, that indicates you've converted a lot of oxygen and it's time for another breath. This is what triggers the automatic involuntary breathing that takes over when you stop thinking about it. This is also what triggers the panic feeling that you get when you know you need air. Your lymph nodes detect too much carbon dioxide and start sending the panic signal to your mind. What this all means is that if your body isn't exchanging oxygen for carbon dioxide, your body doesn't even realize it's asphixiating. If there's no oxygen in your lungs to start with, then there won't be any carbon dioxide building up in the lungs, and you will feel no sensation of needing a breath at all. You'll feel just fine for a few seconds and then *poof* you're gone as the blood going to your brain runs out of oxygen and your brain activity just plain stops.

So if you're ever in a halon gas system when it goes off - DO NOT BREATHE. Just hold whatever breath happens to already in your lungs and get out. The instinct is to hold your breath by first inhaling your lungs full and THEN holding it, but that's the worst thing you could do, as explained above. The tricky part is remembering to override that instinct.

Re:why does this matter?

[Sc00ter]

can you read?

"which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org "

security.debian.org and non-us.debian.org are GONE (well, burning, hopefully data can be recovered). Yes there's mirrors, but it still sucks.

Vunerability

[e8johan]

This shows the vunerability of putting all computers in one building. To have a safe network one needs to spread (duplicate) the information over several computer at several locations. How far apart these locations has to be is depending on how important you data is.

It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup. By that I don't only mean that tape that always seems to go missing when needed, but multiple sites (or at least buildings) that provide redundancy.

Re:Vunerability

[Psiren]

To have a safe network one needs to spread (duplicate) the information over several computer at several locations.

What a stunning idea. Perhaps they should call it the internet... ;-)

Re:Vunerability

[nbvb]

TSM (Tivoli Storage Manager) does a great job at mitigating all of those problems:

1) Sending tapes off-site too early -- TSM allows you to have multiple "storage pools" within one library. Basically, you backup your data into one storage pool (set of tapes) and then duplicate it to the off-site copies. Then the DRM (Disaster Recovery Manager) scripts automatically eject the tapes from the library for packaging to send off-site. You don't ship off your on-site copies, so when you need to do a restore, it's always here.

2) Partly full tapes -- TSM has a concept of "migrations" where it moves data between tapes to better utilize them. The internal DB automatically knows where the data is, so you don't have to worry about which tape has which days' backups -- it doesn't matter.

3) Full backupset -- TSM 4.2 introduced "portable backup sets" -- basically, you backup your data into the TSM server as normal, then "generate" a backupset -- takes all the files for a particular node and writes them to a series of tapes which you then eject and store off-site. When you need to recover from those tapes, the server component isn't needed -- locally attached tape on the server and the client piece of TSM is all you need --- speeds up recovery by a few hours!

TSM's a great product. Does lots and lots of great things that other vendors are only now trying to figure out.

Sure, it's pricy, but how much is your data worth?

Re:Vunerability

[coupland]

It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup.

Uhhh, yes there is... I suspect you either know nothing about IT or are fresh out of college. DRP (disaster recovery planning) factors in things such as criticality of data, cost, and acceptable downtime. A university payroll system may need to be back up within 12 hours of a major incident, so in addition to tape backups you might have a failover site. Contrary to your simplistic post, even the richest corporations rarely have failover sites of their own. They simply contract out to a DRP vendor who have these types of machines lying dormant in a glass room, waiting to cut over. On the other hand a university FTP site is probably classified as low risk, low impact. So you would rely on off-site backup tapes and perhaps only restore when you've arranged for an alternate site and taken delivery of new servers. You don't pay millions of dollars to have two glass rooms just so you can have uninterrupted FTP service...

Re:Vunerability

[pellaeon]

Try doing that on a university-wide multi-gigabit capable network on the budget of the average Dutch university. Our universities aren't like M$ in cash, you know. I know, I'm an admin at one myself.

I just hope they're well insured....poor colleagues...

On the upside: they may get a squeaky-clean start when this blows over :-)

A good reminder..

[Martigan80]

To never keep back-ups in the same physical location.

Re:A good reminder..

[Zocalo]

Or even general location for that matter. A friend of mine did disaster recovery work for IBM after the Trade Towers attack. They had their data center in Tower 1 and their backup center in Tower 2. After six weeks of what was essentially scrabbling through rubble they managed to recover a single spindle. The company concerned became another statistic, and part of an important lesson in DR implementation; safety increases with distance.

Re:A good reminder..

[Martin Blank]

Cantor Fitzgerald, the company hardest-hit in the attack with 700+ employees killed, had just that viewpoint. eSpeed, a subsidiary of CF, had all trading activities back up and running by the morning of Sept 13, due to the multiple locations and real-time backups being done on the network. My understanding is that the time from loss of connection with CF headquarters to full resumption of its trading activity around the world was about 46 hours.

Re:A good reminder..

[Jugalator]

To never keep back-ups in the same physical location.

That's why I use Kazaa to keep backups of my pr0n all over the world!

Re:A good reminder..

[Martin Blank]

My mistake: 47 hours.

Details on the recovery:
http://www.baselinemag.com/article2/0,3 959,36807,0 0.asp

Some information on the scale of what eSpeed handles:
http://wstonline.com/story/mag/WST200110 08S0007

Even Cantor Fitzgerald mentions it in their Q3/01 results summary:
http://www.cantor.com/articles/article11 192001.htm

Re:A good reminder..

[athakur999]

I personally don't keep backups. However, I'm sure at least two other me's in parallel dimensions do, so I'm probably good to go.

Hopefully the other me's in the other dimensions aren't counting on me to keep backups. Those idiots.

So much for server areas never burning down

[Martin Blank]

This is something I keep hearing from people, particularly those who balk at paying for upgrades to expensive fire suppression systems. "They're always built so well that there's never been a case of a server room destroyed by fire." I always did doubt that, and now whenever they tell me that, I can point to this.

What a shame.

Re:So much for server areas never burning down

[Plutor]

The fact that "The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings" leads me to believe that the fire didn't start in the server area. Lots of server rooms were destroyed on September 11, for example, but it wasn't the fault of the room's design, or the presence or lack of fire suppression systems. If the whole building is burning down, fire suppression in one room is only going to work until the floor and ceiling collapse.

Re:So much for server areas never burning down

[mmol_6453]

Here in Grand Rapids, Michigan, any cable that passes through the heater system has to be Plenum. And considering that, at Greybar, you have a choice between Plenum and riser-cable, most of the new (as in, last twenty years) telecom wiring in Grand Rapids is fire-resistant. The insulation of Plenum will liquify before it burns...

In a related announcement...

[Zocalo]

The University of Twente's attempts to overclock the new AMD Opteron and Nvidia GeForceFX card in the same case are declared a failure. "We certainly won't be building a Beowolf cluster of these..." commented a spokesperson.

Re:In a related announcement...

[OrangeSpyderMan]

UNIX? They're probably not even circumcised! Savages!

You're thinking of that Linux distro with a BSD style ports things aren't you? What's it called again, Gentle, Gentool - Gentile Linux!!! Thats right! :-)

Maybe..

[jedie]

Someone took "firewall" too seriously? :)
(and this isn't *flame*bait :p)

Priorities

[brianvan]

Was anyone killed?

If not, was anyone hurt?

If not, do they have insurance?

If they do... well, I'm sure someone just lost their masterpiece pr0n directory, but otherwise, things like this happen. (ask Hemos) You have to make it through such things. In this case, it was a commercial (educational) building and no one is homeless, so it's less of a tragedy than usual. Let's hope that they rebuild with something better and newer.

That said, I get the feeling that those plumes of smoke really are millions of dollars floating away in the wind...

Re:Priorities

[bzzzt]

No, no and no.

The university doesn't insure itself agains this kind of disaster because they are able to set aside enough money to cover the cost themselves and don't have to pay the insurer's profit.

Now how they are going to deal with this and their financial troubles (they're almost broke) is another issue...

last syslog messages from SURFnet routers

[ahu]

lo0.ar5.enschede1.surf.net 3613: Nov 20 07:20:50.927 UTC: %ENV_MON-2-TEMP: Hotpoint temp sensor(slot 18) temperature has reached WARNING level at 61(C)

few seconds later on the local side:
lo0.cr2.amsterdam2.surf.net 1146: Nov 20 07:20:56.458 UTC: %CLNS-5-ADJCHANGE: ISIS: Adjacency to ar5.enschede1 (POS2/0) Down, interface deleted(non-iih)

Re:last syslog messages from SURFnet routers

[Jugalator]

I wonder if they got the Linux "easter egg" error message "Printer on fire"? I guess it would be appropriate for once.

Re:last syslog messages from SURFnet routers

[CoolVibe]

Those are Cisco syslog messages. No linux easter eggs.

Halon dumps?

[wiredog]

The last time I worked in a NOC, it ran Vaxes, but we had a halon dump. A Big Red Button that got smacked by the last person out of the room. The halon would smother any fire by replacing all the oxygen in the room (which was why the last guy out hit the button). Why wasn't there a halon dump in this NOC? Or, if there was one, what happened?

I hope Debian practices good management principles by having offsite backup.

Re:Halon dumps?

[lamj]

Halon DO NOT replace oxygen in the room to extinguish the fire. It breaks the chain reaction of fire, basically stop the elements of fire to react with each other.

Most scenario would only require a less than 8% of concentration to take out the fire. Under 10% and you can still breath.

Problem of Halon is when over 900 degree C, it breaks down into hydrogen fluoride, hydrogen bromide and bromine - stuff that are toxic. So, run!

Re:why does this matter?

[brianvan]

They're not gone. They're just experiencing a Distributed Combustion Denial-of-Service attack.

Debian Security mirror

[lemmen]

In case one might be interested, Essent mirrored security.debian.org.

You can use debian.essentkabel.com to download the latest security updates (in case you haven't already). Please note this is NOT an official mirror.

Re:Debian Security mirror

[Tottori]

I don't want to besmirch Essent or anyone else, but please don't install security-critical software from a mirror some guy posted on Slashdot. apt-get doesn't check signatures, so if you put a compromised or malicious mirror in your sources.list, it's game over.

Debian haven't released a security advisory since yesterday, so it's deeply unlikely you'll need an update before they get a new security.debian.org online tomorrow.

OK, folks, admit it!

[matija]

Who posted a link to UT's webserver on slashdot?

Re:OK, folks, admit it!

[erpbridge]

And why would you do something like that? Think, McFly, think!

More Pictures from the webcam

[CuriousGeorge113]

Here's some statics taken from their webcam. Someone might want to mirror these before they get slash'd. Webcam PicsPics

Also, here's what seems to be the only close up I could find of the fire. pics

Halon is probably illegal today

[mangu]

Effective as it was, Halon was also a major ozone-hole cause, so it has been outlawed in most countries by now.

Dangerous server rooms

[Bazman]

Lets hope the place didnt look like this beforehand:

Dangerous server rooms

Update from SARA (SURFnet NOC)

[mdav]

Here's an update from SARA (that's where I work), the network operator for SURFnet. SURFnet is very busy ordering new equipment and fixing the 2 x 10 Gbit/s lamda's to Enschede. We hope to restore connectivity a.s.a.p. Greetings, Marco

Re:why does this matter?

[The Dobber]

Boy Scouts Of America? Which merit badge would that be?

It must have been Microsoft

[micaiah]

"Witnesses saw a large balding man monkey dancing from the scene and a slim geeky man with glasses trailing behind continuously adjusting his glasses." An in other news....

LFS mirror affected too

[decarelbitter]

The Dutch LFS FTP mirror was also hosted at the University of Twente, which means it's also down. The Dutch HTTP mirror should work properly, since it's outside utwente.nl space.
Last news is that HP (Who supplied most of the UT backbone equipment) is on its way with emergency equipment to have things up and running somewhere tomorrow.

Re:Halon dumps? -- not if the whole building is in

[earthy]

The fire did not start in the server rooms. What happened was that a fire started in one of the lecture rooms (and a smallish one at that, number A108) that just happened to be in the same wing of the TWRC building that also housed the server rooms (yes, multiple). It then proceeded to take out two entire wings and threaten other buildings nearby as well as the library.

Now, I would *love* to see a halon system capable of stopping that...

Owh, and the fire seems to be under control by now, as evidenced on http://webcam.traserv.com/
(which you can contrast to http://www-infstud.sci.kun.nl/~arthurvl/ispy.jpg (taken at about 09:40 CET this morning)).

PGP Keyserver root was hosted by SURFNet

[Erik_]

I believe, the open and distributed network of Keyserver.net (distibuted network of PGP keyservers) was hosted by SURFNet. This network is a distributed network holding PGP and OpenPGP keys. The loss the to UT NOC could have an impact on the updating of key-rings across the keyserver.net network.

Re:why does this matter?

[Angry White Guy]

The arson and destruction of private property badges. If a car burnt as well, they're well on their way to achieving the elusive 'inciting riots' and 'urban terrorist' badges. These last two were proudly held by a select few: Ted Kazinsky, Che Guevera, and Jimmy Carter

security updates mirrored worldwide.

[novakreo]

Actually, Debian security updates are usually also added to the proposed-updates section, which means that they are available on nearly all Debian mirrors worldwide.

Something like
deb ftp://ftp.XX.debian.org/pub/debian/ proposed-updates main contrib non-free
deb ftp://ftp.XX.debian.org/pub/debian-non-US/ proposed-updates/non-US main contrib non-free
(replace XX with your ISO country code) in your /etc/apt/sources.list should work well.

You can find mirrors on Debian's website.

BACKUP!!!

[Beliskner]

Now we can find out how secure and hardened Debian really is. You are as good as your latest backup.
BACKUPS BACKUPS BACKUPS Off-site! I've had enough of people who are talking about RAID-5 because 5TB tape drive arrays are too slow. Always keep your BACKUPS!

Translation of the Twentsche Courant Tubantia

[TeeWee]

[Apologies for the poor translation, no time for a better attempt]

The burning building contains the IT department and a part of the faculty of Business Administration [Closest I could come up with: Bestuurskunde]. The building has three storeys.

The university fears the loss of its network facilities and is trying to save the main computer. According to a spokesman this network is amongst the fastest in Europe. Most classes are expected to resume as normal today. [According to other sources, this is an exam week, meaning few classes anyway. This is also a reason that few students were around so chances were that this also reduced potential casualties]

The fire department is fighting the fire with 25 firefighters and expects to need the entire day to extinguish the fire. No dangerous materials have been released by the fire at this point. At the moment nothing is known about the cause of this fire.

Halon systems aren't illegal, but....

[mks113]

You can't install new halon systems, but existing ones are still legal (in Canada, anyway).

However, I wouldn't want to have to fill out all the paperwork involved with a discharge! We had an accidental discharge (a leak, I believe) and they decided it was enough impetus to remove the system.

I think they are using CO2 now. The advantage of Halon is that you can breathe quite comfortably in an atmosphere that will not sustain fire. CO2 works just about as effectively but will not sustain life.

OTOH, more recent studies have shown that just because you can still breath in a Halon infiltrated environment doesn't mean that there are no health effects!

I expect there are more CO2 systems going in now, with lots of alarms to make sure people get out before the atmosphere gets unlivable.

Michael

Re:Halon systems aren't illegal, but....

[914]

Halon will kill you just as dead, just as quickly, as CO2.

Both gasses extinguish fire by excluding oxygen... which isn't good for people either.

The difference is that one cannot comfortably breath CO2, an involuntary physiological reaction makes it impossible. (next time you open a bottle of Coke, try sniffing the little cloud of CO2 that forms in the top of the bottle)

Halon, otoh, is perfectly comofortable to breathe in and out, but will provide no oxygen. This is similar to breathing helium (recommended as the nicest way to commit suicide, after N20) in that one would be quite comfortable right up until one passed out from hypoxia. Death follows soon after.

Also, i've read in boating magazines that undersized Halon systems used in engine compartments can be dangerous. Apparently, if the diesel engine is running when the system fires and there isn't enough Halon to kill the engine, the burned Halon/air/diesel mixture produces some really nasty toxic gasses.

Anyhow.. enough rambling!

Re:Halon systems aren't illegal, but....

[Dahan]

Halon, otoh, is perfectly comofortable to breathe in and out, but will provide no oxygen.

Why does it need to provide oxygen? The air provides oxygen. How come you're not worried that air is about 80% nitrogen, which also provides no oxygen? See, humans don't need to breathe 100% oxygen... we do fine with much less. The advantage of Halon over CO2 is that it does not extinguish a fire by displacing oxygen. It will put out a fire at concentrations of about 5%, leaving plenty of oxygen to breathe.

the burned Halon/air/diesel mixture produces some really nasty toxic gasses.

And a fire doesn't? If you have an undersized system installed, you're gonna have problems in a fire anyways. With a proper system, the small quantity of toxic gasses produced by the Halon decomposition before the fire is extinguished (which is a fraction of a second--Halon systems have been used for explosion suppression) is much preferable to the large quantity of toxic gasses and heat produced by a fire.

About as good as it gets with only two sites...

[Kjella]

I mean seriously, each tower collapsed because it was hit by its own plane. If one tower had been in NY, one in California and both were still hit by a plane each, the result would be exactly the same.

The lesson should be: Primary back-up is a very good start, but secondary/tertiary back-up is the thing if it's that critical.

Kjella

Re:About as good as it gets with only two sites...

[operagost]

Considering how several other buildings in the WTC were damaged or destroyed just from debris, I'd say that even if only one tower had fallen, there's a good chance everything in the second would have been trashed.

Really, not having your backups in close proximity to the data center IS good policy.

Re:About as good as it gets with only two sites...

[Fencepost]

No, you want two geographically diverse sites in case of natural disasters or other events that keep you from reaching (physically or electronically) one.

I'm sure that there's been lots of study of this, but I suspect that a good distance is 1-2 hours drive. Far enough to avoid most of the impact from things like a big chunk of the city being shut down for weeks, close enough to get to (with some inconvenience) if necessary.

In the Chicago area it might be something like downtown Chicago and Schaumburg, Naperville, Aurora or even Rockford (at the 2-hour mark).

Re:About as good as it gets with only two sites...

[0x69]

Bullcrap. BOTH towers had been exposed as very vulnerable to ONE bomb in the underground parking garage long before. BOTH towers were connected to the same small pieces of the electrical, telecom, water, sewer, gas, etc. systems, and Chicago demonstrated the dangers of that a few year ago.

The only [gag] good [retch] reasons to have the backup in the other tower are spelled "lazy" and "stupid".

Re:No Halon?

[lamj]

Depends, if the fire starts inside the NOC then there's a high chance that Halon would extinguish the fire but if the fire started elsewhere then spread to the NOC, most likely the fire suppression systems are not designed to handle that.

Moreover, Halon system are no longer installed (globally) since 1987 (Montreal Protocol) due to its CFC damaging effect. Most systems already installed are replaced by FM-200. Water, Argon, FE-13, Inergen and a few others are all possible replacement.

Disaster Recovery plans

[ACK!!]

This brings to the forefront of my mind the fact my organization is currently devising disaster recovery plans for our new building.

The NIH Computer Center Disaster Recovery Plan is available online somewhere in pdf format and provided a good deal of insite.

Does anyone else have good hints on texts and outline for good disaster recovery plans?

___________________________________________

Re:Disaster Recovery plans

[wiredog]

The Gartner Group put some stuff up after 9/11. Most of it is common sense.

Do full backups weekly, store copies offsite. Incremental backups daily, copies offsite also. If you can afford it (or can't afford any downtime), have emergency backup hardware (enough for minimal operations) in an offsite storage facility. Old hardware that would otherwise be thrown out is good for this (remember, it's for an emergency). Have a supplier who can get replacement hardware to you in a hurry (so you can get off of those old 90 MHz Pentium servers).

The most vital part of the plan, after backups, is good insurance. If the building burns to the ground Monday morning, you want to be able to call the insurer Monday Noon, and have the check in hand Tuesday morning at the latest.

These recommendations do not cover disasters such as 767s flying into the building and killing all the sysops. Earthquakes dropping the building on the same. Etc. The people are the most important part of any company and, if too many of them are lost at once, the company probably is lost too.

Unless you have really good (and expensive)insurance which can provide enough funds for you to hire new people, get them trained, and keep the company solvent while you do so.

The world's most warped error message...

[meringuoid]

... finally gets to make an appearance in earnest.

'lp1 on fire'

Ah HAH!!!

[wiredog]

Some damn fool wrote an assembly program that used the dreaded HCF instruction, didn't they?

Oops.

[Effugas]

Maybe I shouldn't have released that code after all...

Yours Truly,

Dan Kaminsky
DoxPara Research
http://www.doxpara.com

Debian is working on recovering

[StormCrow]

Quote from debian-devel-announce

At around 8 this morning (local time) a fire started in the computing facilities of Twente University. This affects Debian, since one of our servers (satie) is hosted there. At this moment it seems very likely that the machine can not be recovered from the fire.

The following services are currently down as a result of this:

security.debian.org
non-us.debian.org
nm.debian.org
qa.debian.org

We are working to restoring these services on another machine and hope to have things in mostly working order by tomorrow. Security advisories are still available at http://www.debian.org/security/

Wichert.

it was the printers!!!

[manual_overide]

lp0 on fire!!!

(sorry, couldn't help myself...)

Re:In other news

[MonoSynth]

fact is that the UT is responsible for 30% of the daily internet-traffic in the Netherlands :)

Breathe Quite Comfortably?

[Guido69]

The advantage of Halon is that you can breathe quite comfortably in an atmosphere that will not sustain fire. CO2 works just about as effectively but will not sustain life.

You've obviously never been in a Halon dump. Either that or you consider burning in your lungs to be "quite comfortable". Not to mention that if you're standing under one of the discharge nozzles at a dump you can get a nasty case of frostbite.

I've personally been through two 1211 dumps and had to enter a computer room and drag staff out after an FM200 dump. It takes about two days to completely stop coughing.

Re:I predict...

[gmack]

I predict they will use their backups and repoint the URL to another location.

It's not as if the debian project didn't have the domain properly setup with 3 diffrent nameservers in 3 completely different locations.

They may even have a working mirror...

Students!

[flippet]

The Register has a story about it. I like the paragraph at the end...

Twente's high-speed network was originally constructed to provide students with access to high-speed Internet access for their studies. It soon became one of the major hubs for peer-to-peer exchange programs like KaZaA. This perceived misuse of Internet resources caused former Dutch education minister Loek Hermans to comment: "It would be nice if the students at Twente University would use their fast connections for information and education purposes, instead of downloading huge amounts of porn."

Phil, just me

Sad

[ledow]

Maybe it was a flame war between students? Were they overclocking? The dangers of using FireWire. Were their harddrives Quantum Fireballs? (They are now) Is this what you get when you try to hot-swap them?

Other sad jokes will no doubt follow.

translation..

[mikevdg]

This is a translation from the telegraaf (which, btw, isn't the best newspaper in holland, but anyway..)

"University network down from fire"

"ENSCHEDE - A fire in the computing center from the university of Twente (UT) in Enschede has caused a few 10's of millions of euros damage on Wednesday.

The fire, which broke out about 8:00am, has disabled the universities network. There were no injuries. The fire department has cleared out several buildings in the nearby facinity. The IT departments of the UT and a part of the business studies department were housed in the building where the fire broke out. The building was three stories high.

The fire department fought the fire with several dozen people and estimated that they would need the whole day to get the fire under control. There are, until now, no dangerous substances released. The cause of the outbreak is not yet known.

According to Van Vught, a backup of the network's data has been made. "All data have been safely stored". Because of exams, there were no classes at the time. There were also few students on the campus on wednesday. Exams on wednesday have been cancelled.

Staff at the UT want to set up a temporary network, which can take several days. Thousands of staff and students can not make use of the campus network until a temporary network has been installed.

One on-looking business studies student is shocked (?): "incredibly sad. It means that I can't do my assignments. Doubtless, a temporary building will need to be set-up to help the situation". The Saxion Polytechn in Enschede has made room available.

The web-site from the Enschede city council is not available because of the fire. The council used the servers from the UT. The Saxion Polytech is also without internet access for the same reason."

Okay, this translation is a wee-bit too literal, but it's understandable.

Michael.

The cause of the fire?

[Lethyos]

So wait, the University of Twente NOC caught fire. Why? Was there a story posted on Slashdot's front page that linked to a server at their location? Or was this fire caused by something other than a hardy slashdotting?

Re:The cause of the fire?

[poot_rootbeer]

They should have just let Milton have his stapler...

You mean HiveCache (formerly MojoNation)?

[Fencepost]

That's exactly what the Mojo Nation folks are doing now. Info at http://www.mojonation.net/.

They Needed Low-Tech Fire Protection

[0x69]

Our ISP bought an old legal office building for their HQ and colo facility. The place was built with file rooms to safeguard tons of irreplaceable paper documents - imagine thick concrete walls & ceilings, with heavy steel fire doors, rated to preserve the contents through an EVERYTHING-else-burned-to-the-ground fire.

Critical stuff is spread between the file rooms, with metal conduit, etc. protecting the few small holes they added for wiring.

Steel & reinforced concrete aren't quite obsolete.

Re:Class IV construction

[haligan]

As a former fire fighter and fire inspector I must comment that yes the construction may have been type IV , but like most people , Architects included, you are forgetting that the contents are mostly flammable, (ie carpet, paint, some ceiling tiles). Mike B.

Do NOT stay at ground level with HALON!

[billstewart]

Halon is heavier than air, so if you're somewhere there's been a Halon dump, DO NOT HIT THE GROUND! Walk/run as upright as you comfortably can. The usual "stay low" advice is because hot air and smoke are lighter than regular air, so the regular air stays near the floor.

Halon (or at least most of the Halons used for fire suppression; not sure about all of them) is non-toxic, though it'll get you a bit high, like nitrous oxide or most solvents, so being stuck in the stuff won't injure you quickly (except from flying objects that were blown around by the gas pressure.) However, it's no substitute for oxygen, and you'll probably be wanting some oxygen real soon now. If you can remember not to breathe in the stuff, try not to breathe deeply, because there's more oxygen left in your lungs than the stuff you'll be breathing in, and unfortunately, while your body can generally figure out not to breathe in water or hot smoke, it's not as good at realizing that near-room-temperature inert gases aren't very useful. Mostly, don't worry about it - find a safe door to run for and run for it, and do some deep breathing once you're outside, and try to close the door behind you.

Things are being restored slowly

[wichert]

Lots of people are helping to restore the lost network and computing facilities. People from XS4ALL drove from one side of the country to bring spare junipers and other equipment. The campus is already back online thanks to Virtu. Mail for student is being stored at a new backup MX server courtesy of Terena and zedz.net

Debian is restoring the lost services on klecker. At this moment qa.debian.org is up and running and the non-US and security archives are available as well, although their backend systems have not been restored yet.

Valuable lessons have been learned though: it is very useful to have machines on standby where you can switches services to when needed. Having backups of important data is also really useful (and we could have done a bit better at that. UTwente apparently has good off-site backups of its own data though). And having good insurance is also definitely useful.