University of Twente NOC Destroyed

JanJoost writes "Around 08.00 CET today the University of Twente Network Operations Center, which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org, caught fire. The UT, which hosted the HAL in august last year is completely unreachable and is not likely to come back up any time soon. The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings. More information can be found at the Telegraaf, Planet Internet and Twentsche Courant. Pictures can be found here and here. It's a shame to see a great infrastructure go down in flames like this."

Comments

More info

[Strike]

From debian-devel, here's a slightly (only slightly) more informative blurb

Re:More info

[Stonehead]

Hey, I posted that to debian-devel. Scary to see it get Slashdot headlines, since this posting from Wichert Akkerman himself is more 'official'.
He's now probably busy setting up klecker.debian.org as the next security.debian.org host. Don't get yourself trojaned, please people, don't panic and just wait for the official Debian announcement that everything has been fixed again. Or play around with inofficial mirrors like these, and there are more. But I feel a bit stupid myself, because - unlike Wichert - I have done nothing myself except forwarding the news and act like a karma whore.

Well Damn...

[jhines0042]

... I hope nobody was hurt.

After that, I wish them luck getting back online.

Re:Well Damn...

[Koos]

I hope nobody was hurt.

All news reports say people were evacuated in time from the building and nobody got hurt.

Funny, I was in this building a few times during the buildup and cleanup of HAL2001. Feels strange now.

Photo's

[fearlezz]

More images can be found on http://www.bsdfreaks.nl/files/brand.htm

In other news

[roguerez]

Capacity usage of the world largest interconnection lines roughly halved today.

The cause of this is thought to be a steadily decline in warez/divx/mp3 traffic. The underlying reason for that is still unknown.

Re:In other news

[MonoSynth]

fact is that the UT is responsible for 30% of the daily internet-traffic in the Netherlands :)

Re:In other news

[JohanV]

Nonsense. UT used to be the largest user in the Netherlands until about 14 months ago. After that, they were surpassed by Delft University of Technology, which until the fire today used about 50% more bandwidth as the UT. And even they aren't the biggest anymore, because the University of Utrecht recently became larger than them.
In total, about 20.000 rooms for individual students in the Netherlands are responsible for over 1 Gbps of traffic. This is non-stop (i.e. during the evening it is closer to 2 Gbps), and over 90+ % is outgoing traffic from 'unknown ports'.

I guess forcing the RIAA and the MPAA to change their business models the hard way counts as innovation nowadays :)

Re:In other news

[osu-neko]

...and over 90+ % is outgoing traffic from 'unknown ports'.

Umm, so? Outgoing traffic is almost invariably from a random port > 1024. That's how TCP connections are generally done.

Re:In other news

[Mnemia]

Except that outgoing traffic is normally low-bandwidth. When you use a web browser the outbound traffic is very small compared to the inbound traffic. So the only thing that could generate that much outbound traffic is servers, and since most "legitimate" server applications use low numbered ports, we're likely talking about P2P servers. I think that was what your parent poster was implying (though that ratio seems like it's got to be an exaggeration to me).

now the engineers come out...

[ravidew]

..to see how this could be prevented in the future. How much fire protection do NOCs owned by the big boys (Verio, WorldCom) have? Offsite backups, too, I hope?

Re:now the engineers come out...

[Yo Grark]

After 911 all the big techfirms created DRA's (disaster recovery area's) in case anything major happened.

I had no less than 1000 companies call asking how to handle licensing and DRA's.

I'd be surprised if they didn't have one, it's pretty much standard practice comercially....

Yo Grark

- Canadian Bred with American Buttering

Re:now the engineers come out...

[rleyton]

Whilst I'm no expert in fire supression systems, I have studied a few data centres in my time as a Senior Sys Admin for a number of companies. Most (decent) NOC's have a fire *supression* system based on gas or (more likely these days) some form of liquid (gas has the nasty side-effect of potentially killing humans).

These are for small fires that can be contained within a data centre, ie. a computer catching fire or emitting smoke. Really good systems are very localised (racks or cage specific). A big fire just isn't going to be stopped by such a system.

I haven't read too much into this particular incident (ie. not at all), but my initial thought was that something more serious must have happened (well, duh!), perhaps a fire outside of the main suppression system (outside of the raised floor area?). Or perhaps the paint on the walls/carpet wasn't fire resistant and just took hold very quickly.

Or a large initial fire (gas leak?) that just didn't die down when the supression system kicked in. Maybe the type of fire (again, gas? oil?) didn't die down because Data Center supression systems presumably focus on electrical fires.

A well, just my 2p's worth.

Re:now the engineers come out...

[Martin Blank]

Most NOCs I've been in have an oxygen-deprivation gas that's dispersed in the case of a fire (after a series of highly visible and audible warnings). The one I'm in right now seems to have gone the inexpensive (and arguably safer) way of a two-stage dry-pipe water sprinkler system where the pipes are usually dry (empty). In case of a smoke detection above a certain level, the pipes are "charged" (filled with water), and if the heat gets above a certain level, the sprinklers go off. Basically, the decision was made that if there's a fire, the equipment will probably need replacement anyway, so why not use sprinklers?

Personally, I'm not too sure of this route. I can understand it, from an environmental and human-safety perspective (the gasses eat the ozone layer, and you *really* don't want to inhale stuff that ties up oxygen at those rates), but if it ever happens here, it's going to be an awfully hurried mess to get everything back up and running in a reasonable timespan, even if the fire is only a small one that doesn't destroy the building.

Re:now the engineers come out...

[Angry White Guy]

Offsite backups are one thing, but in a case of sheer devistation (such as this), it really doesen't help too much. They have to find a building, fiber, computers, cabling, everything to replace it. Backups, although an important part in the puzzle, are ultimately the last, centermost peices.

After all is said and done, the U still has to deal with insurance companies, getting specs for colocated equipment, etc. It's not an easy process (if you've dealt with your insurance company not wanting to pay for a dent in your car, imagine telling them that the dent would cost millions to repair) and I doubt that they will be up and operational in less than 6 months.

I just find it ironic that they put all of their fault-tolerant redundant systems in the same building, only to have it burn down. Distributed computing should mean more than just two computers sitting right next to each other.

Re:now the engineers come out...

[Nos.]

We have a similar two stage system as mentioned above. When the installers came by a year later for a test, they must have "forgotten" how they hooked it up. Before the sprinklers actually start spraying water, the breaker that runs the power to that room is automatically flipped. The UPS we purchased for the servers is also hooked up so it doesn't do a safe power down, it just powers down.

Well, they came in to do the test, everything but actually spraying water, and managed to flip the breaker. It was interesting to note, that only a few of the machines actually powered off. Some are on smaller single machine UPSs, so those stayed running. The most expensive piece of equipment in there, a Nortel Telephony Switch which handles 6 PRI lines (almost 150 ISDN lines) is on battery backup, and kept running.

All in all very little powered down. In the even of a real fire, we would have had sparks flying everywhere. What did management do when I mentioned this to them? That's right, nothing.

Re:now the engineers come out...

[AlphaInsight]

Actually FM-200 and Inergen are the current replacements for Halon. Non Ozone-depleting and won't kill you if you're trapped in a NOC. Just lowers the avaliable O2 in the air. You'll get out of breath easily, but as long as you stay calm you won't have to worry.

Re:now the engineers come out...

[Anonymous Coward]

"gas has the nasty side-effect of potentially killing humans"

This hasn't been a real problem since the original Halon systems. There is generally a long protracted countdown (15-20secs is a long time when talking about fires.) even on the newer systems such as inergen. Potentially dangerous gases have breathing apparatus in the protected area just in case. (ie: someone is electrocuted, fire, can't move person, grab breathing mask, put on their head (or grab two and move them) and leave room.)

Though these systems are very expensive. We have the largest data center (by floorspace) in Queensland (multi-storey). When someone "mistook" the "release without interrupt" button for the exit door release, it cost AUS$10-12 thousand to replace the gas(inergen). Of course now it's _much_ harder to make this mistake. (Involving a hammer and a glass cover).

But then again, the person in question was known to play indoor cricket between the mainframes. I'm sure that a fast moving ball had absolutely nothing to do with tripping an inset switch on the wall.

Re:now the engineers come out...

[CaptainZapp]

NOC's have a fire *supression* system based on gas or (more likely these days) some form of liquid (gas has the nasty side-effect of potentially killing humans).

When I worked for DEC they actually had sprinkler systems in their data centers (I couldn't verify first hand that every data center had sprinkler, but I was told so).

The reasoning was, that even if they was one big, bad, evil, wicked water damage in a data center, that the company considered this to be better then one killed employee by gas extuinguishing systems.

That was in the early 90ties

Re:now the engineers come out...

[Mysticalfruit]

Which DEC building did you work in? The company I work for is in the Powdermill Rd. Building and our Lab's are still all sprinklers. However the way the system is setup, as soon as a sprinkler head goes off, the power shuts off.

Re:now the engineers come out...

[Martin Blank]

As was mentioned by someone else, these systems trip the power circuits before they go off. Unlike one of the other posters, though, we have no alternate UPS systems aside from the phone switch battery backups, which are kept a couple of feet off the ground in water-proof containers to minimize the chances of sparks from short circuits.

Re:now the engineers come out...

[Lumpy]

Ahem... flooding a room with carbon dioxide does nothing to the environment or hurts you other than oxygen deprivation. yet this is the system in my server room.. one button or the temprature sensor goes off.. the doors automatically close (steel fire doors) and the room is flooded with C02.

HALON also did not eat up oxygen... it displaced it. and it's msds sheets also shown the human hazard to be minimal other than oxygen deprivation.

there are no oxygen consuming fire supression systems on the market. it's displacement or smothering or temperature reducing (water).

Re:now the engineers come out...

[Stephen Maturin]

-- Which is why most LARGE companies generally do not maintain their own backup data center for their business continuity plan, but lease space/capacity from companies like SunGard, which provide offsite storage & backup data centers.
Even with scheduling regular business continuity plan tests & sending people offsite for a week or so, these plans are cheaper than maintainig a 2nd data center.

Re:now the engineers come out...

[arri]

I am not doubting that these new gases will allow you to survive but I've been too close for comfort to a fire in a Chemistry lab and without adequate training (not in the use of a fire-extinguisher but in the behaviour in case of fire) the last thing I did was "stay calm"... I just hope that places which install this system give adequate training to their people as to what can be done.

Even the obvious like "lie low as smoke rises and there is more oxygen at ground level" and to "try and cover your mouth & nose". At least, this is what I learned from that incident, sadly after the fact.

Re:now the engineers come out...

[buckeyeguy]

Our server room also has a water sprinkler system... due to less-than-forward-looking regulations in the dumb little suburb we are located in, all areas of the building HAD to have them, regardless of any other fire-suppression we might have paid for. Hope no one lights a match...

Re:now the engineers come out...

[rosewood]

The key is insurance moneys which will buy them all new toys and we will be better off then when we started.

Re:now the engineers come out...

[EMH_Mark3]

Heh. You're trapped in a room filled with fire & smoke, the oxygen is being sucked out of the room and you want to stay calm? Sure, no problem :)

Re:now the engineers come out...

[strictnein]

You'll get out of breath easily, but as long as you stay calm you won't have to worry

And, as we all know, there's nothing more calming than being in a room that's on fire and having some funky gas poured into the room that makes it harder to breath

Re:now the engineers come out...

[kapelski]

I have 20 some servers in UUNet's (WorldCom's) Ashburn, VA datacenter (where MAE East is located). When I asked them about fire suppression they told me they use a two stage water system that is localized above many many zones throughout the data center. They said the idea is that since the datacenter is so large and they have so many customers with varying SLAs, their goal is to put the fire out at its source before it can spread, while attempting not to affect the operation of other cages/racks. In other words, apparently they don't want to saturate the whole datacenter, just the location of the fire. If something catastrophic happened that would affect the whole of such a large building, then pretty much everything is a loss anyway. Nevertheless, I trudge out there once a week to take backup tapes to an offsite vault...

I also asked them what happens if a plane crashes on takeoff from Dulles airport (only a few miles away and where the plane that hit the Pentagon took off from) and hits the datacenter, and they said they had the flight path altered when they built the datacenter to minimize the chance of such a thing. They didn't admit to it, but I bet that that building is hardened, too.

Re:now the engineers come out...

[DunbarTheInept]

When you can't get oxygen from the surrounding air, the worst thing in the world to do is take a breath. You will survive much longer by fighting the urge to take a breath, and instead just hold the air already in your lungs in place until you can get out. You don't convert all the oxygen in your lungs into carbon dioxide with each breath - far from it. You just convert a portion of it, so the air you expell just has a smaller percentage of oxygen than the air you breathe in. But there's still quite a bit there that never got converted. (That's why mouth-to-mouth breathing can help someone - the air you breathe out still has enough oxygen in it to be a lot better than no air at all.)

But if you breathe that air out and breathe oxygen-less air in, you will pass out very fast. Most people have the misconception that you can do without oxygen for a minute or two before dying. That's not true - your body needs to consume fresh oxygen at a continuing rate just to function at all, it's just that your lungs can HOLD a small supply of oxygen to supply this need for a minute or two. Get rid of that oxygen by breathing it out and replacing it with oxygenless air, and you're going to pass out in just a few seconds, and be dead shortly thereafter.

And the worst part is you won't FEEL like anything is wrong. Your body is unable to measure the level of oxygen in your lungs. Instead your body senses the level of carbon dioxide in your lungs. As the by-product of normal breathing, when carbon dioxide has built up enough, that indicates you've converted a lot of oxygen and it's time for another breath. This is what triggers the automatic involuntary breathing that takes over when you stop thinking about it. This is also what triggers the panic feeling that you get when you know you need air. Your lymph nodes detect too much carbon dioxide and start sending the panic signal to your mind. What this all means is that if your body isn't exchanging oxygen for carbon dioxide, your body doesn't even realize it's asphixiating. If there's no oxygen in your lungs to start with, then there won't be any carbon dioxide building up in the lungs, and you will feel no sensation of needing a breath at all. You'll feel just fine for a few seconds and then *poof* you're gone as the blood going to your brain runs out of oxygen and your brain activity just plain stops.

So if you're ever in a halon gas system when it goes off - DO NOT BREATHE. Just hold whatever breath happens to already in your lungs and get out. The instinct is to hold your breath by first inhaling your lungs full and THEN holding it, but that's the worst thing you could do, as explained above. The tricky part is remembering to override that instinct.

Re:now the engineers come out...

[CynicTheHedgehog]

The FM-200 guy that trained us on the use of our fire-suppression system said that FM-200 causes an endothermic reaction with the fire--effectively sucking the heat out of it. He said it didn't do anything with the oxygen in the room, and you could stay and play in it as long as you wanted, provided the fire wasn't out of hand (FM-200 reactions produce some lung irritants with hot fires), but by then you'd be more worried about the fire itself.

Re:now the engineers come out...

[DunbarTheInept]

C02 is a poison to the human body if you have it in high concentrations. (That's the big air problem they had on Apollo 13. They had enough O2, but not enough filtering capacity to get rid of the CO2.) How quickly it would kill you depends on how diluted it is with other gasses in the air. the small percentage of it in the earth's atmopshere isn't enough to kill you, obviously. But being in a room full of it is NOT a good idea.

And CO2 has the very very nasty side effect of tricking your body into thinking it's choking and needs another breath whether it really does or not. (Which was another problen on Apollo 13 - the high level of CO2 was causing the crew to breathe way too fast.) Your body is unable to detect oxygen in the lungs. It only detects CO2. CO2 triggers your automatic involuntary breathing, and it triggers your panic feeling when you think you're asphixiating. Get excess CO2 in your lungs and you will be filled with an almost overwhelming urge to take another breath, which if you are in a room with lots of CO2, will make you have even more CO2 in your lungs and make you have even more physcal desire to take another breath, and so on. This is the worst thing to do because you are getting rid of what little oxygen is left in your lungs and replacing it with oxygen-less air.

Re:now the engineers come out...

[frost22]

but you aren't going to die from the carbon dioxide... you die from lack of oxygen as i said in the post...

Sorry, wrong.

Carbon dioxide is lethal up from about 30% air volume. No matter what the rest of the air consist of - even if that contains comfortably much oxygen. As far as I remeber my paramedic training, it's called an acidosis - essentially, CO2 at high percentages soures your blood, to a point where your metabolism cant handle it any more.

Re:now the engineers come out...

[frost22]

If NL Universities are like German ones, they don't have any insurance, as mandated by law.

The key idea is the state itrself is the largest solidarity community anyway, so paying insurance is waste of state funds.

Needless to say, in reality, you are fucked when something big happens to your computers. Maybe in next years Budget, you get some additional money to replace them. Or not.

Re:now the engineers come out...

[Bios_Hakr]

As long as they use distilled water in the system, it'd be ok. Water itself isn't conductive, just the impurities in it. You could flood the entire NOC and still have live traffic.

Of course, corrosion would be a major prob after the water was shut off.

Re:why does this matter?

[Sc00ter]

can you read?

"which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org "

security.debian.org and non-us.debian.org are GONE (well, burning, hopefully data can be recovered). Yes there's mirrors, but it still sucks.

Vunerability

[e8johan]

This shows the vunerability of putting all computers in one building. To have a safe network one needs to spread (duplicate) the information over several computer at several locations. How far apart these locations has to be is depending on how important you data is.

It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup. By that I don't only mean that tape that always seems to go missing when needed, but multiple sites (or at least buildings) that provide redundancy.

Re:Vunerability

[Psiren]

To have a safe network one needs to spread (duplicate) the information over several computer at several locations.

What a stunning idea. Perhaps they should call it the internet... ;-)

Re:Vunerability

[xyote]

When iSCSI becomes real you are going to see a lot more raid-1 mirroing with one half of the mirror at a remote location. Also using remote tape drives for backup. Currently if you do store your backup tapes offsite, there is still that logistical problem of getting them offsite in a timely manner. Things like if you have 10 partly full DLT tapes at $$ a tape, do you send then offsite right away or do you wait for the next set of backups to fill them up. The former can be quite expensive. Also, since most restores are from most recent backup tapes, sending them offsite right away causes delays in your restores while you wait for the tape to come back. Remote tape backup will solve a lot of these issues.

Re:Vunerability

[nbvb]

TSM (Tivoli Storage Manager) does a great job at mitigating all of those problems:

1) Sending tapes off-site too early -- TSM allows you to have multiple "storage pools" within one library. Basically, you backup your data into one storage pool (set of tapes) and then duplicate it to the off-site copies. Then the DRM (Disaster Recovery Manager) scripts automatically eject the tapes from the library for packaging to send off-site. You don't ship off your on-site copies, so when you need to do a restore, it's always here.

2) Partly full tapes -- TSM has a concept of "migrations" where it moves data between tapes to better utilize them. The internal DB automatically knows where the data is, so you don't have to worry about which tape has which days' backups -- it doesn't matter.

3) Full backupset -- TSM 4.2 introduced "portable backup sets" -- basically, you backup your data into the TSM server as normal, then "generate" a backupset -- takes all the files for a particular node and writes them to a series of tapes which you then eject and store off-site. When you need to recover from those tapes, the server component isn't needed -- locally attached tape on the server and the client piece of TSM is all you need --- speeds up recovery by a few hours!

TSM's a great product. Does lots and lots of great things that other vendors are only now trying to figure out.

Sure, it's pricy, but how much is your data worth?

Re:Vunerability

[coupland]

It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup.

Uhhh, yes there is... I suspect you either know nothing about IT or are fresh out of college. DRP (disaster recovery planning) factors in things such as criticality of data, cost, and acceptable downtime. A university payroll system may need to be back up within 12 hours of a major incident, so in addition to tape backups you might have a failover site. Contrary to your simplistic post, even the richest corporations rarely have failover sites of their own. They simply contract out to a DRP vendor who have these types of machines lying dormant in a glass room, waiting to cut over. On the other hand a university FTP site is probably classified as low risk, low impact. So you would rely on off-site backup tapes and perhaps only restore when you've arranged for an alternate site and taken delivery of new servers. You don't pay millions of dollars to have two glass rooms just so you can have uninterrupted FTP service...

Re:Vunerability

[pellaeon]

Try doing that on a university-wide multi-gigabit capable network on the budget of the average Dutch university. Our universities aren't like M$ in cash, you know. I know, I'm an admin at one myself.

I just hope they're well insured....poor colleagues...

On the upside: they may get a squeaky-clean start when this blows over :-)

Re:Vunerability

[operagost]

Maybe something like this.

800 kilometers should do it.

Re:Vunerability

[soramimicake]

This shows the vunerability of putting all computers in one building.

Actually, the debian.org names affected all point to a single box, satie.debian.org. Your point is still valid, though. Come to think about it, that makes it _more_ valid, as only one box crashing will take down all your services.

Re:Vunerability

[complexmath]

One notable exception being the finance industry. In NYC, most of the larger trading organizations maintain DR sites somewhere else in case of something like, oh, Sept 11. The issue here though is that these companies need a fully capable and connected trading floor rather than just a backup server room, and the short-term loss potential of the industry is such that DR costs are generally quite reasonable in comparison.

Re:Vunerability

[e8johan]

Great news! I hope that you'll recover quickly so that everything will be up and running!

A good reminder..

[Martigan80]

To never keep back-ups in the same physical location.

Re:A good reminder..

[Zocalo]

Or even general location for that matter. A friend of mine did disaster recovery work for IBM after the Trade Towers attack. They had their data center in Tower 1 and their backup center in Tower 2. After six weeks of what was essentially scrabbling through rubble they managed to recover a single spindle. The company concerned became another statistic, and part of an important lesson in DR implementation; safety increases with distance.

Re:A good reminder..

[Martin Blank]

Cantor Fitzgerald, the company hardest-hit in the attack with 700+ employees killed, had just that viewpoint. eSpeed, a subsidiary of CF, had all trading activities back up and running by the morning of Sept 13, due to the multiple locations and real-time backups being done on the network. My understanding is that the time from loss of connection with CF headquarters to full resumption of its trading activity around the world was about 46 hours.

Re:A good reminder..

[Jugalator]

To never keep back-ups in the same physical location.

That's why I use Kazaa to keep backups of my pr0n all over the world!

Re:A good reminder..

[Tom]

absolutely. A local power company I sometimes work with has the following simple policy:

It is not considered a backup until it's stored in two locations at least 10 miles apart.

Re:A good reminder..

[LWATCDR]

A secure Freenet for backups maybe?
encrypt your data and put it on freenet style system?
You could then recover from anywhere on the network.

Re:A good reminder..

[Martin Blank]

My mistake: 47 hours.

Details on the recovery:
http://www.baselinemag.com/article2/0,3 959,36807,0 0.asp

Some information on the scale of what eSpeed handles:
http://wstonline.com/story/mag/WST200110 08S0007

Even Cantor Fitzgerald mentions it in their Q3/01 results summary:
http://www.cantor.com/articles/article11 192001.htm

Re:A good reminder..

[RAMMS EIN]

``safety increases with distance.''
If only life were that simple. Yes, keeping backups accross the country gives better protection against localized disasters than keeping them in the same room, but the increased distance also increases the risk that something happens on the way, and also increases the time required to get things up and running again.

Re:A good reminder..

[athakur999]

I personally don't keep backups. However, I'm sure at least two other me's in parallel dimensions do, so I'm probably good to go.

Hopefully the other me's in the other dimensions aren't counting on me to keep backups. Those idiots.

Re:A good reminder..

[roalt]

To never keep back-ups in the same physical location.

The university made full backups this weekend and the building where the backups are is approximately 3km away from the Computer Centre that burned down.

In fact: this 'backup' building used to be one from the Dutch Bank (say the Federal Bank in the US) and the backups are also stored in the original (giant) safe !

Re:why does this matter?

[fearlezz]

It's the fastest university network in the netherlands, and it's running a lot of mirrors (debian mirrors and others). The most important thing: terabytes of warez. :) (I believe it was the BSA lighting the fire :) )

So much for server areas never burning down

[Martin Blank]

This is something I keep hearing from people, particularly those who balk at paying for upgrades to expensive fire suppression systems. "They're always built so well that there's never been a case of a server room destroyed by fire." I always did doubt that, and now whenever they tell me that, I can point to this.

What a shame.

Re:So much for server areas never burning down

[CuriousGeorge113]

Yes, but did this server room have one of those fancy fire suppression systems? If so, yes, you can point to this an laugh. If it didn't, your friends that always say "Server rooms are never destroyed by fires because of their fire suppression systems." will have the last laugh.

I.E. "See, if they would have had a fire suppression system, this would have never happened."

Re:So much for server areas never burning down

[Plutor]

The fact that "The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings" leads me to believe that the fire didn't start in the server area. Lots of server rooms were destroyed on September 11, for example, but it wasn't the fault of the room's design, or the presence or lack of fire suppression systems. If the whole building is burning down, fire suppression in one room is only going to work until the floor and ceiling collapse.

Re:So much for server areas never burning down

[jhines]

A server room perhaps, they are generally kept very clean.

However, it usually sits next to a room full of cables, and the telco plant. This is flammable. Look at the pictures, very heavy black smoke, so this is very likely.

An old cable plant has lots of PVC and other cables that are not fire resistant the way modern Teflon is, and that will still burn, if it is hot enough.

Re:So much for server areas never burning down

[aaarrrgggh]

The thing that is odd (knowing absolutely nothing about the building in question) is that there typically aren't sufficient materials of combustion in a data center to create such a substantial fire! (Sparing large printers and paper rolls.)

A fuel vault should have foam or CO2 suppression, neutralizing that risk. The building should have a full sprinkler system, to minimize the risk of spread of fire. Hopefully, the computer and tape rooms have Halon/FM200 to extinguish a fire on the first signs of smoke. It is very rare that a building would be completely lost if these components are working properly.

The flip side of it is that the more things that are done to prevent an accidental sprinkler discharge (pre-action systems, double-interlocked pre-action, abort stations, etc.) increase the time for a system to respond, and consequently increase the size of the fire that has to be put out.

Re:So much for server areas never burning down

[aaarrrgggh]

The cables are not fire resistive (normally); they are low smoke. (Unless you are british, in which case they could actually be somewhat fire resistive.) Ultimately, given a high enough temperature everything burns. If a fire lasts more than 2 hours, you expect to loose the building.

After 30 minutes or so, all the cases, finishes, rubber will burn, and that will give you the black smoke, but it doesn't indicate the initial source of combustion.

Re:So much for server areas never burning down

[ReelOddeeo]

Yes, but did this server room have one of those fancy fire suppression systems?

Followup question: Was it a fire suppression system, or was it a Microsoft fire suppression system running Windows CE and Visual Basic?

Re:So much for server areas never burning down

[mmol_6453]

Here in Grand Rapids, Michigan, any cable that passes through the heater system has to be Plenum. And considering that, at Greybar, you have a choice between Plenum and riser-cable, most of the new (as in, last twenty years) telecom wiring in Grand Rapids is fire-resistant. The insulation of Plenum will liquify before it burns...

Re:So much for server areas never burning down

[bad-badtz-maru]

Plenum insulation doesn't liquify any more than regular insulation and it definitely can't be considered fire-resistant, it just gives off less toxic smoke when it burns. Plenum is not really a term that describes a type of insulation, it's a term that refers to a building space that is used as a circulation space for HVAC. This space is typically between the drop ceiling and the roof. When burning, plenum-rated cable will not induce as much smoke into the plenum as regular cable does.

maru

In a related announcement...

[Zocalo]

The University of Twente's attempts to overclock the new AMD Opteron and Nvidia GeForceFX card in the same case are declared a failure. "We certainly won't be building a Beowolf cluster of these..." commented a spokesperson.

Re:In a related announcement...

[OrangeSpyderMan]

UNIX? They're probably not even circumcised! Savages!

You're thinking of that Linux distro with a BSD style ports things aren't you? What's it called again, Gentle, Gentool - Gentile Linux!!! Thats right! :-)

Maybe..

[jedie]

Someone took "firewall" too seriously? :)
(and this isn't *flame*bait :p)

Re:Maybe..

[poot_rootbeer]

http://www.cluebomb.com/comics/cluebomb/cartoon3 .g if

Priorities

[brianvan]

Was anyone killed?

If not, was anyone hurt?

If not, do they have insurance?

If they do... well, I'm sure someone just lost their masterpiece pr0n directory, but otherwise, things like this happen. (ask Hemos) You have to make it through such things. In this case, it was a commercial (educational) building and no one is homeless, so it's less of a tragedy than usual. Let's hope that they rebuild with something better and newer.

That said, I get the feeling that those plumes of smoke really are millions of dollars floating away in the wind...

Re:Priorities

[bzzzt]

No, no and no.

The university doesn't insure itself agains this kind of disaster because they are able to set aside enough money to cover the cost themselves and don't have to pay the insurer's profit.

Now how they are going to deal with this and their financial troubles (they're almost broke) is another issue...

Re:Priorities

[dazed-n-confused]

If they do... well, I'm sure someone just lost their masterpiece pr0n directory, but otherwise, things like this happen. (ask Hemos)

Remind me: when did Hemos lose his masterpiece pr0n directory?

Re:Priorities

[brianvan]

It's a shame I remember these things. I really need to get out more.

Re:Priorities

[ReelOddeeo]

Let's hope that they rebuild with something better and newer.

There is a problem with this. Longhorn is not out yet. We know it will at least be newer than XP. As for better, well, Microsoft has promised that it will be.

last syslog messages from SURFnet routers

[ahu]

lo0.ar5.enschede1.surf.net 3613: Nov 20 07:20:50.927 UTC: %ENV_MON-2-TEMP: Hotpoint temp sensor(slot 18) temperature has reached WARNING level at 61(C)

few seconds later on the local side:
lo0.cr2.amsterdam2.surf.net 1146: Nov 20 07:20:56.458 UTC: %CLNS-5-ADJCHANGE: ISIS: Adjacency to ar5.enschede1 (POS2/0) Down, interface deleted(non-iih)

Re:last syslog messages from SURFnet routers

[Jugalator]

I wonder if they got the Linux "easter egg" error message "Printer on fire"? I guess it would be appropriate for once.

Re:last syslog messages from SURFnet routers

[Anonymous Coward]

ping lo0.ar5.enschede1.surf.net
PING lo0.ar5.enschede1.surf.net (145.145.255.14): 56 data bytes
--- lo0.ar5.enschede1.surf.net ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss :-(((

Re:last syslog messages from SURFnet routers

[CoolVibe]

Those are Cisco syslog messages. No linux easter eggs.

Re:last syslog messages from SURFnet routers

[Jugalator]

Those are Cisco syslog messages. No linux easter eggs.

Of course, but surely they must have had *some* Linux box there that noticed strange things going on in the printer room. :-)

Re:last syslog messages from SURFnet routers

[CoolVibe]

Maybe one with a LM chipset in there perhaps? Who knows. :) A fire in the room would surely increase the mobo temperature. Allright, who's got MRTG or RRDtool graphs?

Re:last syslog messages from SURFnet routers

[jelle]

Network traffic rrdtool graph.

Hope they ...

[zensonic]

... got backups. Only fools ditch the backup procedure!

If the actually have backups it's only a matter of money/time before sites will be up'n'running again.

Re:Hope they ...

[Martin Blank]

I imagine they were well-insured. On the greed side of things, some major server company is going to report a small spike in Q4/02 server sales in Europe.

"Hi, we need a quote for a few hundred new servers and a couple dozen new routers and switches. Oh, and we need them *now*. Hello? Hello...? Dammit! Another sales rep fainted!"

Re:Hope they ...

[LinuxHam]

nah, we can get 'em up with a coupla z800's sometime next week. A few thousand virtual servers in a single 19" rack. They'll get tons of floor space back and maybe this time they'll consider DR to another z800 in another building on campus.

Halon dumps?

[wiredog]

The last time I worked in a NOC, it ran Vaxes, but we had a halon dump. A Big Red Button that got smacked by the last person out of the room. The halon would smother any fire by replacing all the oxygen in the room (which was why the last guy out hit the button). Why wasn't there a halon dump in this NOC? Or, if there was one, what happened?

I hope Debian practices good management principles by having offsite backup.

Re:Halon dumps?

[Entrope]

The blurbs imply ("The fire department has given up every hope on protecting the server area") that the fire originated elsewhere and spread -- fire supression systems are good for fires that start in server rooms, but if somebody puts tinfoil in the microwave next door, the fire may be just too big by the time it gets to the server room.

Re:Halon dumps?

[Martin Blank]

Depending on whether there was a recent overhaul of the system, Halon is now illegal in most countries. Existing systems were grandfathered in, but new systems are prohibited from using it. I'm not sure which gas replaced it, but I understand that it's incredibly expensive, and not quite as effective. Of course, as has been pointed out, if the fire originated (and thus was fed from) elsewhere, the gas would have been useless once the storage tanks were empty.

Re:Halon dumps?

[lamj]

Halon DO NOT replace oxygen in the room to extinguish the fire. It breaks the chain reaction of fire, basically stop the elements of fire to react with each other.

Most scenario would only require a less than 8% of concentration to take out the fire. Under 10% and you can still breath.

Problem of Halon is when over 900 degree C, it breaks down into hydrogen fluoride, hydrogen bromide and bromine - stuff that are toxic. So, run!

Re:Halon dumps?

[Get Behind the Mule]

Halon DO NOT replace oxygen in the room to extinguish the fire. It breaks the chain reaction of fire, basically stop the elements of fire to react with each other.

Most scenario would only require a less than 8% of concentration to take out the fire. Under 10% and you can still breath.

Really? Well, I worked in a server room with a Halon system back in the 80's, and the big scary warning signs said that you wouldn't be able to breathe within n seconds after the stuff fell. So always be prepared to dash for the door.

It was pretty ominous, looking at those two big ol' red buttons on opposite walls with the all-caps, white-on-red warnings next to them, imagining that you could just go wham!, and maybe kill a few people, if they couldn't think & run fast enough. And imagining that in case of a catastrophe, you just might have to do it, and then scream at everybody to get the hell out, wondering whether you saved some lives or ended a few.

Re:Halon dumps?

[nolife]

Halon DO NOT replace oxygen in the room to extinguish the fire.

It cools, replaces O2 and works to break the chemical chain. At least according to this

There is a fire triangle. The points of the triangle represent heat, oxygen, and a combustable material. If you take any one of the three away a fire will go out. Halon and other devices (Baking Soda, AFFF, PKP, etcc) replaces or prevents introduction of O2 causing the fire to go out. Use of water removes the heat, CO2 replaces the O2 and has a slight cooling effect. Halon also breaks the chemical reaction. A problem with O2 removing substances is if the temperature has not gone down, reintoduction of O2 will cause a reflash (like opening the door to early).

http://www.nifc.gov/pres_visit/whatisfire.html
http://www.usbr.gov/power/data/fist/fist5~2/5~2_1. htm
http://www.princeton.edu/~ehs/theater/Appendi xA.ht ml

Re:Halon dumps?

[mmol_6453]

Dry-pipe sprinklers aren't new...I've got a ten-year old Firefighter's training book at home that describes their effectiveness and how to turn them off after the fire is knocked down.

Re:Halon dumps?

[mmol_6453]

Considering that the pyrolysis reaction for each compound is different, I find it's rather difficult to know how to break down each reaction without displacing oxygen--the common component in nearly all fires.

(I say "nearly" because, once it's started, acetylene doesn't require oxygen to continue burning. They get away with using it for welding because the gas escapes the cylinder faster than the fire can creap up the stream. At least, that's how it was explained to me.)

Re:Halon dumps?

[oh]

Again, this is from memory, but isn't acetylene the gas commonly used in whats commonly called and oxy torch? Every acetylene torch I've looked at had two cylenters, and two hoses. One was a black cylender with a white top for the oxygen, and a slightly bigger, red-brown cylender for the acetylene.

see this

Re:Halon dumps?

[mmol_6453]

It requires oxygen to ignite, but does not require it to continue burning. See the firefighting section of the MSDS ("Right To Know") sheet. Look at the entry under "Flammability Limits in Air."

It can only be 2.2% of the gas in a given volume of normal atmosphere, to ignite, but, once started, it can continue to burn even after it is the only volatile gas left.

I predict...

[bLanark]

That in the next year, most new OS documenation and projects will have several URLs built in for all online stuff (gentoo's Portage; TLDP; having a sourceforge page and a mirror, for example).

OS people learn quickly from mistakes like this.

Re:I predict...

[gmack]

I predict they will use their backups and repoint the URL to another location.

It's not as if the debian project didn't have the domain properly setup with 3 diffrent nameservers in 3 completely different locations.

They may even have a working mirror...

Re:why does this matter?

[brianvan]

They're not gone. They're just experiencing a Distributed Combustion Denial-of-Service attack.

Debian Security mirror

[lemmen]

In case one might be interested, Essent mirrored security.debian.org.

You can use debian.essentkabel.com to download the latest security updates (in case you haven't already). Please note this is NOT an official mirror.

Re:Debian Security mirror

[Tottori]

I don't want to besmirch Essent or anyone else, but please don't install security-critical software from a mirror some guy posted on Slashdot. apt-get doesn't check signatures, so if you put a compromised or malicious mirror in your sources.list, it's game over.

Debian haven't released a security advisory since yesterday, so it's deeply unlikely you'll need an update before they get a new security.debian.org online tomorrow.

OK, folks, admit it!

[matija]

Who posted a link to UT's webserver on slashdot?

Re:OK, folks, admit it!

[McFly69]

Sheee... don't tell anyone.. it was me. *hides under bed*

Re:OK, folks, admit it!

[erpbridge]

And why would you do something like that? Think, McFly, think!

More Pictures from the webcam

[CuriousGeorge113]

Here's some statics taken from their webcam. Someone might want to mirror these before they get slash'd. Webcam PicsPics

Also, here's what seems to be the only close up I could find of the fire. pics

Halon is probably illegal today

[mangu]

Effective as it was, Halon was also a major ozone-hole cause, so it has been outlawed in most countries by now.

Re:Halon is probably illegal today

[phil reed]

Wouldn't have helped. A halon system in the server room doesn't do much when the entire building goes up.

Oh Dear....

[Zech Harvey]

Is there anything that could cause this naturally? I mean, judging from the pictures it looked fairly large and out of control...I hope no one caused this purposefully. I've never had to deal with a catastrophe like this, luckily...I send my best wishes of luck and hope to those involved and pray no one got hurt. =(

Re:Oh Dear....

[BabyDave]

I hope no one caused this purposefully.

Cue another round of "Bill Gates/MS evil anti-Linux conspiracy" posts ...

Re:Oh Dear....

[SagSaw]

At my university the computer center is housed in the oldest building on campus. This building also contains a foundry, welding shop, E.E. labs, etc. While I have no idea what the circumstances in this instance were, I imagine many acedemic institutions have similar set-ups. It's not overly difficult so see how a fire could start and quickly spread in such a building.

Dangerous server rooms

[Bazman]

Lets hope the place didnt look like this beforehand:

Dangerous server rooms

Re:why does this matter?

[smnolde]

Time for them to switch to FreeBSD. With mirrors all over the world I don't have to worry about where to go for the source.

However, any fire destroying the building is a *bad* thing and if it's arson I hope they pin his nuts to the wall. I hope the data in the NOC is restored quickly.

Update from SARA (SURFnet NOC)

[mdav]

Here's an update from SARA (that's where I work), the network operator for SURFnet. SURFnet is very busy ordering new equipment and fixing the 2 x 10 Gbit/s lamda's to Enschede. We hope to restore connectivity a.s.a.p. Greetings, Marco

Re:why does this matter?

[The Dobber]

Boy Scouts Of America? Which merit badge would that be?

It must have been Microsoft

[micaiah]

"Witnesses saw a large balding man monkey dancing from the scene and a slim geeky man with glasses trailing behind continuously adjusting his glasses." An in other news....

Re:It must have been Microsoft

[$rtbl_this]

Am I the only one who just pictured the Gentlemen and their henchmen from the Buffy episode Hush? **Shudder** I'll never take the piss out of Steve Ballmer again.

Re:It must have been Microsoft

[Kashif Shaikh]

Yeah, microsoft is that desparate to take out Debian. Or...

It may be a university that didn't want M$ 'donation' so that they could have more windoze courses.

Either way, I think the monkey is trying to do a rain dance!

LFS mirror affected too

[decarelbitter]

The Dutch LFS FTP mirror was also hosted at the University of Twente, which means it's also down. The Dutch HTTP mirror should work properly, since it's outside utwente.nl space.
Last news is that HP (Who supplied most of the UT backbone equipment) is on its way with emergency equipment to have things up and running somewhere tomorrow.

Oh, well...

[larien]

Time to find out if their disaster recovery procedures work... There'll be a heck of a lot of running around trying to get some kind of infrastructure back in place now, I'd imagine.

Re:Lazy commie bastard

[zztzed]

I have yet to see any machine translators that will do Dutch to English.

Re:Halon dumps? -- not if the whole building is in

[earthy]

The fire did not start in the server rooms. What happened was that a fire started in one of the lecture rooms (and a smallish one at that, number A108) that just happened to be in the same wing of the TWRC building that also housed the server rooms (yes, multiple). It then proceeded to take out two entire wings and threaten other buildings nearby as well as the library.

Now, I would *love* to see a halon system capable of stopping that...

Owh, and the fire seems to be under control by now, as evidenced on http://webcam.traserv.com/
(which you can contrast to http://www-infstud.sci.kun.nl/~arthurvl/ispy.jpg (taken at about 09:40 CET this morning)).

Re:Erm...

[Stonehead]

Hmz. Didn't that get translated? There are no casualties and there are full backups (from last weekend, anyway) from all the student and research servers. For the rest, I don't know - probably not. The damage has roughly been estimated on 23 million euro for the building and 17 million euro for everything that was inside. But the building is still smoking - it's only 6.5 hours ago that the fire started.

PGP Keyserver root was hosted by SURFNet

[Erik_]

I believe, the open and distributed network of Keyserver.net (distibuted network of PGP keyservers) was hosted by SURFNet. This network is a distributed network holding PGP and OpenPGP keys. The loss the to UT NOC could have an impact on the updating of key-rings across the keyserver.net network.

Re:why does this matter?

[Angry White Guy]

The arson and destruction of private property badges. If a car burnt as well, they're well on their way to achieving the elusive 'inciting riots' and 'urban terrorist' badges. These last two were proudly held by a select few: Ted Kazinsky, Che Guevera, and Jimmy Carter

security updates mirrored worldwide.

[novakreo]

Actually, Debian security updates are usually also added to the proposed-updates section, which means that they are available on nearly all Debian mirrors worldwide.

Something like
deb ftp://ftp.XX.debian.org/pub/debian/ proposed-updates main contrib non-free
deb ftp://ftp.XX.debian.org/pub/debian-non-US/ proposed-updates/non-US main contrib non-free
(replace XX with your ISO country code) in your /etc/apt/sources.list should work well.

You can find mirrors on Debian's website.

OK

[wiredog]

It started elsewhere. What about other fire supression systems? Sprinklers and the like?

Re:OK

[Hektor_Troy]

Well, not knowing what's in room 108A, I can think of several types of fires, where sprinklersystems won't help.

Magnisium catching fire.
Na (which I can't remember the English name for) stock in the room would also spell disaster.

BACKUP!!!

[Beliskner]

Now we can find out how secure and hardened Debian really is. You are as good as your latest backup.
BACKUPS BACKUPS BACKUPS Off-site! I've had enough of people who are talking about RAID-5 because 5TB tape drive arrays are too slow. Always keep your BACKUPS!

Re:BACKUP!!!

[noahm]

Umm, what does "secure and hardened" have to do with backups? From my experience, secure and hardened is often counter to backups. Every additional copy of the data you have is an additional security concern.

That said, people are currently working on restoring the machine now, and the rough ETA is given as tomorrow. So backups are not an issue.

noah

Re:BACKUP!!!

[Beliskner]

Umm, what does "secure and hardened" have to do with backups?

Secure is meant in the "safe and secure" sense, not the "securely encrypted" sense. Hardened is used in the sense that the Internet is hardened, having been designed to survive a nuclear war.

Dual language is also used in various religous texts - e.g. "convince the non-believers" can mean talk to them, or it can mean kill them all.

Translation of the Twentsche Courant Tubantia

[TeeWee]

[Apologies for the poor translation, no time for a better attempt]

The burning building contains the IT department and a part of the faculty of Business Administration [Closest I could come up with: Bestuurskunde]. The building has three storeys.

The university fears the loss of its network facilities and is trying to save the main computer. According to a spokesman this network is amongst the fastest in Europe. Most classes are expected to resume as normal today. [According to other sources, this is an exam week, meaning few classes anyway. This is also a reason that few students were around so chances were that this also reduced potential casualties]

The fire department is fighting the fire with 25 firefighters and expects to need the entire day to extinguish the fire. No dangerous materials have been released by the fire at this point. At the moment nothing is known about the cause of this fire.

Re:Translation of the Twentsche Courant Tubantia

[CoolVibe]

No dangerous materials have been released by the fire at this point.

Oh? I heard lots of asbestos was released into the air because of this fire.

Re:Translation of the Twentsche Courant Tubantia

[TheLink]

Someone tried to cancel an exam and succeeded?

Halon systems aren't illegal, but....

[mks113]

You can't install new halon systems, but existing ones are still legal (in Canada, anyway).

However, I wouldn't want to have to fill out all the paperwork involved with a discharge! We had an accidental discharge (a leak, I believe) and they decided it was enough impetus to remove the system.

I think they are using CO2 now. The advantage of Halon is that you can breathe quite comfortably in an atmosphere that will not sustain fire. CO2 works just about as effectively but will not sustain life.

OTOH, more recent studies have shown that just because you can still breath in a Halon infiltrated environment doesn't mean that there are no health effects!

I expect there are more CO2 systems going in now, with lots of alarms to make sure people get out before the atmosphere gets unlivable.

Michael

Re:Halon systems aren't illegal, but....

[srpatterson]

We've got a couple of server rooms at work with big (5ft tall) CO2 extinguishers, you have 10 seconds to leave the server room after they activate.

Strangely enough, there are no servers in there, just hubs, routers and comms gear.

Re:Halon systems aren't illegal, but....

[914]

Halon will kill you just as dead, just as quickly, as CO2.

Both gasses extinguish fire by excluding oxygen... which isn't good for people either.

The difference is that one cannot comfortably breath CO2, an involuntary physiological reaction makes it impossible. (next time you open a bottle of Coke, try sniffing the little cloud of CO2 that forms in the top of the bottle)

Halon, otoh, is perfectly comofortable to breathe in and out, but will provide no oxygen. This is similar to breathing helium (recommended as the nicest way to commit suicide, after N20) in that one would be quite comfortable right up until one passed out from hypoxia. Death follows soon after.

Also, i've read in boating magazines that undersized Halon systems used in engine compartments can be dangerous. Apparently, if the diesel engine is running when the system fires and there isn't enough Halon to kill the engine, the burned Halon/air/diesel mixture produces some really nasty toxic gasses.

Anyhow.. enough rambling!

Re:Halon systems aren't illegal, but....

[Dahan]

Halon, otoh, is perfectly comofortable to breathe in and out, but will provide no oxygen.

Why does it need to provide oxygen? The air provides oxygen. How come you're not worried that air is about 80% nitrogen, which also provides no oxygen? See, humans don't need to breathe 100% oxygen... we do fine with much less. The advantage of Halon over CO2 is that it does not extinguish a fire by displacing oxygen. It will put out a fire at concentrations of about 5%, leaving plenty of oxygen to breathe.

the burned Halon/air/diesel mixture produces some really nasty toxic gasses.

And a fire doesn't? If you have an undersized system installed, you're gonna have problems in a fire anyways. With a proper system, the small quantity of toxic gasses produced by the Halon decomposition before the fire is extinguished (which is a fraction of a second--Halon systems have been used for explosion suppression) is much preferable to the large quantity of toxic gasses and heat produced by a fire.

About as good as it gets with only two sites...

[Kjella]

I mean seriously, each tower collapsed because it was hit by its own plane. If one tower had been in NY, one in California and both were still hit by a plane each, the result would be exactly the same.

The lesson should be: Primary back-up is a very good start, but secondary/tertiary back-up is the thing if it's that critical.

Kjella

Re:About as good as it gets with only two sites...

[operagost]

Considering how several other buildings in the WTC were damaged or destroyed just from debris, I'd say that even if only one tower had fallen, there's a good chance everything in the second would have been trashed.

Really, not having your backups in close proximity to the data center IS good policy.

Re:About as good as it gets with only two sites...

[Fencepost]

No, you want two geographically diverse sites in case of natural disasters or other events that keep you from reaching (physically or electronically) one.

I'm sure that there's been lots of study of this, but I suspect that a good distance is 1-2 hours drive. Far enough to avoid most of the impact from things like a big chunk of the city being shut down for weeks, close enough to get to (with some inconvenience) if necessary.

In the Chicago area it might be something like downtown Chicago and Schaumburg, Naperville, Aurora or even Rockford (at the 2-hour mark).

Re:About as good as it gets with only two sites...

[0x69]

Bullcrap. BOTH towers had been exposed as very vulnerable to ONE bomb in the underground parking garage long before. BOTH towers were connected to the same small pieces of the electrical, telecom, water, sewer, gas, etc. systems, and Chicago demonstrated the dangers of that a few year ago.

The only [gag] good [retch] reasons to have the backup in the other tower are spelled "lazy" and "stupid".

Re:About as good as it gets with only two sites...

[gaudior]

I wish some of the downtown outfits would put some data centers out here in Rockford. I need a job.

Re:No Halon?

[lamj]

Depends, if the fire starts inside the NOC then there's a high chance that Halon would extinguish the fire but if the fire started elsewhere then spread to the NOC, most likely the fire suppression systems are not designed to handle that.

Moreover, Halon system are no longer installed (globally) since 1987 (Montreal Protocol) due to its CFC damaging effect. Most systems already installed are replaced by FM-200. Water, Argon, FE-13, Inergen and a few others are all possible replacement.

Re:English, you insensitive clod!

[Jugalator]

Hey how about a link in English for those who are unfortunately unilingual?!?

There are some news at The Register.

Disaster Recovery plans

[ACK!!]

This brings to the forefront of my mind the fact my organization is currently devising disaster recovery plans for our new building.

The NIH Computer Center Disaster Recovery Plan is available online somewhere in pdf format and provided a good deal of insite.

Does anyone else have good hints on texts and outline for good disaster recovery plans?

___________________________________________

Re:Disaster Recovery plans

[wiredog]

The Gartner Group put some stuff up after 9/11. Most of it is common sense.

Do full backups weekly, store copies offsite. Incremental backups daily, copies offsite also. If you can afford it (or can't afford any downtime), have emergency backup hardware (enough for minimal operations) in an offsite storage facility. Old hardware that would otherwise be thrown out is good for this (remember, it's for an emergency). Have a supplier who can get replacement hardware to you in a hurry (so you can get off of those old 90 MHz Pentium servers).

The most vital part of the plan, after backups, is good insurance. If the building burns to the ground Monday morning, you want to be able to call the insurer Monday Noon, and have the check in hand Tuesday morning at the latest.

These recommendations do not cover disasters such as 767s flying into the building and killing all the sysops. Earthquakes dropping the building on the same. Etc. The people are the most important part of any company and, if too many of them are lost at once, the company probably is lost too.

Unless you have really good (and expensive)insurance which can provide enough funds for you to hire new people, get them trained, and keep the company solvent while you do so.

Re:Disaster Recovery plans

[noz]

"...good insurance. If the building burns to the ground Monday morning, you want to be able to call the insurer Monday Noon, and have the check in hand Tuesday morning at the latest."

This is good advice. Sites need insurance, but demanding payment in under 24 hours is not reasonable. If insurance for fires is unconditional, then within one to five days is reasonable, but then again, many people will have conditions in their insurance contract that may either be void in certain circumstances, or simply require investigation under others.

Yeah

[wiredog]

It might have been a chemistry lab with willy pete (white phosphorus), sodium (Na), or something like that. Or a gas main could've blown.

In Maryland a couple days ago several businesses in the same building burned to the ground after a medical supply company caught fire. The company supplied oxygen tanks.

The world's most warped error message...

[meringuoid]

... finally gets to make an appearance in earnest.

'lp1 on fire'

Ah HAH!!!

[wiredog]

Some damn fool wrote an assembly program that used the dreaded HCF instruction, didn't they?

Re:Ah HAH!!!

[Ziviyr]

The 64 bit HCF must be pretty powerful.

(though I really am a bit sad to hear such well purposed hardware combust)

Oops.

[Effugas]

Maybe I shouldn't have released that code after all...

Yours Truly,

Dan Kaminsky
DoxPara Research
http://www.doxpara.com

Comment removed

[account_deleted]

Comment removed based on user account deletion

Debian is working on recovering

[StormCrow]

Quote from debian-devel-announce

At around 8 this morning (local time) a fire started in the computing facilities of Twente University. This affects Debian, since one of our servers (satie) is hosted there. At this moment it seems very likely that the machine can not be recovered from the fire.

The following services are currently down as a result of this:

security.debian.org
non-us.debian.org
nm.debian.org
qa.debian.org

We are working to restoring these services on another machine and hope to have things in mostly working order by tomorrow. Security advisories are still available at http://www.debian.org/security/

Wichert.

it was the printers!!!

[manual_overide]

lp0 on fire!!!

(sorry, couldn't help myself...)

Breathe Quite Comfortably?

[Guido69]

The advantage of Halon is that you can breathe quite comfortably in an atmosphere that will not sustain fire. CO2 works just about as effectively but will not sustain life.

You've obviously never been in a Halon dump. Either that or you consider burning in your lungs to be "quite comfortable". Not to mention that if you're standing under one of the discharge nozzles at a dump you can get a nasty case of frostbite.

I've personally been through two 1211 dumps and had to enter a computer room and drag staff out after an FM200 dump. It takes about two days to completely stop coughing.

Not funny...

[distributed.karma]

Looks like somebody forgot to patch the firewall. *ducks*

Cause? Injuries?

[nurb432]

Just curious if everyone is ok, and if they have found a cause yet..

*insert M$ terrorist joke here*

Students!

[flippet]

The Register has a story about it. I like the paragraph at the end...

Twente's high-speed network was originally constructed to provide students with access to high-speed Internet access for their studies. It soon became one of the major hubs for peer-to-peer exchange programs like KaZaA. This perceived misuse of Internet resources caused former Dutch education minister Loek Hermans to comment: "It would be nice if the students at Twente University would use their fast connections for information and education purposes, instead of downloading huge amounts of porn."

Phil, just me

Re:WARNING: Above post a GOATSE!

[ealar dlanvuli]

Apparently UT was a major node for KazAA, and'a primo source for warez and pr0n.

And the firefighters will determine it was lit by a professionaly arsonist, but never be able to figure out what orginization funded him..

Re:English, you insensitive clod!

[CoolVibe]

Learn dutch then.

The slashdot posting says it all really, the Utwente NOC burned down. I myself am dutch, so I grokked the links quite fine :)

Many friends of mine study at utwente. I was already wondering why they weren't on IRC, but since I heard the news I knew why.

Oh, AFAIK, utwente kept off-site backups of all data, so all is probably not lost.

Sad

[ledow]

Maybe it was a flame war between students? Were they overclocking? The dangers of using FireWire. Were their harddrives Quantum Fireballs? (They are now) Is this what you get when you try to hot-swap them?

Other sad jokes will no doubt follow.

translation..

[mikevdg]

This is a translation from the telegraaf (which, btw, isn't the best newspaper in holland, but anyway..)

"University network down from fire"

"ENSCHEDE - A fire in the computing center from the university of Twente (UT) in Enschede has caused a few 10's of millions of euros damage on Wednesday.

The fire, which broke out about 8:00am, has disabled the universities network. There were no injuries. The fire department has cleared out several buildings in the nearby facinity. The IT departments of the UT and a part of the business studies department were housed in the building where the fire broke out. The building was three stories high.

The fire department fought the fire with several dozen people and estimated that they would need the whole day to get the fire under control. There are, until now, no dangerous substances released. The cause of the outbreak is not yet known.

According to Van Vught, a backup of the network's data has been made. "All data have been safely stored". Because of exams, there were no classes at the time. There were also few students on the campus on wednesday. Exams on wednesday have been cancelled.

Staff at the UT want to set up a temporary network, which can take several days. Thousands of staff and students can not make use of the campus network until a temporary network has been installed.

One on-looking business studies student is shocked (?): "incredibly sad. It means that I can't do my assignments. Doubtless, a temporary building will need to be set-up to help the situation". The Saxion Polytechn in Enschede has made room available.

The web-site from the Enschede city council is not available because of the fire. The council used the servers from the UT. The Saxion Polytech is also without internet access for the same reason."

Okay, this translation is a wee-bit too literal, but it's understandable.

Michael.

The cause of the fire?

[Lethyos]

So wait, the University of Twente NOC caught fire. Why? Was there a story posted on Slashdot's front page that linked to a server at their location? Or was this fire caused by something other than a hardy slashdotting?

Re:The cause of the fire?

[poot_rootbeer]

They should have just let Milton have his stapler...

Could this be...?

[fruity1983]

Microsoft said they were looking for new strategies to counter the rise of linux, but who woulda thought they'd go this far?

Donations?

[Akardam]

I understand from a post a ways down that the University is currently having financial difficulties. Does anyone know if there's been something setup, whether for the University in general or Debian in specific to accept donations? I know (even though I don't use Debian) that I would contribute a few dollars to help them pay for all the new equipment, and I suspect that there's more'n a few geeks out there who feel the same way.

Utwente - The proof that admins shouldn't smoke

[fluor2]

Utwente - The proof that admins shouldn't smoke

security.debian.org backup online

[m0i]

Apparently, klecker.debian.org (194.109.137.218) is now hosting security.debian.org (as seen in NS updates propagating).

Re:Halon dumps? -- not if the whole building is in

[Salamander]

I would *love* to see a halon system capable of stopping that...

I wouldn't. Any halon system capable of doing that would literally be a killer in its own right.

You mean HiveCache (formerly MojoNation)?

[Fencepost]

That's exactly what the Mojo Nation folks are doing now. Info at http://www.mojonation.net/.

Conspiracy Theory!!!

[fantomas]

Come on guys, where are the conspiracy theories???

UT hosted HAL2001 last year, surely it was the FBI / CIA / KGB / SMERSH / Mysterons / [enter your bad guys of choice...] getting back at those pesky linux hackers?

fire supression systems?

[eclectric]

sure, they kill everyone in the room, but they're also pretty good at putting out fires. What are they using... water?

So, when does the pledge drive start?

[bADlOGIN]

Since Debian doesn't have a 40 Billion dollar monopoly warchest to draw from, I'm assuming that some funds will need to be raised to get new boxes. This is a perfect time for uses to step up and make a contribution back.

Re:WARNING: Above post a GOATSE!

[ReelOddeeo]

>>Apparently UT was a major node for KazAA...

And the firefighters will determine it was lit by a professionaly arsonist, but never be able to figure out what orginization funded him..

....but never be able to figure out which of the *AA's funded him.

This isn't homeowner's insurance

[wiredog]

Business insurance is more costly, but has better service. If the word gets out that a certain company isn't able/willing to pay claims on policies with high quarterly premiums expeditiously, then all the companies that used to buy from that agency will go elsewhere. Fast.

Re:This isn't homeowner's insurance

[bad-badtz-maru]

Have you ever filed a claim against a business insurance policy? I was involved in the filing of a $1.1M USD claim involving hurricane damage and I assure you, the check is not there the next day... or the day after... or the day after... even though our annual premium, for more than seven years, had been more than four times the amount of the claim. And this was a major insurance carrier.

maru

Class IV construction

[mmol_6453]

I don't doubt it. That's a Class IV construction; I.e. mostly concrete and steel. It's really hard for fire to spread in buildings like that; You'd have to have notably flammable materials in any direction the spreads.

An excellent example of Class IV construction would be any recently build NFL stadium. I haven't been in one often, but I was struck by how far apart any flammable material (food carts, etc.) is. Even the hallway ceiling I saw was actually the bottom of the concrete steps above.

Re:Class IV construction

[haligan]

As a former fire fighter and fire inspector I must comment that yes the construction may have been type IV , but like most people , Architects included, you are forgetting that the contents are mostly flammable, (ie carpet, paint, some ceiling tiles). Mike B.

Re:Class IV construction

[mmol_6453]

As a former fire fighter and fire inspector

Explains your username. :)

...forgetting that the contents are mostly flammable, (ie carpet, paint, some ceiling tiles).

I thought about that, but I presumed that the materials would be somewhat spaced out; distance between couches for comfort, room between rows of computers (in labs) for safety. Things like that.

All in all, though, good point.

They Needed Low-Tech Fire Protection

[0x69]

Our ISP bought an old legal office building for their HQ and colo facility. The place was built with file rooms to safeguard tons of irreplaceable paper documents - imagine thick concrete walls & ceilings, with heavy steel fire doors, rated to preserve the contents through an EVERYTHING-else-burned-to-the-ground fire.

Critical stuff is spread between the file rooms, with metal conduit, etc. protecting the few small holes they added for wiring.

Steel & reinforced concrete aren't quite obsolete.

more information

[Ruliz Galaxor]

I live about 5km away and at 11:00 CET I could still see a lot of brown/grey smoke in the air.

frontpage.fok.nl (a dutch news site) reports the following, well at least not the article itself but in a reaction to it: "The D-wing was recently renovated to make it more 'fireproof', so there are no harmful gasses and such.
Every morning a backup is made from all the data. Unfortunetely, before this backup could be taken, it was already destroyed, which means data from yesterday morning is lost.

Tonight (CET of course) there will probably be an emergency network running at 10kb/s and the server will probably be online in 3 days. Until then Teletop (some kind of information system for communication between tutors and students) and the utwente email will be down. Something about the building: The TWRC building some kind of monument which included that the outside of the building cannot be changed. This means the renovation plans have to be the same as the orignal building."

At least I hope they got rid of the air-problem, because it really stinked in that building anyways. Too bad everything to lost. They were just about to move all the servers to another location.
I was just printing my report which I had to hand in at 12:00 CET when I heard it. I worked the last 24 hours to complete the damn thing and then this. :(
Lucky for me I don't have internet from the university, but it still sucks bigtime.

this guy probably did it...

[h4x0r-3l337]

Jos Boersema, usenet cook

(yeah, I know physics department != NOC, but this guy isn't very bright)

Insurance money... not much

[erpbridge]

Back the end of Sept 2001 we had a lightning strike on campus where I work. It twisted around a tree, then went into the ground.

The UPS's worked great, and none of the power was affected at all. Howerver, the problem was when the lightning hit the ground, it traveled in all directions... eventually hitting the buildings themselves, and found the bundles of CAT-5 running below the floors, in the walls.

Being a small campus, we only lost 14 workstation network cards, 4 printer network cards, and 1 workstation. However, one of those buildings was where our main fiber / copper switch was for the campus. We lost two blades on that, as well as a few lesser switches.

The moral being: Insurance company only gave us about 40% of what everything was worth. In something like the fire we're all talking about, they may give some more, but nowhere near enough to bring the campus back up to where it was. There is going to be some significant cost to this university.

Re:Insurance money... not much

[rosewood]

Someone was under insured then or not properly insured

You can be covered so that everything ruined from that would be COVERED

but hey, that costs MONEY

No! MPAA/RIAA

[bsd-mon]

It soon became one of the major hubs for peer-to-peer exchange programs like KaZaA

DMCA enforcement in full effect

Hey! Those are in order!

[mmol_6453]

Interesting that almost all the pics there are in time-sequence...You could make an animation out of it.

I don't know how, but I encourage other people to try.

Two sites don't cost much more than one

[billstewart]

Instead of putting all the computers in one building, put half of them in two buildings. The main cost difference is some extra real-estate preparation. And if you've got a multi-gigabit network, it's going to multiple buildings, so spread out your routers. Huge centralized computers were really useful back when mainframes were the size of dozens of refrigerators, but if you're using modern machines, they're either rack-mounted or pizza-boxes, and or else they're PC-shaped. But you've got multiples of them and can spread them around.

Re:Two sites don't cost much more than one

[billstewart]

No, but while you probably need more total air conditioning if you split between two buildings, you need only about half as much space per building as you did before, because you're putting about half the stuff in each. So instead of one building with N square meters of server rooms and another building with N square meters of cheap low-quality graduate student desk space (:-), you need two buildings, each one with N/2 square meters of computers and N/2 square meters of desks. Obviously some pieces of equipment can't be split, so you have to decide how badly you need two of them for reliability - but for almost everything in the computer business, prices are much lower than in past years, comparable to expensive physics or chemistry equipment (which is more likely to catch fire.)

While it's too bad that the equipment was lost, and much worse that data was lost, one advantage of replacing old computers with new ones is that the same amount of computer capacity will cost you less money with new equipment, and will use less floor space, electricity, and air conditioning, and will cost less to maintain than the old equipment. Depending on your campus wiring infrastructure, it may be possible to take advantage of some other new architectural developments, like peer-to-peer storage. It's not just for pirate music - the cost of PC data storage is almost down to $1000 per terabyte, and some of the applications can be reliable enough to use it effectively.

The cause is clear

[Maskirovka]

I guess the their boss fired the bofh.

Do NOT stay at ground level with HALON!

[billstewart]

Halon is heavier than air, so if you're somewhere there's been a Halon dump, DO NOT HIT THE GROUND! Walk/run as upright as you comfortably can. The usual "stay low" advice is because hot air and smoke are lighter than regular air, so the regular air stays near the floor.

Halon (or at least most of the Halons used for fire suppression; not sure about all of them) is non-toxic, though it'll get you a bit high, like nitrous oxide or most solvents, so being stuck in the stuff won't injure you quickly (except from flying objects that were blown around by the gas pressure.) However, it's no substitute for oxygen, and you'll probably be wanting some oxygen real soon now. If you can remember not to breathe in the stuff, try not to breathe deeply, because there's more oxygen left in your lungs than the stuff you'll be breathing in, and unfortunately, while your body can generally figure out not to breathe in water or hot smoke, it's not as good at realizing that near-room-temperature inert gases aren't very useful. Mostly, don't worry about it - find a safe door to run for and run for it, and do some deep breathing once you're outside, and try to close the door behind you.

Things are being restored slowly

[wichert]

Lots of people are helping to restore the lost network and computing facilities. People from XS4ALL drove from one side of the country to bring spare junipers and other equipment. The campus is already back online thanks to Virtu. Mail for student is being stored at a new backup MX server courtesy of Terena and zedz.net

Debian is restoring the lost services on klecker. At this moment qa.debian.org is up and running and the non-US and security archives are available as well, although their backend systems have not been restored yet.

Valuable lessons have been learned though: it is very useful to have machines on standby where you can switches services to when needed. Having backups of important data is also really useful (and we could have done a bit better at that. UTwente apparently has good off-site backups of its own data though). And having good insurance is also definitely useful.

Backup strategies vary widely

[billstewart]

Some companies run their own backup data centers, some use backup / disaster recovery vendors, some don't. It varies a lot by customer and industry. Another strategy that's been emerging rapidly is the use of Internet colo centers as backup sites (primarily the rackspace-oriented colo rather than managed hosting colo.) Engineering companies and companies that formed from mergers of similarly-sized companies often have two data centers anyway, so they'll use one to back up the other.

The big value of many of the disaster recover vendors, beside consulting skills, was traditionally that they owned mainframes, and assuming not too many people had disasters at the same time, they could mount your data on their mainframe without you having to spend a few million dollars to buy a spare mainframe - so they were basically selling part-time use of the mainframes. Transaction-based online services companies could sometimes live with this kind of environment, but not usually - if it takes a day to rebuild your database from backup tapes and journalling, that's probably ok for a manufacturing company, and maybe ok for a traditional bank, but it's a total disaster for an airline ticket sales system or an online stockbroker; even a couple of hours may be too expensive. So those companies often have to run their own backup sites, or at the very least will have a dedicated backup mainframe at the backup vendor.

One of the interesting technologies that's starting to affect this business is metropolitan-distance fiber optics for Storage Area Networks. Somebody else mentioned iSCSI, but for the mainframe world, Fibre Channel lets you connect processors and disk farms at distances up to about 30km, and for real computers (:-) Gigabit Ethernet on fiber also has a long range (and eventually 10gigE will be mature.) This is especially useful in some niche markets - Wall Street trading and banking firms can put disk farms in New Jersey, partly for physical diversity and partly for real estate cost reasons. They're also good for other downtown/suburban connections. Sometimes these are managed by the customer, but the internet data center business and local fiber providers are going hog-wild trying to sell this. (I work for AT&T, which is selling this.)

Re:Nice Moderation

[scott_evil]

... I hope nobody was hurt.

After that, I wish them luck getting back online.

Someone explain how the FUCK that can be considered insightful?

Yet again we see the need for moderators to pass an IQ test.

Re:Terrorism

[Koos]

I'm surprised noone has screamed "OH NO TERRORISM" yet.

It did not happen in the US ;-)