Slashdot Mirror

← Back to Stories (view on slashdot.org)

University of Twente NOC Destroyed

Posted by michael on from the debian-users-grief-stricken dept.

JanJoost writes "Around 08.00 CET today the University of Twente Network Operations Center, which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org, caught fire. The UT, which hosted the HAL in august last year is completely unreachable and is not likely to come back up any time soon. The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings. More information can be found at the Telegraaf, Planet Internet and Twentsche Courant. Pictures can be found here and here. It's a shame to see a great infrastructure go down in flames like this."

40 of 465 comments (clear)

  1. More info by Strike · · Score: 5, Informative

    From debian-devel, here's a slightly (only slightly) more informative blurb

    1. Re:More info by Stonehead · · Score: 5, Informative

      Hey, I posted that to debian-devel. Scary to see it get Slashdot headlines, since this posting from Wichert Akkerman himself is more 'official'.
      He's now probably busy setting up klecker.debian.org as the next security.debian.org host. Don't get yourself trojaned, please people, don't panic and just wait for the official Debian announcement that everything has been fixed again. Or play around with inofficial mirrors like these, and there are more. But I feel a bit stupid myself, because - unlike Wichert - I have done nothing myself except forwarding the news and act like a karma whore.

  2. Photo's by fearlezz · · Score: 5, Informative

    More images can be found on http://www.bsdfreaks.nl/files/brand.htm

    --
    .sig: No such file or directory
  3. now the engineers come out... by ravidew · · Score: 5, Insightful

    ..to see how this could be prevented in the future. How much fire protection do NOCs owned by the big boys (Verio, WorldCom) have? Offsite backups, too, I hope?

    1. Re:now the engineers come out... by rleyton · · Score: 5, Interesting

      Whilst I'm no expert in fire supression systems, I have studied a few data centres in my time as a Senior Sys Admin for a number of companies. Most (decent) NOC's have a fire *supression* system based on gas or (more likely these days) some form of liquid (gas has the nasty side-effect of potentially killing humans).

      These are for small fires that can be contained within a data centre, ie. a computer catching fire or emitting smoke. Really good systems are very localised (racks or cage specific). A big fire just isn't going to be stopped by such a system.

      I haven't read too much into this particular incident (ie. not at all), but my initial thought was that something more serious must have happened (well, duh!), perhaps a fire outside of the main suppression system (outside of the raised floor area?). Or perhaps the paint on the walls/carpet wasn't fire resistant and just took hold very quickly.

      Or a large initial fire (gas leak?) that just didn't die down when the supression system kicked in. Maybe the type of fire (again, gas? oil?) didn't die down because Data Center supression systems presumably focus on electrical fires.

      A well, just my 2p's worth.

      --
      ooooooh! What does this button do? - DeeDee, Dexters Lab.
    2. Re:now the engineers come out... by Martin+Blank · · Score: 5, Interesting

      Most NOCs I've been in have an oxygen-deprivation gas that's dispersed in the case of a fire (after a series of highly visible and audible warnings). The one I'm in right now seems to have gone the inexpensive (and arguably safer) way of a two-stage dry-pipe water sprinkler system where the pipes are usually dry (empty). In case of a smoke detection above a certain level, the pipes are "charged" (filled with water), and if the heat gets above a certain level, the sprinklers go off. Basically, the decision was made that if there's a fire, the equipment will probably need replacement anyway, so why not use sprinklers?

      Personally, I'm not too sure of this route. I can understand it, from an environmental and human-safety perspective (the gasses eat the ozone layer, and you *really* don't want to inhale stuff that ties up oxygen at those rates), but if it ever happens here, it's going to be an awfully hurried mess to get everything back up and running in a reasonable timespan, even if the fire is only a small one that doesn't destroy the building.

      --
      You can never go home again... but I guess you can shop there.
    3. Re:now the engineers come out... by AlphaInsight · · Score: 5, Informative

      Actually FM-200 and Inergen are the current replacements for Halon. Non Ozone-depleting and won't kill you if you're trapped in a NOC. Just lowers the avaliable O2 in the air. You'll get out of breath easily, but as long as you stay calm you won't have to worry.

    4. Re:now the engineers come out... by DunbarTheInept · · Score: 5, Insightful

      When you can't get oxygen from the surrounding air, the worst thing in the world to do is take a breath. You will survive much longer by fighting the urge to take a breath, and instead just hold the air already in your lungs in place until you can get out. You don't convert all the oxygen in your lungs into carbon dioxide with each breath - far from it. You just convert a portion of it, so the air you expell just has a smaller percentage of oxygen than the air you breathe in. But there's still quite a bit there that never got converted. (That's why mouth-to-mouth breathing can help someone - the air you breathe out still has enough oxygen in it to be a lot better than no air at all.)

      But if you breathe that air out and breathe oxygen-less air in, you will pass out very fast. Most people have the misconception that you can do without oxygen for a minute or two before dying. That's not true - your body needs to consume fresh oxygen at a continuing rate just to function at all, it's just that your lungs can HOLD a small supply of oxygen to supply this need for a minute or two. Get rid of that oxygen by breathing it out and replacing it with oxygenless air, and you're going to pass out in just a few seconds, and be dead shortly thereafter.

      And the worst part is you won't FEEL like anything is wrong. Your body is unable to measure the level of oxygen in your lungs. Instead your body senses the level of carbon dioxide in your lungs. As the by-product of normal breathing, when carbon dioxide has built up enough, that indicates you've converted a lot of oxygen and it's time for another breath. This is what triggers the automatic involuntary breathing that takes over when you stop thinking about it. This is also what triggers the panic feeling that you get when you know you need air. Your lymph nodes detect too much carbon dioxide and start sending the panic signal to your mind. What this all means is that if your body isn't exchanging oxygen for carbon dioxide, your body doesn't even realize it's asphixiating. If there's no oxygen in your lungs to start with, then there won't be any carbon dioxide building up in the lungs, and you will feel no sensation of needing a breath at all. You'll feel just fine for a few seconds and then *poof* you're gone as the blood going to your brain runs out of oxygen and your brain activity just plain stops.

      So if you're ever in a halon gas system when it goes off - DO NOT BREATHE. Just hold whatever breath happens to already in your lungs and get out. The instinct is to hold your breath by first inhaling your lungs full and THEN holding it, but that's the worst thing you could do, as explained above. The tricky part is remembering to override that instinct.

      --

      Don't label something "offtopic" unless you know the topic well enough to tell what's on topic.

  4. Vunerability by e8johan · · Score: 5, Interesting

    This shows the vunerability of putting all computers in one building. To have a safe network one needs to spread (duplicate) the information over several computer at several locations. How far apart these locations has to be is depending on how important you data is.

    It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup. By that I don't only mean that tape that always seems to go missing when needed, but multiple sites (or at least buildings) that provide redundancy.

    1. Re:Vunerability by Psiren · · Score: 5, Funny

      To have a safe network one needs to spread (duplicate) the information over several computer at several locations.

      What a stunning idea. Perhaps they should call it the internet... ;-)

    2. Re:Vunerability by pellaeon · · Score: 5, Insightful

      Try doing that on a university-wide multi-gigabit capable network on the budget of the average Dutch university. Our universities aren't like M$ in cash, you know. I know, I'm an admin at one myself.

      I just hope they're well insured....poor colleagues...

      On the upside: they may get a squeaky-clean start when this blows over :-)

      --
      -- /bin/coffee missing. universe halted.
  5. A good reminder.. by Martigan80 · · Score: 5, Insightful

    To never keep back-ups in the same physical location.

    --
    This SIG pulled due to lack of funding. (This damn war is costing too much!)
    1. Re:A good reminder.. by Zocalo · · Score: 5, Insightful

      Or even general location for that matter. A friend of mine did disaster recovery work for IBM after the Trade Towers attack. They had their data center in Tower 1 and their backup center in Tower 2. After six weeks of what was essentially scrabbling through rubble they managed to recover a single spindle. The company concerned became another statistic, and part of an important lesson in DR implementation; safety increases with distance.

      --
      UNIX? They're not even circumcised! Savages!
    2. Re:A good reminder.. by Jugalator · · Score: 5, Funny

      To never keep back-ups in the same physical location.

      That's why I use Kazaa to keep backups of my pr0n all over the world!

      --
      Beware: In C++, your friends can see your privates!
  6. So much for server areas never burning down by Martin+Blank · · Score: 5, Interesting

    This is something I keep hearing from people, particularly those who balk at paying for upgrades to expensive fire suppression systems. "They're always built so well that there's never been a case of a server room destroyed by fire." I always did doubt that, and now whenever they tell me that, I can point to this.

    What a shame.

    --
    You can never go home again... but I guess you can shop there.
    1. Re:So much for server areas never burning down by Plutor · · Score: 5, Insightful

      The fact that "The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings" leads me to believe that the fire didn't start in the server area. Lots of server rooms were destroyed on September 11, for example, but it wasn't the fault of the room's design, or the presence or lack of fire suppression systems. If the whole building is burning down, fire suppression in one room is only going to work until the floor and ceiling collapse.

  7. In a related announcement... by Zocalo · · Score: 5, Funny

    The University of Twente's attempts to overclock the new AMD Opteron and Nvidia GeForceFX card in the same case are declared a failure. "We certainly won't be building a Beowolf cluster of these..." commented a spokesperson.

    --
    UNIX? They're not even circumcised! Savages!
  8. Priorities by brianvan · · Score: 5, Insightful

    Was anyone killed?

    If not, was anyone hurt?

    If not, do they have insurance?

    If they do... well, I'm sure someone just lost their masterpiece pr0n directory, but otherwise, things like this happen. (ask Hemos) You have to make it through such things. In this case, it was a commercial (educational) building and no one is homeless, so it's less of a tragedy than usual. Let's hope that they rebuild with something better and newer.

    That said, I get the feeling that those plumes of smoke really are millions of dollars floating away in the wind...

    1. Re:Priorities by bzzzt · · Score: 5, Informative

      No, no and no.

      The university doesn't insure itself agains this kind of disaster because they are able to set aside enough money to cover the cost themselves and don't have to pay the insurer's profit.

      Now how they are going to deal with this and their financial troubles (they're almost broke) is another issue...

  9. last syslog messages from SURFnet routers by ahu · · Score: 5, Interesting

    lo0.ar5.enschede1.surf.net 3613: Nov 20 07:20:50.927 UTC: %ENV_MON-2-TEMP: Hotpoint temp sensor(slot 18) temperature has reached WARNING level at 61(C)

    few seconds later on the local side:
    lo0.cr2.amsterdam2.surf.net 1146: Nov 20 07:20:56.458 UTC: %CLNS-5-ADJCHANGE: ISIS: Adjacency to ar5.enschede1 (POS2/0) Down, interface deleted(non-iih)

    1. Re:last syslog messages from SURFnet routers by Jugalator · · Score: 5, Funny

      I wonder if they got the Linux "easter egg" error message "Printer on fire"? I guess it would be appropriate for once.

      --
      Beware: In C++, your friends can see your privates!
  10. Re:why does this matter? by brianvan · · Score: 5, Funny

    They're not gone. They're just experiencing a Distributed Combustion Denial-of-Service attack.

  11. Debian Security mirror by lemmen · · Score: 5, Informative

    In case one might be interested, Essent mirrored security.debian.org.

    You can use debian.essentkabel.com to download the latest security updates (in case you haven't already). Please note this is NOT an official mirror.

    --
    Daxy's Networking Blog
  12. OK, folks, admit it! by matija · · Score: 5, Funny

    Who posted a link to UT's webserver on slashdot?

    --
    Duct tape + WD40 => DevOps
  13. More Pictures from the webcam by CuriousGeorge113 · · Score: 5, Informative
    Here's some statics taken from their webcam. Someone might want to mirror these before they get slash'd. Webcam PicsPics

    Also, here's what seems to be the only close up I could find of the fire. pics

    --
    No man is an island, But if you take a bunch of dead guys and tie them together, they make a pretty good raft.
  14. Update from SARA (SURFnet NOC) by mdav · · Score: 5, Informative

    Here's an update from SARA (that's where I work), the network operator for SURFnet. SURFnet is very busy ordering new equipment and fixing the 2 x 10 Gbit/s lamda's to Enschede. We hope to restore connectivity a.s.a.p. Greetings, Marco

  15. It must have been Microsoft by micaiah · · Score: 5, Funny

    "Witnesses saw a large balding man monkey dancing from the scene and a slim geeky man with glasses trailing behind continuously adjusting his glasses." An in other news....

  16. LFS mirror affected too by decarelbitter · · Score: 5, Informative

    The Dutch LFS FTP mirror was also hosted at the University of Twente, which means it's also down. The Dutch HTTP mirror should work properly, since it's outside utwente.nl space.
    Last news is that HP (Who supplied most of the UT backbone equipment) is on its way with emergency equipment to have things up and running somewhere tomorrow.

  17. Re:Halon dumps? -- not if the whole building is in by earthy · · Score: 5, Informative

    The fire did not start in the server rooms. What happened was that a fire started in one of the lecture rooms (and a smallish one at that, number A108) that just happened to be in the same wing of the TWRC building that also housed the server rooms (yes, multiple). It then proceeded to take out two entire wings and threaten other buildings nearby as well as the library.

    Now, I would *love* to see a halon system capable of stopping that...

    Owh, and the fire seems to be under control by now, as evidenced on http://webcam.traserv.com/
    (which you can contrast to http://www-infstud.sci.kun.nl/~arthurvl/ispy.jpg (taken at about 09:40 CET this morning)).

  18. PGP Keyserver root was hosted by SURFNet by Erik_ · · Score: 5, Insightful

    I believe, the open and distributed network of Keyserver.net (distibuted network of PGP keyservers) was hosted by SURFNet. This network is a distributed network holding PGP and OpenPGP keys. The loss the to UT NOC could have an impact on the updating of key-rings across the keyserver.net network.

  19. security updates mirrored worldwide. by novakreo · · Score: 5, Informative

    Actually, Debian security updates are usually also added to the proposed-updates section, which means that they are available on nearly all Debian mirrors worldwide.

    Something like
    deb ftp://ftp.XX.debian.org/pub/debian/ proposed-updates main contrib non-free
    deb ftp://ftp.XX.debian.org/pub/debian-non-US/ proposed-updates/non-US main contrib non-free
    (replace XX with your ISO country code) in your /etc/apt/sources.list should work well.

    You can find mirrors on Debian's website.

    --
    O frabjous day! Callooh! Callay!
  20. BACKUP!!! by Beliskner · · Score: 5, Insightful

    Now we can find out how secure and hardened Debian really is. You are as good as your latest backup.
    BACKUPS BACKUPS BACKUPS Off-site! I've had enough of people who are talking about RAID-5 because 5TB tape drive arrays are too slow. Always keep your BACKUPS!

    --
    A caveman dreams of being us, the incalculable power and riches. We dream of being Q, then what?
  21. About as good as it gets with only two sites... by Kjella · · Score: 5, Insightful

    I mean seriously, each tower collapsed because it was hit by its own plane. If one tower had been in NY, one in California and both were still hit by a plane each, the result would be exactly the same.

    The lesson should be: Primary back-up is a very good start, but secondary/tertiary back-up is the thing if it's that critical.

    Kjella

    --
    Live today, because you never know what tomorrow brings
  22. The world's most warped error message... by meringuoid · · Score: 5, Funny

    ... finally gets to make an appearance in earnest.

    'lp1 on fire'

    --
    Real Daleks don't climb stairs - they level the building.
  23. Re:Disaster Recovery plans by wiredog · · Score: 5, Insightful
    The Gartner Group put some stuff up after 9/11. Most of it is common sense.

    Do full backups weekly, store copies offsite. Incremental backups daily, copies offsite also. If you can afford it (or can't afford any downtime), have emergency backup hardware (enough for minimal operations) in an offsite storage facility. Old hardware that would otherwise be thrown out is good for this (remember, it's for an emergency). Have a supplier who can get replacement hardware to you in a hurry (so you can get off of those old 90 MHz Pentium servers).

    The most vital part of the plan, after backups, is good insurance. If the building burns to the ground Monday morning, you want to be able to call the insurer Monday Noon, and have the check in hand Tuesday morning at the latest.

    These recommendations do not cover disasters such as 767s flying into the building and killing all the sysops. Earthquakes dropping the building on the same. Etc. The people are the most important part of any company and, if too many of them are lost at once, the company probably is lost too.

    Unless you have really good (and expensive)insurance which can provide enough funds for you to hire new people, get them trained, and keep the company solvent while you do so.

    --

    Best Slashdot Co
  24. Debian is working on recovering by StormCrow · · Score: 5, Informative

    Quote from debian-devel-announce

    At around 8 this morning (local time) a fire started in the computing facilities of Twente University. This affects Debian, since one of our servers (satie) is hosted there. At this moment it seems very likely that the machine can not be recovered from the fire.

    The following services are currently down as a result of this:

    security.debian.org
    non-us.debian.org
    nm.debian.org
    qa.debian.org

    We are working to restoring these services on another machine and hope to have things in mostly working order by tomorrow. Security advisories are still available at http://www.debian.org/security/

    Wichert.

  25. Breathe Quite Comfortably? by Guido69 · · Score: 5, Informative

    The advantage of Halon is that you can breathe quite comfortably in an atmosphere that will not sustain fire. CO2 works just about as effectively but will not sustain life.

    You've obviously never been in a Halon dump. Either that or you consider burning in your lungs to be "quite comfortable". Not to mention that if you're standing under one of the discharge nozzles at a dump you can get a nasty case of frostbite.

    I've personally been through two 1211 dumps and had to enter a computer room and drag staff out after an FM200 dump. It takes about two days to completely stop coughing.

    --
    - If we aren't supposed to eat animals, then why are they made out of meat? - Steven Wright
  26. Students! by flippet · · Score: 5, Funny
    The Register has a story about it. I like the paragraph at the end...

    Twente's high-speed network was originally constructed to provide students with access to high-speed Internet access for their studies. It soon became one of the major hubs for peer-to-peer exchange programs like KaZaA. This perceived misuse of Internet resources caused former Dutch education minister Loek Hermans to comment: "It would be nice if the students at Twente University would use their fast connections for information and education purposes, instead of downloading huge amounts of porn."

    Phil, just me

    --
    "Cattle Prods solve most of life's little problems."
  27. Sad by ledow · · Score: 5, Funny

    Maybe it was a flame war between students? Were they overclocking? The dangers of using FireWire. Were their harddrives Quantum Fireballs? (They are now) Is this what you get when you try to hot-swap them?

    Other sad jokes will no doubt follow.

  28. Re:The cause of the fire? by poot_rootbeer · · Score: 5, Funny


    They should have just let Milton have his stapler...