Slashdot Mirror

← Back to Stories (view on slashdot.org)

University of Twente NOC Destroyed

Posted by michael on from the debian-users-grief-stricken dept.

JanJoost writes "Around 08.00 CET today the University of Twente Network Operations Center, which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org, caught fire. The UT, which hosted the HAL in august last year is completely unreachable and is not likely to come back up any time soon. The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings. More information can be found at the Telegraaf, Planet Internet and Twentsche Courant. Pictures can be found here and here. It's a shame to see a great infrastructure go down in flames like this."

18 of 465 comments (clear)

  1. Vunerability by e8johan · · Score: 5, Interesting

    This shows the vunerability of putting all computers in one building. To have a safe network one needs to spread (duplicate) the information over several computer at several locations. How far apart these locations has to be is depending on how important you data is.

    It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup. By that I don't only mean that tape that always seems to go missing when needed, but multiple sites (or at least buildings) that provide redundancy.

    1. Re:Vunerability by xyote · · Score: 2, Interesting

      When iSCSI becomes real you are going to see a lot more raid-1 mirroing with one half of the mirror at a remote location. Also using remote tape drives for backup. Currently if you do store your backup tapes offsite, there is still that logistical problem of getting them offsite in a timely manner. Things like if you have 10 partly full DLT tapes at $$ a tape, do you send then offsite right away or do you wait for the next set of backups to fill them up. The former can be quite expensive. Also, since most restores are from most recent backup tapes, sending them offsite right away causes delays in your restores while you wait for the tape to come back. Remote tape backup will solve a lot of these issues.

    2. Re:Vunerability by Anonymous Coward · · Score: 1, Interesting

      I'm an auditor (pause for tomato dodging) and this reminded me of a couple risks I had to nail clients for.

      The first kept their AS400 farm running the majority of their critical applications in a building near their chemical plant (*ding*). Normally this would only be considered a minor risk, however they were in a low-lying area of Louisiana. We were actually looking UP at the Mississippi River (well, we would have without the dykes). They never seemed to grasp why we were so appalled that we could see a WATERLINE from a previous flood about 6 feet above groundlevel!

      Another had a backup site and stored their tape backups at a site that was DIRECTLY in the path of one of the busiest runways at an extremely busy airport. We actually had to pause conversations when anything jet-powered was taking off. They didn't consider that to be a risk either.

  2. So much for server areas never burning down by Martin+Blank · · Score: 5, Interesting

    This is something I keep hearing from people, particularly those who balk at paying for upgrades to expensive fire suppression systems. "They're always built so well that there's never been a case of a server room destroyed by fire." I always did doubt that, and now whenever they tell me that, I can point to this.

    What a shame.

    --
    You can never go home again... but I guess you can shop there.
    1. Re:So much for server areas never burning down by mmol_6453 · · Score: 3, Interesting

      Here in Grand Rapids, Michigan, any cable that passes through the heater system has to be Plenum. And considering that, at Greybar, you have a choice between Plenum and riser-cable, most of the new (as in, last twenty years) telecom wiring in Grand Rapids is fire-resistant. The insulation of Plenum will liquify before it burns...

      --
      What's this Submit thingy do?
  3. last syslog messages from SURFnet routers by ahu · · Score: 5, Interesting

    lo0.ar5.enschede1.surf.net 3613: Nov 20 07:20:50.927 UTC: %ENV_MON-2-TEMP: Hotpoint temp sensor(slot 18) temperature has reached WARNING level at 61(C)

    few seconds later on the local side:
    lo0.cr2.amsterdam2.surf.net 1146: Nov 20 07:20:56.458 UTC: %CLNS-5-ADJCHANGE: ISIS: Adjacency to ar5.enschede1 (POS2/0) Down, interface deleted(non-iih)

    1. Re:last syslog messages from SURFnet routers by Anonymous Coward · · Score: 1, Interesting

      temperature has reached WARNING level at 61(C)

      FYI- 61 degrees C = 141 degrees F.

      Sprinkler systems kick in around 150 to 160 degrees; temperature at the top of the room was probably higher.

      My bet is that router died(along with everything else) when the sprinkler system came on and power was cut to the room(most fire suppression systems in datacenters, water or gas, cut power to the entire room.)

      It dated back to the earliest systems- "big red switches" usually have what are called "Molly guards", which are supposedly named for the daughter of a Univ. of Calif. programmer whose daughter hit the button 3 times in one day.

      The same jargon dictionary mentions that in the 'early dates', said BRS fired a shell into the mains bar of the mainframe to sever it.

      That said, 60 degrees is about the max operating temperature for most electronics. So take your pick :-|

  4. Halon dumps? by wiredog · · Score: 4, Interesting
    The last time I worked in a NOC, it ran Vaxes, but we had a halon dump. A Big Red Button that got smacked by the last person out of the room. The halon would smother any fire by replacing all the oxygen in the room (which was why the last guy out hit the button). Why wasn't there a halon dump in this NOC? Or, if there was one, what happened?

    I hope Debian practices good management principles by having offsite backup.

    --

    Best Slashdot Co
  5. Re:now the engineers come out... by rleyton · · Score: 5, Interesting

    Whilst I'm no expert in fire supression systems, I have studied a few data centres in my time as a Senior Sys Admin for a number of companies. Most (decent) NOC's have a fire *supression* system based on gas or (more likely these days) some form of liquid (gas has the nasty side-effect of potentially killing humans).

    These are for small fires that can be contained within a data centre, ie. a computer catching fire or emitting smoke. Really good systems are very localised (racks or cage specific). A big fire just isn't going to be stopped by such a system.

    I haven't read too much into this particular incident (ie. not at all), but my initial thought was that something more serious must have happened (well, duh!), perhaps a fire outside of the main suppression system (outside of the raised floor area?). Or perhaps the paint on the walls/carpet wasn't fire resistant and just took hold very quickly.

    Or a large initial fire (gas leak?) that just didn't die down when the supression system kicked in. Maybe the type of fire (again, gas? oil?) didn't die down because Data Center supression systems presumably focus on electrical fires.

    A well, just my 2p's worth.

    --
    ooooooh! What does this button do? - DeeDee, Dexters Lab.
  6. Re:now the engineers come out... by Martin+Blank · · Score: 5, Interesting

    Most NOCs I've been in have an oxygen-deprivation gas that's dispersed in the case of a fire (after a series of highly visible and audible warnings). The one I'm in right now seems to have gone the inexpensive (and arguably safer) way of a two-stage dry-pipe water sprinkler system where the pipes are usually dry (empty). In case of a smoke detection above a certain level, the pipes are "charged" (filled with water), and if the heat gets above a certain level, the sprinklers go off. Basically, the decision was made that if there's a fire, the equipment will probably need replacement anyway, so why not use sprinklers?

    Personally, I'm not too sure of this route. I can understand it, from an environmental and human-safety perspective (the gasses eat the ozone layer, and you *really* don't want to inhale stuff that ties up oxygen at those rates), but if it ever happens here, it's going to be an awfully hurried mess to get everything back up and running in a reasonable timespan, even if the fire is only a small one that doesn't destroy the building.

    --
    You can never go home again... but I guess you can shop there.
  7. Warning Sounds by Anonymous Coward · · Score: 1, Interesting

    If you ever smell smoke in a computer enviroment, take it serious! I once was in a computer pool room and only the smell of burning paper made me aware of the fire. Only then i realized a beeping noise, simlar to the noises some computers or cardreaders make to draw maintainace attention towards them.
    In that environment an old fashion siren would have been a much better solution.

  8. Re:A good reminder.. by Martin+Blank · · Score: 3, Interesting

    Cantor Fitzgerald, the company hardest-hit in the attack with 700+ employees killed, had just that viewpoint. eSpeed, a subsidiary of CF, had all trading activities back up and running by the morning of Sept 13, due to the multiple locations and real-time backups being done on the network. My understanding is that the time from loss of connection with CF headquarters to full resumption of its trading activity around the world was about 46 hours.

    --
    You can never go home again... but I guess you can shop there.
  9. Halon systems aren't illegal, but.... by mks113 · · Score: 4, Interesting

    You can't install new halon systems, but existing ones are still legal (in Canada, anyway).

    However, I wouldn't want to have to fill out all the paperwork involved with a discharge! We had an accidental discharge (a leak, I believe) and they decided it was enough impetus to remove the system.

    I think they are using CO2 now. The advantage of Halon is that you can breathe quite comfortably in an atmosphere that will not sustain fire. CO2 works just about as effectively but will not sustain life.

    OTOH, more recent studies have shown that just because you can still breath in a Halon infiltrated environment doesn't mean that there are no health effects!

    I expect there are more CO2 systems going in now, with lots of alarms to make sure people get out before the atmosphere gets unlivable.

    Michael

    1. Re:Halon systems aren't illegal, but.... by srpatterson · · Score: 2, Interesting

      We've got a couple of server rooms at work with big (5ft tall) CO2 extinguishers, you have 10 seconds to leave the server room after they activate.

      Strangely enough, there are no servers in there, just hubs, routers and comms gear.

      --
      -- The Heineken Uncertainty Principle: You can never be sure how many bears you had last night.
  10. Re:now the engineers come out... by Nos. · · Score: 3, Interesting
    We have a similar two stage system as mentioned above. When the installers came by a year later for a test, they must have "forgotten" how they hooked it up. Before the sprinklers actually start spraying water, the breaker that runs the power to that room is automatically flipped. The UPS we purchased for the servers is also hooked up so it doesn't do a safe power down, it just powers down.

    Well, they came in to do the test, everything but actually spraying water, and managed to flip the breaker. It was interesting to note, that only a few of the machines actually powered off. Some are on smaller single machine UPSs, so those stayed running. The most expensive piece of equipment in there, a Nortel Telephony Switch which handles 6 PRI lines (almost 150 ISDN lines) is on battery backup, and kept running.

    All in all very little powered down. In the even of a real fire, we would have had sparks flying everywhere. What did management do when I mentioned this to them? That's right, nothing.

  11. Re:now the engineers come out... by CaptainZapp · · Score: 3, Interesting
    NOC's have a fire *supression* system based on gas or (more likely these days) some form of liquid (gas has the nasty side-effect of potentially killing humans).

    When I worked for DEC they actually had sprinkler systems in their data centers (I couldn't verify first hand that every data center had sprinkler, but I was told so).

    The reasoning was, that even if they was one big, bad, evil, wicked water damage in a data center, that the company considered this to be better then one killed employee by gas extuinguishing systems.

    That was in the early 90ties

    --
    ich bin der musikant

    mit taschenrechner in der hand

    kraftwerk

  12. Re:A good reminder.. by Martin+Blank · · Score: 3, Interesting

    My mistake: 47 hours.

    Details on the recovery:
    http://www.baselinemag.com/article2/0,3 959,36807,0 0.asp

    Some information on the scale of what eSpeed handles:
    http://wstonline.com/story/mag/WST200110 08S0007

    Even Cantor Fitzgerald mentions it in their Q3/01 results summary:
    http://www.cantor.com/articles/article11 192001.htm

    --
    You can never go home again... but I guess you can shop there.
  13. Re:In other news by JohanV · · Score: 2, Interesting

    Nonsense. UT used to be the largest user in the Netherlands until about 14 months ago. After that, they were surpassed by Delft University of Technology, which until the fire today used about 50% more bandwidth as the UT. And even they aren't the biggest anymore, because the University of Utrecht recently became larger than them.
    In total, about 20.000 rooms for individual students in the Netherlands are responsible for over 1 Gbps of traffic. This is non-stop (i.e. during the evening it is closer to 2 Gbps), and over 90+ % is outgoing traffic from 'unknown ports'.

    I guess forcing the RIAA and the MPAA to change their business models the hard way counts as innovation nowadays :)