Slashdot Mirror
University of Twente NOC Destroyed
JanJoost writes "Around 08.00 CET today the University of Twente Network Operations Center, which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org, caught fire.
The UT, which hosted the HAL in august last year is completely unreachable and is not likely to come back up any time soon. The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings.
More information can be found at the Telegraaf,
Planet Internet and Twentsche Courant.
Pictures can be found here and here.
It's a shame to see a great infrastructure go down in flames like this."
... I hope nobody was hurt.
After that, I wish them luck getting back online.
42 - So long and thanks for all the fish.
..to see how this could be prevented in the future. How much fire protection do NOCs owned by the big boys (Verio, WorldCom) have? Offsite backups, too, I hope?
To never keep back-ups in the same physical location.
This SIG pulled due to lack of funding. (This damn war is costing too much!)
Was anyone killed?
If not, was anyone hurt?
If not, do they have insurance?
If they do... well, I'm sure someone just lost their masterpiece pr0n directory, but otherwise, things like this happen. (ask Hemos) You have to make it through such things. In this case, it was a commercial (educational) building and no one is homeless, so it's less of a tragedy than usual. Let's hope that they rebuild with something better and newer.
That said, I get the feeling that those plumes of smoke really are millions of dollars floating away in the wind...
Is there anything that could cause this naturally? I mean, judging from the pictures it looked fairly large and out of control...I hope no one caused this purposefully. I've never had to deal with a catastrophe like this, luckily...I send my best wishes of luck and hope to those involved and pray no one got hurt. =(
Zech Harvey, MCSE, MCDBA, CCNA
I.E. "See, if they would have had a fire suppression system, this would have never happened."
No man is an island, But if you take a bunch of dead guys and tie them together, they make a pretty good raft.
I believe, the open and distributed network of Keyserver.net (distibuted network of PGP keyservers) was hosted by SURFNet. This network is a distributed network holding PGP and OpenPGP keys. The loss the to UT NOC could have an impact on the updating of key-rings across the keyserver.net network.
The fact that "The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings" leads me to believe that the fire didn't start in the server area. Lots of server rooms were destroyed on September 11, for example, but it wasn't the fault of the room's design, or the presence or lack of fire suppression systems. If the whole building is burning down, fire suppression in one room is only going to work until the floor and ceiling collapse.
Now we can find out how secure and hardened Debian really is. You are as good as your latest backup.
BACKUPS BACKUPS BACKUPS Off-site! I've had enough of people who are talking about RAID-5 because 5TB tape drive arrays are too slow. Always keep your BACKUPS!
A caveman dreams of being us, the incalculable power and riches. We dream of being Q, then what?
I mean seriously, each tower collapsed because it was hit by its own plane. If one tower had been in NY, one in California and both were still hit by a plane each, the result would be exactly the same.
The lesson should be: Primary back-up is a very good start, but secondary/tertiary back-up is the thing if it's that critical.
Kjella
Live today, because you never know what tomorrow brings
I just lost my breakfast. Thanks ever so much.
If you want to read about the fire, go here. Apparently UT was a major node for KazAA, and'a primo source for warez and pr0n.
Wanted: One witty yet thought provoking
Halon DO NOT replace oxygen in the room to extinguish the fire. It breaks the chain reaction of fire, basically stop the elements of fire to react with each other.
Most scenario would only require a less than 8% of concentration to take out the fire. Under 10% and you can still breath.
Problem of Halon is when over 900 degree C, it breaks down into hydrogen fluoride, hydrogen bromide and bromine - stuff that are toxic. So, run!
Do full backups weekly, store copies offsite. Incremental backups daily, copies offsite also. If you can afford it (or can't afford any downtime), have emergency backup hardware (enough for minimal operations) in an offsite storage facility. Old hardware that would otherwise be thrown out is good for this (remember, it's for an emergency). Have a supplier who can get replacement hardware to you in a hurry (so you can get off of those old 90 MHz Pentium servers).
The most vital part of the plan, after backups, is good insurance. If the building burns to the ground Monday morning, you want to be able to call the insurer Monday Noon, and have the check in hand Tuesday morning at the latest.
These recommendations do not cover disasters such as 767s flying into the building and killing all the sysops. Earthquakes dropping the building on the same. Etc. The people are the most important part of any company and, if too many of them are lost at once, the company probably is lost too.
Unless you have really good (and expensive)insurance which can provide enough funds for you to hire new people, get them trained, and keep the company solvent while you do so.
Best Slashdot Co
It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup.
Uhhh, yes there is... I suspect you either know nothing about IT or are fresh out of college. DRP (disaster recovery planning) factors in things such as criticality of data, cost, and acceptable downtime. A university payroll system may need to be back up within 12 hours of a major incident, so in addition to tape backups you might have a failover site. Contrary to your simplistic post, even the richest corporations rarely have failover sites of their own. They simply contract out to a DRP vendor who have these types of machines lying dormant in a glass room, waiting to cut over. On the other hand a university FTP site is probably classified as low risk, low impact. So you would rely on off-site backup tapes and perhaps only restore when you've arranged for an alternate site and taken delivery of new servers. You don't pay millions of dollars to have two glass rooms just so you can have uninterrupted FTP service...
Try doing that on a university-wide multi-gigabit capable network on the budget of the average Dutch university. Our universities aren't like M$ in cash, you know. I know, I'm an admin at one myself.
:-)
I just hope they're well insured....poor colleagues...
On the upside: they may get a squeaky-clean start when this blows over
--
That's exactly what the Mojo Nation folks are doing now. Info at http://www.mojonation.net/.
fencepost
just a little off
Since Debian doesn't have a 40 Billion dollar monopoly warchest to draw from, I'm assuming that some funds will need to be raised to get new boxes. This is a perfect time for uses to step up and make a contribution back.
*** Sigs are a stupid waste of bandwidth.
Umm, so? Outgoing traffic is almost invariably from a random port > 1024. That's how TCP connections are generally done.
"Convictions are more dangerous enemies of truth than lies."
Our ISP bought an old legal office building for their HQ and colo facility. The place was built with file rooms to safeguard tons of irreplaceable paper documents - imagine thick concrete walls & ceilings, with heavy steel fire doors, rated to preserve the contents through an EVERYTHING-else-burned-to-the-ground fire.
Critical stuff is spread between the file rooms, with metal conduit, etc. protecting the few small holes they added for wiring.
Steel & reinforced concrete aren't quite obsolete.
It's easy to make up & spread cool- and credible-sounding stuff. Finding & checking hard facts is hard work.
Why does it need to provide oxygen? The air provides oxygen. How come you're not worried that air is about 80% nitrogen, which also provides no oxygen? See, humans don't need to breathe 100% oxygen... we do fine with much less. The advantage of Halon over CO2 is that it does not extinguish a fire by displacing oxygen. It will put out a fire at concentrations of about 5%, leaving plenty of oxygen to breathe.
the burned Halon/air/diesel mixture produces some really nasty toxic gasses.
And a fire doesn't? If you have an undersized system installed, you're gonna have problems in a fire anyways. With a proper system, the small quantity of toxic gasses produced by the Halon decomposition before the fire is extinguished (which is a fraction of a second--Halon systems have been used for explosion suppression) is much preferable to the large quantity of toxic gasses and heat produced by a fire.
After that, I wish them luck getting back online.
Someone explain how the FUCK that can be considered insightful?
Yet again we see the need for moderators to pass an IQ test.