Slashdot Mirror
University of Twente NOC Destroyed
JanJoost writes "Around 08.00 CET today the University of Twente Network Operations Center, which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org, caught fire.
The UT, which hosted the HAL in august last year is completely unreachable and is not likely to come back up any time soon. The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings.
More information can be found at the Telegraaf,
Planet Internet and Twentsche Courant.
Pictures can be found here and here.
It's a shame to see a great infrastructure go down in flames like this."
From debian-devel, here's a slightly (only slightly) more informative blurb
More images can be found on http://www.bsdfreaks.nl/files/brand.htm
.sig: No such file or directory
..to see how this could be prevented in the future. How much fire protection do NOCs owned by the big boys (Verio, WorldCom) have? Offsite backups, too, I hope?
"which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org "
security.debian.org and non-us.debian.org are GONE (well, burning, hopefully data can be recovered). Yes there's mirrors, but it still sucks.
Free Mac Mini
This shows the vunerability of putting all computers in one building. To have a safe network one needs to spread (duplicate) the information over several computer at several locations. How far apart these locations has to be is depending on how important you data is.
It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup. By that I don't only mean that tape that always seems to go missing when needed, but multiple sites (or at least buildings) that provide redundancy.
To never keep back-ups in the same physical location.
This SIG pulled due to lack of funding. (This damn war is costing too much!)
This is something I keep hearing from people, particularly those who balk at paying for upgrades to expensive fire suppression systems. "They're always built so well that there's never been a case of a server room destroyed by fire." I always did doubt that, and now whenever they tell me that, I can point to this.
What a shame.
You can never go home again... but I guess you can shop there.
The University of Twente's attempts to overclock the new AMD Opteron and Nvidia GeForceFX card in the same case are declared a failure. "We certainly won't be building a Beowolf cluster of these..." commented a spokesperson.
UNIX? They're not even circumcised! Savages!
Someone took "firewall" too seriously? :) :p)
(and this isn't *flame*bait
"The majority is always sane, Louis." -- Nessus
http://slashdot.jp
Was anyone killed?
If not, was anyone hurt?
If not, do they have insurance?
If they do... well, I'm sure someone just lost their masterpiece pr0n directory, but otherwise, things like this happen. (ask Hemos) You have to make it through such things. In this case, it was a commercial (educational) building and no one is homeless, so it's less of a tragedy than usual. Let's hope that they rebuild with something better and newer.
That said, I get the feeling that those plumes of smoke really are millions of dollars floating away in the wind...
lo0.ar5.enschede1.surf.net 3613: Nov 20 07:20:50.927 UTC: %ENV_MON-2-TEMP: Hotpoint temp sensor(slot 18) temperature has reached WARNING level at 61(C)
few seconds later on the local side:
lo0.cr2.amsterdam2.surf.net 1146: Nov 20 07:20:56.458 UTC: %CLNS-5-ADJCHANGE: ISIS: Adjacency to ar5.enschede1 (POS2/0) Down, interface deleted(non-iih)
I hope Debian practices good management principles by having offsite backup.
Best Slashdot Co
They're not gone. They're just experiencing a Distributed Combustion Denial-of-Service attack.
In case one might be interested, Essent mirrored security.debian.org.
You can use debian.essentkabel.com to download the latest security updates (in case you haven't already). Please note this is NOT an official mirror.
Daxy's Networking Blog
Who posted a link to UT's webserver on slashdot?
Duct tape + WD40 => DevOps
Also, here's what seems to be the only close up I could find of the fire. pics
No man is an island, But if you take a bunch of dead guys and tie them together, they make a pretty good raft.
Effective as it was, Halon was also a major ozone-hole cause, so it has been outlawed in most countries by now.
Dangerous server rooms
Here's an update from SARA (that's where I work), the network operator for SURFnet. SURFnet is very busy ordering new equipment and fixing the 2 x 10 Gbit/s lamda's to Enschede. We hope to restore connectivity a.s.a.p. Greetings, Marco
"Witnesses saw a large balding man monkey dancing from the scene and a slim geeky man with glasses trailing behind continuously adjusting his glasses." An in other news....
The Dutch LFS FTP mirror was also hosted at the University of Twente, which means it's also down. The Dutch HTTP mirror should work properly, since it's outside utwente.nl space.
Last news is that HP (Who supplied most of the UT backbone equipment) is on its way with emergency equipment to have things up and running somewhere tomorrow.
The fire did not start in the server rooms. What happened was that a fire started in one of the lecture rooms (and a smallish one at that, number A108) that just happened to be in the same wing of the TWRC building that also housed the server rooms (yes, multiple). It then proceeded to take out two entire wings and threaten other buildings nearby as well as the library.
Now, I would *love* to see a halon system capable of stopping that...
Owh, and the fire seems to be under control by now, as evidenced on http://webcam.traserv.com/
(which you can contrast to http://www-infstud.sci.kun.nl/~arthurvl/ispy.jpg (taken at about 09:40 CET this morning)).
I believe, the open and distributed network of Keyserver.net (distibuted network of PGP keyservers) was hosted by SURFNet. This network is a distributed network holding PGP and OpenPGP keys. The loss the to UT NOC could have an impact on the updating of key-rings across the keyserver.net network.
Actually, Debian security updates are usually also added to the proposed-updates section, which means that they are available on nearly all Debian mirrors worldwide.
Something like /etc/apt/sources.list should work well.
deb ftp://ftp.XX.debian.org/pub/debian/ proposed-updates main contrib non-free
deb ftp://ftp.XX.debian.org/pub/debian-non-US/ proposed-updates/non-US main contrib non-free
(replace XX with your ISO country code) in your
You can find mirrors on Debian's website.
O frabjous day! Callooh! Callay!
Now we can find out how secure and hardened Debian really is. You are as good as your latest backup.
BACKUPS BACKUPS BACKUPS Off-site! I've had enough of people who are talking about RAID-5 because 5TB tape drive arrays are too slow. Always keep your BACKUPS!
A caveman dreams of being us, the incalculable power and riches. We dream of being Q, then what?
You can't install new halon systems, but existing ones are still legal (in Canada, anyway).
However, I wouldn't want to have to fill out all the paperwork involved with a discharge! We had an accidental discharge (a leak, I believe) and they decided it was enough impetus to remove the system.
I think they are using CO2 now. The advantage of Halon is that you can breathe quite comfortably in an atmosphere that will not sustain fire. CO2 works just about as effectively but will not sustain life.
OTOH, more recent studies have shown that just because you can still breath in a Halon infiltrated environment doesn't mean that there are no health effects!
I expect there are more CO2 systems going in now, with lots of alarms to make sure people get out before the atmosphere gets unlivable.
Michael
I mean seriously, each tower collapsed because it was hit by its own plane. If one tower had been in NY, one in California and both were still hit by a plane each, the result would be exactly the same.
The lesson should be: Primary back-up is a very good start, but secondary/tertiary back-up is the thing if it's that critical.
Kjella
Live today, because you never know what tomorrow brings
... finally gets to make an appearance in earnest.
'lp1 on fire'
Real Daleks don't climb stairs - they level the building.
Do full backups weekly, store copies offsite. Incremental backups daily, copies offsite also. If you can afford it (or can't afford any downtime), have emergency backup hardware (enough for minimal operations) in an offsite storage facility. Old hardware that would otherwise be thrown out is good for this (remember, it's for an emergency). Have a supplier who can get replacement hardware to you in a hurry (so you can get off of those old 90 MHz Pentium servers).
The most vital part of the plan, after backups, is good insurance. If the building burns to the ground Monday morning, you want to be able to call the insurer Monday Noon, and have the check in hand Tuesday morning at the latest.
These recommendations do not cover disasters such as 767s flying into the building and killing all the sysops. Earthquakes dropping the building on the same. Etc. The people are the most important part of any company and, if too many of them are lost at once, the company probably is lost too.
Unless you have really good (and expensive)insurance which can provide enough funds for you to hire new people, get them trained, and keep the company solvent while you do so.
Best Slashdot Co
Some damn fool wrote an assembly program that used the dreaded HCF instruction, didn't they?
Best Slashdot Co
Quote from debian-devel-announce
At around 8 this morning (local time) a fire started in the computing facilities of Twente University. This affects Debian, since one of our servers (satie) is hosted there. At this moment it seems very likely that the machine can not be recovered from the fire.
The following services are currently down as a result of this:
security.debian.org
non-us.debian.org
nm.debian.org
qa.debian.org
We are working to restoring these services on another machine and hope to have things in mostly working order by tomorrow. Security advisories are still available at http://www.debian.org/security/
Wichert.
fact is that the UT is responsible for 30% of the daily internet-traffic in the Netherlands :)
The advantage of Halon is that you can breathe quite comfortably in an atmosphere that will not sustain fire. CO2 works just about as effectively but will not sustain life.
You've obviously never been in a Halon dump. Either that or you consider burning in your lungs to be "quite comfortable". Not to mention that if you're standing under one of the discharge nozzles at a dump you can get a nasty case of frostbite.
I've personally been through two 1211 dumps and had to enter a computer room and drag staff out after an FM200 dump. It takes about two days to completely stop coughing.
- If we aren't supposed to eat animals, then why are they made out of meat? - Steven Wright
Twente's high-speed network was originally constructed to provide students with access to high-speed Internet access for their studies. It soon became one of the major hubs for peer-to-peer exchange programs like KaZaA. This perceived misuse of Internet resources caused former Dutch education minister Loek Hermans to comment: "It would be nice if the students at Twente University would use their fast connections for information and education purposes, instead of downloading huge amounts of porn."
Phil, just me
"Cattle Prods solve most of life's little problems."
Maybe it was a flame war between students? Were they overclocking? The dangers of using FireWire. Were their harddrives Quantum Fireballs? (They are now) Is this what you get when you try to hot-swap them?
Other sad jokes will no doubt follow.
So wait, the University of Twente NOC caught fire. Why? Was there a story posted on Slashdot's front page that linked to a server at their location? Or was this fire caused by something other than a hardy slashdotting?
Why bother.
Our ISP bought an old legal office building for their HQ and colo facility. The place was built with file rooms to safeguard tons of irreplaceable paper documents - imagine thick concrete walls & ceilings, with heavy steel fire doors, rated to preserve the contents through an EVERYTHING-else-burned-to-the-ground fire.
Critical stuff is spread between the file rooms, with metal conduit, etc. protecting the few small holes they added for wiring.
Steel & reinforced concrete aren't quite obsolete.
It's easy to make up & spread cool- and credible-sounding stuff. Finding & checking hard facts is hard work.
Halon (or at least most of the Halons used for fire suppression; not sure about all of them) is non-toxic, though it'll get you a bit high, like nitrous oxide or most solvents, so being stuck in the stuff won't injure you quickly (except from flying objects that were blown around by the gas pressure.) However, it's no substitute for oxygen, and you'll probably be wanting some oxygen real soon now. If you can remember not to breathe in the stuff, try not to breathe deeply, because there's more oxygen left in your lungs than the stuff you'll be breathing in, and unfortunately, while your body can generally figure out not to breathe in water or hot smoke, it's not as good at realizing that near-room-temperature inert gases aren't very useful. Mostly, don't worry about it - find a safe door to run for and run for it, and do some deep breathing once you're outside, and try to close the door behind you.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks