University of Twente NOC Destroyed

JanJoost writes "Around 08.00 CET today the University of Twente Network Operations Center, which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org, caught fire. The UT, which hosted the HAL in august last year is completely unreachable and is not likely to come back up any time soon. The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings. More information can be found at the Telegraaf, Planet Internet and Twentsche Courant. Pictures can be found here and here. It's a shame to see a great infrastructure go down in flames like this."

Comments

More info

[Strike]

From debian-devel, here's a slightly (only slightly) more informative blurb

Re:More info

[Stonehead]

Hey, I posted that to debian-devel. Scary to see it get Slashdot headlines, since this posting from Wichert Akkerman himself is more 'official'.
He's now probably busy setting up klecker.debian.org as the next security.debian.org host. Don't get yourself trojaned, please people, don't panic and just wait for the official Debian announcement that everything has been fixed again. Or play around with inofficial mirrors like these, and there are more. But I feel a bit stupid myself, because - unlike Wichert - I have done nothing myself except forwarding the news and act like a karma whore.

Well Damn...

[jhines0042]

... I hope nobody was hurt.

After that, I wish them luck getting back online.

Re:Well Damn...

[Koos]

I hope nobody was hurt.

All news reports say people were evacuated in time from the building and nobody got hurt.

Funny, I was in this building a few times during the buildup and cleanup of HAL2001. Feels strange now.

Re:Well Damn...

[Fembot]

I hope they had offsite backup of anything important :-)

Photo's

[fearlezz]

More images can be found on http://www.bsdfreaks.nl/files/brand.htm

In other news

[roguerez]

Capacity usage of the world largest interconnection lines roughly halved today.

The cause of this is thought to be a steadily decline in warez/divx/mp3 traffic. The underlying reason for that is still unknown.

Re:In other news

[MonoSynth]

fact is that the UT is responsible for 30% of the daily internet-traffic in the Netherlands :)

Re:In other news

[JohanV]

Nonsense. UT used to be the largest user in the Netherlands until about 14 months ago. After that, they were surpassed by Delft University of Technology, which until the fire today used about 50% more bandwidth as the UT. And even they aren't the biggest anymore, because the University of Utrecht recently became larger than them.
In total, about 20.000 rooms for individual students in the Netherlands are responsible for over 1 Gbps of traffic. This is non-stop (i.e. during the evening it is closer to 2 Gbps), and over 90+ % is outgoing traffic from 'unknown ports'.

I guess forcing the RIAA and the MPAA to change their business models the hard way counts as innovation nowadays :)

Re:In other news

[osu-neko]

...and over 90+ % is outgoing traffic from 'unknown ports'.

Umm, so? Outgoing traffic is almost invariably from a random port > 1024. That's how TCP connections are generally done.

Re:In other news

[Mnemia]

Except that outgoing traffic is normally low-bandwidth. When you use a web browser the outbound traffic is very small compared to the inbound traffic. So the only thing that could generate that much outbound traffic is servers, and since most "legitimate" server applications use low numbered ports, we're likely talking about P2P servers. I think that was what your parent poster was implying (though that ratio seems like it's got to be an exaggeration to me).

Re:In other news

[evalhalla]

traffic on the low numbered ports is low-bandwith: after the request has been made (on the port on which the server is listening) all of the data goes on a random > 1024 one.

anyway it is quite probable that in this case also most of the requests are for high-numbered ports.

now the engineers come out...

[ravidew]

..to see how this could be prevented in the future. How much fire protection do NOCs owned by the big boys (Verio, WorldCom) have? Offsite backups, too, I hope?

Re:now the engineers come out...

[Yo Grark]

After 911 all the big techfirms created DRA's (disaster recovery area's) in case anything major happened.

I had no less than 1000 companies call asking how to handle licensing and DRA's.

I'd be surprised if they didn't have one, it's pretty much standard practice comercially....

Yo Grark

- Canadian Bred with American Buttering

Re:now the engineers come out...

[rleyton]

Whilst I'm no expert in fire supression systems, I have studied a few data centres in my time as a Senior Sys Admin for a number of companies. Most (decent) NOC's have a fire *supression* system based on gas or (more likely these days) some form of liquid (gas has the nasty side-effect of potentially killing humans).

These are for small fires that can be contained within a data centre, ie. a computer catching fire or emitting smoke. Really good systems are very localised (racks or cage specific). A big fire just isn't going to be stopped by such a system.

I haven't read too much into this particular incident (ie. not at all), but my initial thought was that something more serious must have happened (well, duh!), perhaps a fire outside of the main suppression system (outside of the raised floor area?). Or perhaps the paint on the walls/carpet wasn't fire resistant and just took hold very quickly.

Or a large initial fire (gas leak?) that just didn't die down when the supression system kicked in. Maybe the type of fire (again, gas? oil?) didn't die down because Data Center supression systems presumably focus on electrical fires.

A well, just my 2p's worth.

Re:now the engineers come out...

[Martin Blank]

Most NOCs I've been in have an oxygen-deprivation gas that's dispersed in the case of a fire (after a series of highly visible and audible warnings). The one I'm in right now seems to have gone the inexpensive (and arguably safer) way of a two-stage dry-pipe water sprinkler system where the pipes are usually dry (empty). In case of a smoke detection above a certain level, the pipes are "charged" (filled with water), and if the heat gets above a certain level, the sprinklers go off. Basically, the decision was made that if there's a fire, the equipment will probably need replacement anyway, so why not use sprinklers?

Personally, I'm not too sure of this route. I can understand it, from an environmental and human-safety perspective (the gasses eat the ozone layer, and you *really* don't want to inhale stuff that ties up oxygen at those rates), but if it ever happens here, it's going to be an awfully hurried mess to get everything back up and running in a reasonable timespan, even if the fire is only a small one that doesn't destroy the building.

Re:now the engineers come out...

[Angry White Guy]

Offsite backups are one thing, but in a case of sheer devistation (such as this), it really doesen't help too much. They have to find a building, fiber, computers, cabling, everything to replace it. Backups, although an important part in the puzzle, are ultimately the last, centermost peices.

After all is said and done, the U still has to deal with insurance companies, getting specs for colocated equipment, etc. It's not an easy process (if you've dealt with your insurance company not wanting to pay for a dent in your car, imagine telling them that the dent would cost millions to repair) and I doubt that they will be up and operational in less than 6 months.

I just find it ironic that they put all of their fault-tolerant redundant systems in the same building, only to have it burn down. Distributed computing should mean more than just two computers sitting right next to each other.

Re:now the engineers come out...

[ihowson]

I guess you'd also have to cut power if it got to the point where the sprinklers were going to go off... I can think of much better places to be than in the middle of a server room with water going everywhere and tons of POWERED ON computers. (bzzzt)

I've never seen a full-on datacentre (we don't have many in .au, although I work a 2-minute walk from one) - I was under the impression that rooms with Halon were usually isolated in some way from the rest of the NOC to prevent people getting trapped in there?

Re:now the engineers come out...

[Nos.]

We have a similar two stage system as mentioned above. When the installers came by a year later for a test, they must have "forgotten" how they hooked it up. Before the sprinklers actually start spraying water, the breaker that runs the power to that room is automatically flipped. The UPS we purchased for the servers is also hooked up so it doesn't do a safe power down, it just powers down.

Well, they came in to do the test, everything but actually spraying water, and managed to flip the breaker. It was interesting to note, that only a few of the machines actually powered off. Some are on smaller single machine UPSs, so those stayed running. The most expensive piece of equipment in there, a Nortel Telephony Switch which handles 6 PRI lines (almost 150 ISDN lines) is on battery backup, and kept running.

All in all very little powered down. In the even of a real fire, we would have had sparks flying everywhere. What did management do when I mentioned this to them? That's right, nothing.

Re:now the engineers come out...

[Anonymous Coward]

After 911 all the big techfirms created DRA's (disaster recovery area's) in case anything major happened.

And six months later, 80% of them scrapped or seriously cut their disaster recovery plans because they were too expensive.

Most people in chare of creating disaster recovery plans were told something like 'We need to be able to be back in busness 100% within 24 hours, but we can't spend any money (or can only spend a woefully indadaquate amount of money) on new servers, redundant network lines, improved backups, offsite tape storage, etc.'

Re:now the engineers come out...

[AlphaInsight]

Actually FM-200 and Inergen are the current replacements for Halon. Non Ozone-depleting and won't kill you if you're trapped in a NOC. Just lowers the avaliable O2 in the air. You'll get out of breath easily, but as long as you stay calm you won't have to worry.

Re:now the engineers come out...

[Anonymous Coward]

"gas has the nasty side-effect of potentially killing humans"

This hasn't been a real problem since the original Halon systems. There is generally a long protracted countdown (15-20secs is a long time when talking about fires.) even on the newer systems such as inergen. Potentially dangerous gases have breathing apparatus in the protected area just in case. (ie: someone is electrocuted, fire, can't move person, grab breathing mask, put on their head (or grab two and move them) and leave room.)

Though these systems are very expensive. We have the largest data center (by floorspace) in Queensland (multi-storey). When someone "mistook" the "release without interrupt" button for the exit door release, it cost AUS$10-12 thousand to replace the gas(inergen). Of course now it's _much_ harder to make this mistake. (Involving a hammer and a glass cover).

But then again, the person in question was known to play indoor cricket between the mainframes. I'm sure that a fast moving ball had absolutely nothing to do with tripping an inset switch on the wall.

Re:now the engineers come out...

[CaptainZapp]

NOC's have a fire *supression* system based on gas or (more likely these days) some form of liquid (gas has the nasty side-effect of potentially killing humans).

When I worked for DEC they actually had sprinkler systems in their data centers (I couldn't verify first hand that every data center had sprinkler, but I was told so).

The reasoning was, that even if they was one big, bad, evil, wicked water damage in a data center, that the company considered this to be better then one killed employee by gas extuinguishing systems.

That was in the early 90ties

Re:now the engineers come out...

[Mysticalfruit]

Which DEC building did you work in? The company I work for is in the Powdermill Rd. Building and our Lab's are still all sprinklers. However the way the system is setup, as soon as a sprinkler head goes off, the power shuts off.

Re:now the engineers come out...

[Martin Blank]

As was mentioned by someone else, these systems trip the power circuits before they go off. Unlike one of the other posters, though, we have no alternate UPS systems aside from the phone switch battery backups, which are kept a couple of feet off the ground in water-proof containers to minimize the chances of sparks from short circuits.

Re:now the engineers come out...

[Lumpy]

Ahem... flooding a room with carbon dioxide does nothing to the environment or hurts you other than oxygen deprivation. yet this is the system in my server room.. one button or the temprature sensor goes off.. the doors automatically close (steel fire doors) and the room is flooded with C02.

HALON also did not eat up oxygen... it displaced it. and it's msds sheets also shown the human hazard to be minimal other than oxygen deprivation.

there are no oxygen consuming fire supression systems on the market. it's displacement or smothering or temperature reducing (water).

Re:now the engineers come out...

[Stephen Maturin]

-- Which is why most LARGE companies generally do not maintain their own backup data center for their business continuity plan, but lease space/capacity from companies like SunGard, which provide offsite storage & backup data centers.
Even with scheduling regular business continuity plan tests & sending people offsite for a week or so, these plans are cheaper than maintainig a 2nd data center.

Re:now the engineers come out...

[arri]

I am not doubting that these new gases will allow you to survive but I've been too close for comfort to a fire in a Chemistry lab and without adequate training (not in the use of a fire-extinguisher but in the behaviour in case of fire) the last thing I did was "stay calm"... I just hope that places which install this system give adequate training to their people as to what can be done.

Even the obvious like "lie low as smoke rises and there is more oxygen at ground level" and to "try and cover your mouth & nose". At least, this is what I learned from that incident, sadly after the fact.

Re:now the engineers come out...

[buckeyeguy]

Our server room also has a water sprinkler system... due to less-than-forward-looking regulations in the dumb little suburb we are located in, all areas of the building HAD to have them, regardless of any other fire-suppression we might have paid for. Hope no one lights a match...

Re:now the engineers come out...

[rosewood]

The key is insurance moneys which will buy them all new toys and we will be better off then when we started.

Re:now the engineers come out...

[EMH_Mark3]

Heh. You're trapped in a room filled with fire & smoke, the oxygen is being sucked out of the room and you want to stay calm? Sure, no problem :)

Re:now the engineers come out...

[kriston]

If carbon dioxide is used you face almost certain asphixiation and death. This happens now and then in apple country where the apples are stored in a mostly CO2 environment in huge air-tight warehouses.

Kris

Re:now the engineers come out...

[Lumpy]

DUH... but you aren't going to die from the carbon dioxide... you die from lack of oxygen as i said in the post... CO2 and HALON are Non-toxic.

hell water can do what you said.... just ask a drowning victim.

Re:now the engineers come out...

[compwizrd]

Put a bucket under the head, with a spigot drilled into the bottom, run a hose down to the floor, and find a drain point from there.

Technically, the sprinkler head is still active

The sprinklers here are ABOVE the drop ceiling, i have yet to figure out how that's going to do any good for the time it takes to soak through a drop ceiling tile enough to make it collapse.

Re:now the engineers come out...

[barnaby]

A high concentraion of CO2 causes an almost irrestible urge to exhale and then inhale. And CO2 displaces oxygen much more "agressively" than halon.

At least thats what they told me in Navy fire fighting school when explaining why the Navy shifted from CO2 to halon for the fixed fire suppression systems in some spaces

Re:now the engineers come out...

[strictnein]

You'll get out of breath easily, but as long as you stay calm you won't have to worry

And, as we all know, there's nothing more calming than being in a room that's on fire and having some funky gas poured into the room that makes it harder to breath

Re:now the engineers come out...

[kapelski]

I have 20 some servers in UUNet's (WorldCom's) Ashburn, VA datacenter (where MAE East is located). When I asked them about fire suppression they told me they use a two stage water system that is localized above many many zones throughout the data center. They said the idea is that since the datacenter is so large and they have so many customers with varying SLAs, their goal is to put the fire out at its source before it can spread, while attempting not to affect the operation of other cages/racks. In other words, apparently they don't want to saturate the whole datacenter, just the location of the fire. If something catastrophic happened that would affect the whole of such a large building, then pretty much everything is a loss anyway. Nevertheless, I trudge out there once a week to take backup tapes to an offsite vault...

I also asked them what happens if a plane crashes on takeoff from Dulles airport (only a few miles away and where the plane that hit the Pentagon took off from) and hits the datacenter, and they said they had the flight path altered when they built the datacenter to minimize the chance of such a thing. They didn't admit to it, but I bet that that building is hardened, too.

Re:now the engineers come out...

[haligan]

From my experience most properly installed and maintained fire suppression system CAN handle a normal fire. Unless you have an act of arson where someone spreads a load of some kind of accelerant or in case of 9-11 you had damage to the system itself and a large amount of accelerant. I know sometimes the Surround and Drown game has to be played, but I have seen some departments that do this in fire that could be stopped. Good fire inspections, visits from the companies that respond to the facility so that they become more familiar with it and good aggressive fire fighting is what should be done. Remember, I said that I realize that sometimes when it's blowing through the roof Surround and Drown is the only option left.

Re:now the engineers come out...

[Cramer]

Wrong. You can be standing in a room with plenty of O2 but too much CO2. If your body absorbs too much CO2, it'll alter your blood chemistry which will kill you with 100% certainty. (It doesn't take much of a change in pH to cause all sorts of problems.) Plus, CO2 displaces O2 with sufficient efficency to prevent adiquate O2 from entering the body (and CO2 to leave.)

Go talk to the experts at NASA about O2 and CO2 levels.

See also: Indoor Air Quality

• Carbon dioxide
  
  Carbon dioxide is a non-toxic gas. It has beneficial uses and is the "fizz" in carbonated beverages. When frozen, it is "dry ice". At concentrations of from 2,500 ppm to 5,000 ppm carbon dioxide can cause headaches. At extremely high levels of 100,000 ppm (10 percent) people lose consciousness in ten minutes, and at 200,000 ppm (20 percent) CO2 causes partial or complete closure of the glottis.

(It's a bit difficult to crawl out of the room after you've blacked out.)

Re:now the engineers come out...

[DunbarTheInept]

When you can't get oxygen from the surrounding air, the worst thing in the world to do is take a breath. You will survive much longer by fighting the urge to take a breath, and instead just hold the air already in your lungs in place until you can get out. You don't convert all the oxygen in your lungs into carbon dioxide with each breath - far from it. You just convert a portion of it, so the air you expell just has a smaller percentage of oxygen than the air you breathe in. But there's still quite a bit there that never got converted. (That's why mouth-to-mouth breathing can help someone - the air you breathe out still has enough oxygen in it to be a lot better than no air at all.)

But if you breathe that air out and breathe oxygen-less air in, you will pass out very fast. Most people have the misconception that you can do without oxygen for a minute or two before dying. That's not true - your body needs to consume fresh oxygen at a continuing rate just to function at all, it's just that your lungs can HOLD a small supply of oxygen to supply this need for a minute or two. Get rid of that oxygen by breathing it out and replacing it with oxygenless air, and you're going to pass out in just a few seconds, and be dead shortly thereafter.

And the worst part is you won't FEEL like anything is wrong. Your body is unable to measure the level of oxygen in your lungs. Instead your body senses the level of carbon dioxide in your lungs. As the by-product of normal breathing, when carbon dioxide has built up enough, that indicates you've converted a lot of oxygen and it's time for another breath. This is what triggers the automatic involuntary breathing that takes over when you stop thinking about it. This is also what triggers the panic feeling that you get when you know you need air. Your lymph nodes detect too much carbon dioxide and start sending the panic signal to your mind. What this all means is that if your body isn't exchanging oxygen for carbon dioxide, your body doesn't even realize it's asphixiating. If there's no oxygen in your lungs to start with, then there won't be any carbon dioxide building up in the lungs, and you will feel no sensation of needing a breath at all. You'll feel just fine for a few seconds and then *poof* you're gone as the blood going to your brain runs out of oxygen and your brain activity just plain stops.

So if you're ever in a halon gas system when it goes off - DO NOT BREATHE. Just hold whatever breath happens to already in your lungs and get out. The instinct is to hold your breath by first inhaling your lungs full and THEN holding it, but that's the worst thing you could do, as explained above. The tricky part is remembering to override that instinct.

Re:now the engineers come out...

[CynicTheHedgehog]

The FM-200 guy that trained us on the use of our fire-suppression system said that FM-200 causes an endothermic reaction with the fire--effectively sucking the heat out of it. He said it didn't do anything with the oxygen in the room, and you could stay and play in it as long as you wanted, provided the fire wasn't out of hand (FM-200 reactions produce some lung irritants with hot fires), but by then you'd be more worried about the fire itself.

Re:now the engineers come out...

[DunbarTheInept]

C02 is a poison to the human body if you have it in high concentrations. (That's the big air problem they had on Apollo 13. They had enough O2, but not enough filtering capacity to get rid of the CO2.) How quickly it would kill you depends on how diluted it is with other gasses in the air. the small percentage of it in the earth's atmopshere isn't enough to kill you, obviously. But being in a room full of it is NOT a good idea.

And CO2 has the very very nasty side effect of tricking your body into thinking it's choking and needs another breath whether it really does or not. (Which was another problen on Apollo 13 - the high level of CO2 was causing the crew to breathe way too fast.) Your body is unable to detect oxygen in the lungs. It only detects CO2. CO2 triggers your automatic involuntary breathing, and it triggers your panic feeling when you think you're asphixiating. Get excess CO2 in your lungs and you will be filled with an almost overwhelming urge to take another breath, which if you are in a room with lots of CO2, will make you have even more CO2 in your lungs and make you have even more physcal desire to take another breath, and so on. This is the worst thing to do because you are getting rid of what little oxygen is left in your lungs and replacing it with oxygen-less air.

Re:now the engineers come out...

[frost22]

but you aren't going to die from the carbon dioxide... you die from lack of oxygen as i said in the post...

Sorry, wrong.

Carbon dioxide is lethal up from about 30% air volume. No matter what the rest of the air consist of - even if that contains comfortably much oxygen. As far as I remeber my paramedic training, it's called an acidosis - essentially, CO2 at high percentages soures your blood, to a point where your metabolism cant handle it any more.

Re:now the engineers come out...

[frost22]

If NL Universities are like German ones, they don't have any insurance, as mandated by law.

The key idea is the state itrself is the largest solidarity community anyway, so paying insurance is waste of state funds.

Needless to say, in reality, you are fucked when something big happens to your computers. Maybe in next years Budget, you get some additional money to replace them. Or not.

Re:now the engineers come out...

[Bios_Hakr]

As long as they use distilled water in the system, it'd be ok. Water itself isn't conductive, just the impurities in it. You could flood the entire NOC and still have live traffic.

Of course, corrosion would be a major prob after the water was shut off.

Re:now the engineers come out...

[vkt-tje]

Those are pictures from may 2000. (the last one is clearly dated may 13th 2000).

On that day a fireworks factory exploded in the city. The university is out of the center and was not at all affected then.

This is a completely different, new story, fortunatly with less lives lost.

Re:why does this matter?

[Sc00ter]

can you read?

"which amongst other things hosts a SURFnet PoP as well as security.debian.org and non-us.debian.org "

security.debian.org and non-us.debian.org are GONE (well, burning, hopefully data can be recovered). Yes there's mirrors, but it still sucks.

Vunerability

[e8johan]

This shows the vunerability of putting all computers in one building. To have a safe network one needs to spread (duplicate) the information over several computer at several locations. How far apart these locations has to be is depending on how important you data is.

It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup. By that I don't only mean that tape that always seems to go missing when needed, but multiple sites (or at least buildings) that provide redundancy.

Re:Vunerability

[Psiren]

To have a safe network one needs to spread (duplicate) the information over several computer at several locations.

What a stunning idea. Perhaps they should call it the internet... ;-)

Re:Vunerability

[xyote]

When iSCSI becomes real you are going to see a lot more raid-1 mirroing with one half of the mirror at a remote location. Also using remote tape drives for backup. Currently if you do store your backup tapes offsite, there is still that logistical problem of getting them offsite in a timely manner. Things like if you have 10 partly full DLT tapes at $$ a tape, do you send then offsite right away or do you wait for the next set of backups to fill them up. The former can be quite expensive. Also, since most restores are from most recent backup tapes, sending them offsite right away causes delays in your restores while you wait for the tape to come back. Remote tape backup will solve a lot of these issues.

Re:Vunerability

[nbvb]

TSM (Tivoli Storage Manager) does a great job at mitigating all of those problems:

1) Sending tapes off-site too early -- TSM allows you to have multiple "storage pools" within one library. Basically, you backup your data into one storage pool (set of tapes) and then duplicate it to the off-site copies. Then the DRM (Disaster Recovery Manager) scripts automatically eject the tapes from the library for packaging to send off-site. You don't ship off your on-site copies, so when you need to do a restore, it's always here.

2) Partly full tapes -- TSM has a concept of "migrations" where it moves data between tapes to better utilize them. The internal DB automatically knows where the data is, so you don't have to worry about which tape has which days' backups -- it doesn't matter.

3) Full backupset -- TSM 4.2 introduced "portable backup sets" -- basically, you backup your data into the TSM server as normal, then "generate" a backupset -- takes all the files for a particular node and writes them to a series of tapes which you then eject and store off-site. When you need to recover from those tapes, the server component isn't needed -- locally attached tape on the server and the client piece of TSM is all you need --- speeds up recovery by a few hours!

TSM's a great product. Does lots and lots of great things that other vendors are only now trying to figure out.

Sure, it's pricy, but how much is your data worth?

Re:Vunerability

[coupland]

It is a shame that a building hosting so many good initiatives should be the one to go, but as always: there is no excuse for not have a backup.

Uhhh, yes there is... I suspect you either know nothing about IT or are fresh out of college. DRP (disaster recovery planning) factors in things such as criticality of data, cost, and acceptable downtime. A university payroll system may need to be back up within 12 hours of a major incident, so in addition to tape backups you might have a failover site. Contrary to your simplistic post, even the richest corporations rarely have failover sites of their own. They simply contract out to a DRP vendor who have these types of machines lying dormant in a glass room, waiting to cut over. On the other hand a university FTP site is probably classified as low risk, low impact. So you would rely on off-site backup tapes and perhaps only restore when you've arranged for an alternate site and taken delivery of new servers. You don't pay millions of dollars to have two glass rooms just so you can have uninterrupted FTP service...

Re:Vunerability

[pellaeon]

Try doing that on a university-wide multi-gigabit capable network on the budget of the average Dutch university. Our universities aren't like M$ in cash, you know. I know, I'm an admin at one myself.

I just hope they're well insured....poor colleagues...

On the upside: they may get a squeaky-clean start when this blows over :-)

Re:Vunerability

[operagost]

Maybe something like this.

800 kilometers should do it.

Re:Vunerability

[Anonymous Coward]

Well, I'm sure that a full risk assessment will be made after the arson investigators have completed their work.

BTW, did anyone find any burned-up butterflies in the area? THAT would explain the cause! :-)

Re:Vunerability

[Anonymous Coward]

I'm an auditor (pause for tomato dodging) and this reminded me of a couple risks I had to nail clients for.

The first kept their AS400 farm running the majority of their critical applications in a building near their chemical plant (*ding*). Normally this would only be considered a minor risk, however they were in a low-lying area of Louisiana. We were actually looking UP at the Mississippi River (well, we would have without the dykes). They never seemed to grasp why we were so appalled that we could see a WATERLINE from a previous flood about 6 feet above groundlevel!

Another had a backup site and stored their tape backups at a site that was DIRECTLY in the path of one of the busiest runways at an extremely busy airport. We actually had to pause conversations when anything jet-powered was taking off. They didn't consider that to be a risk either.

Re:Vunerability

[soramimicake]

This shows the vunerability of putting all computers in one building.

Actually, the debian.org names affected all point to a single box, satie.debian.org. Your point is still valid, though. Come to think about it, that makes it _more_ valid, as only one box crashing will take down all your services.

Re:Vunerability

[complexmath]

One notable exception being the finance industry. In NYC, most of the larger trading organizations maintain DR sites somewhere else in case of something like, oh, Sept 11. The issue here though is that these companies need a fully capable and connected trading floor rather than just a backup server room, and the short-term loss potential of the industry is such that DR costs are generally quite reasonable in comparison.

Re:Vunerability

[borft]

well, 99% of the data isn;t lost. Only the router and the uplink where in the building :( We already have a 54 Mbit connection back up again, but that is nothinh compared to 1 Gbit.... And we are behind a maquerading firewall, so, no mirrors as of yet :( BUT, we'll be back! :P

Re:Vunerability

[e8johan]

Great news! I hope that you'll recover quickly so that everything will be up and running!

A good reminder..

[Martigan80]

To never keep back-ups in the same physical location.

Re:A good reminder..

[Zocalo]

Or even general location for that matter. A friend of mine did disaster recovery work for IBM after the Trade Towers attack. They had their data center in Tower 1 and their backup center in Tower 2. After six weeks of what was essentially scrabbling through rubble they managed to recover a single spindle. The company concerned became another statistic, and part of an important lesson in DR implementation; safety increases with distance.

Re:A good reminder..

[Martin Blank]

Cantor Fitzgerald, the company hardest-hit in the attack with 700+ employees killed, had just that viewpoint. eSpeed, a subsidiary of CF, had all trading activities back up and running by the morning of Sept 13, due to the multiple locations and real-time backups being done on the network. My understanding is that the time from loss of connection with CF headquarters to full resumption of its trading activity around the world was about 46 hours.

Re:A good reminder..

[PD]

No point in keeping your backups any further away than your most remote customer. If the customers are gone, you don't need the backups.

Re:A good reminder..

[Jugalator]

To never keep back-ups in the same physical location.

That's why I use Kazaa to keep backups of my pr0n all over the world!

Re:A good reminder..

[Tom]

absolutely. A local power company I sometimes work with has the following simple policy:

It is not considered a backup until it's stored in two locations at least 10 miles apart.

Re:A good reminder..

[LWATCDR]

A secure Freenet for backups maybe?
encrypt your data and put it on freenet style system?
You could then recover from anywhere on the network.

Re:A good reminder..

[JonKatzIsAnIdiot]

Do you have any links to more info about the two companies you mentioned? Sounds like it would be an interesting read.

Re:A good reminder..

[Martin Blank]

My mistake: 47 hours.

Details on the recovery:
http://www.baselinemag.com/article2/0,3 959,36807,0 0.asp

Some information on the scale of what eSpeed handles:
http://wstonline.com/story/mag/WST200110 08S0007

Even Cantor Fitzgerald mentions it in their Q3/01 results summary:
http://www.cantor.com/articles/article11 192001.htm

Re:A good reminder..

[RAMMS EIN]

``safety increases with distance.''
If only life were that simple. Yes, keeping backups accross the country gives better protection against localized disasters than keeping them in the same room, but the increased distance also increases the risk that something happens on the way, and also increases the time required to get things up and running again.

Re:A good reminder..

[athakur999]

I personally don't keep backups. However, I'm sure at least two other me's in parallel dimensions do, so I'm probably good to go.

Hopefully the other me's in the other dimensions aren't counting on me to keep backups. Those idiots.

Re:A good reminder..

[osu-neko]

I always try to hook up with buddies in other *continents*.

If only the Atlanteans had thought of this...

Re:A good reminder..

[Menkhaf]

"Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it ;)" - Linus Torvalds (just had to post it)

Re:A good reminder..

[roalt]

To never keep back-ups in the same physical location.

The university made full backups this weekend and the building where the backups are is approximately 3km away from the Computer Centre that burned down.

In fact: this 'backup' building used to be one from the Dutch Bank (say the Federal Bank in the US) and the backups are also stored in the original (giant) safe !

Erm...

[Apiakun]

Yowch :( That looks like one hell of a fire. I certainly hope there were recent backups. I'll observe a moment of silence for all that data gone up in smoke, and I suggest a 15 ping salute.

Re:Erm...

[Stonehead]

Hmz. Didn't that get translated? There are no casualties and there are full backups (from last weekend, anyway) from all the student and research servers. For the rest, I don't know - probably not. The damage has roughly been estimated on 23 million euro for the building and 17 million euro for everything that was inside. But the building is still smoking - it's only 6.5 hours ago that the fire started.

Damn!

[devilkin]

I really hope everyone got out ok... Its always a shame to see a NOC go down, especially one of a university where communication is quite vital... DK

Re:why does this matter?

[fearlezz]

It's the fastest university network in the netherlands, and it's running a lot of mirrors (debian mirrors and others). The most important thing: terabytes of warez. :) (I believe it was the BSA lighting the fire :) )

So much for server areas never burning down

[Martin Blank]

This is something I keep hearing from people, particularly those who balk at paying for upgrades to expensive fire suppression systems. "They're always built so well that there's never been a case of a server room destroyed by fire." I always did doubt that, and now whenever they tell me that, I can point to this.

What a shame.

Re:So much for server areas never burning down

[CuriousGeorge113]

Yes, but did this server room have one of those fancy fire suppression systems? If so, yes, you can point to this an laugh. If it didn't, your friends that always say "Server rooms are never destroyed by fires because of their fire suppression systems." will have the last laugh.

I.E. "See, if they would have had a fire suppression system, this would have never happened."

Re:So much for server areas never burning down

[Plutor]

The fact that "The fire department has given up every hope on protecting the server area and is now trying to protect the surrounding buildings" leads me to believe that the fire didn't start in the server area. Lots of server rooms were destroyed on September 11, for example, but it wasn't the fault of the room's design, or the presence or lack of fire suppression systems. If the whole building is burning down, fire suppression in one room is only going to work until the floor and ceiling collapse.

Re:So much for server areas never burning down

[jhines]

A server room perhaps, they are generally kept very clean.

However, it usually sits next to a room full of cables, and the telco plant. This is flammable. Look at the pictures, very heavy black smoke, so this is very likely.

An old cable plant has lots of PVC and other cables that are not fire resistant the way modern Teflon is, and that will still burn, if it is hot enough.

Re:So much for server areas never burning down

[aaarrrgggh]

The thing that is odd (knowing absolutely nothing about the building in question) is that there typically aren't sufficient materials of combustion in a data center to create such a substantial fire! (Sparing large printers and paper rolls.)

A fuel vault should have foam or CO2 suppression, neutralizing that risk. The building should have a full sprinkler system, to minimize the risk of spread of fire. Hopefully, the computer and tape rooms have Halon/FM200 to extinguish a fire on the first signs of smoke. It is very rare that a building would be completely lost if these components are working properly.

The flip side of it is that the more things that are done to prevent an accidental sprinkler discharge (pre-action systems, double-interlocked pre-action, abort stations, etc.) increase the time for a system to respond, and consequently increase the size of the fire that has to be put out.

Re:So much for server areas never burning down

[aaarrrgggh]

The cables are not fire resistive (normally); they are low smoke. (Unless you are british, in which case they could actually be somewhat fire resistive.) Ultimately, given a high enough temperature everything burns. If a fire lasts more than 2 hours, you expect to loose the building.

After 30 minutes or so, all the cases, finishes, rubber will burn, and that will give you the black smoke, but it doesn't indicate the initial source of combustion.

Re:So much for server areas never burning down

[ReelOddeeo]

Yes, but did this server room have one of those fancy fire suppression systems?

Followup question: Was it a fire suppression system, or was it a Microsoft fire suppression system running Windows CE and Visual Basic?

Re:So much for server areas never burning down

[mmol_6453]

Here in Grand Rapids, Michigan, any cable that passes through the heater system has to be Plenum. And considering that, at Greybar, you have a choice between Plenum and riser-cable, most of the new (as in, last twenty years) telecom wiring in Grand Rapids is fire-resistant. The insulation of Plenum will liquify before it burns...

Re:So much for server areas never burning down

[bad-badtz-maru]

Plenum insulation doesn't liquify any more than regular insulation and it definitely can't be considered fire-resistant, it just gives off less toxic smoke when it burns. Plenum is not really a term that describes a type of insulation, it's a term that refers to a building space that is used as a circulation space for HVAC. This space is typically between the drop ceiling and the roof. When burning, plenum-rated cable will not induce as much smoke into the plenum as regular cable does.

maru

Re:why does this matter?

[seosamh]

From the description it sounds like this is one of the major Debian archives. Not a Debian user, or Dutch speaker, so I'm guessing from the original post.

In a related announcement...

[Zocalo]

The University of Twente's attempts to overclock the new AMD Opteron and Nvidia GeForceFX card in the same case are declared a failure. "We certainly won't be building a Beowolf cluster of these..." commented a spokesperson.

Re:In a related announcement...

[OrangeSpyderMan]

UNIX? They're probably not even circumcised! Savages!

You're thinking of that Linux distro with a BSD style ports things aren't you? What's it called again, Gentle, Gentool - Gentile Linux!!! Thats right! :-)

Re:In a related announcement...

[penfold69]

Debian users retaliate after flame war on Slashdot.org

See what happens when you don't use a bong

[DrSkwid]

those crazy Dutch and their roaches

Maybe..

[jedie]

Someone took "firewall" too seriously? :)
(and this isn't *flame*bait :p)

Re:Maybe..

[poot_rootbeer]

http://www.cluebomb.com/comics/cluebomb/cartoon3 .g if

Well..

[flatface]

Better submit this one to Bugzilla...

Priorities

[brianvan]

Was anyone killed?

If not, was anyone hurt?

If not, do they have insurance?

If they do... well, I'm sure someone just lost their masterpiece pr0n directory, but otherwise, things like this happen. (ask Hemos) You have to make it through such things. In this case, it was a commercial (educational) building and no one is homeless, so it's less of a tragedy than usual. Let's hope that they rebuild with something better and newer.

That said, I get the feeling that those plumes of smoke really are millions of dollars floating away in the wind...

Re:Priorities

[bzzzt]

No, no and no.

The university doesn't insure itself agains this kind of disaster because they are able to set aside enough money to cover the cost themselves and don't have to pay the insurer's profit.

Now how they are going to deal with this and their financial troubles (they're almost broke) is another issue...

Re:Priorities

[nukey56]

That said, I get the feeling that those plumes of smoke really are millions of dollars floating away in the wind...

Well, it was mostly a bunch of warez anyways. Warez in the wind.. could be a catch song title.

Re:Priorities

[tinus]

Actually they do, but only for large amounts of damage.

The problems they will face are the things that you can't buy insurance for, like the cost of not running the network for a few days, not having room for employees and students, things like that.

Re:Priorities

[Plutor]

"... they are able to set aside enough money to cover the cost themselves ..."
"... they're almost broke ..."

Eh?

Re:Priorities

[pa3gvr]

Let's hope that they rebuild with something better and newer.

It was already one of the fastest infrastructures of Europe. Maby now it will become one of the fastest of the world.

Sjaak

Re:Priorities

[dazed-n-confused]

If they do... well, I'm sure someone just lost their masterpiece pr0n directory, but otherwise, things like this happen. (ask Hemos)

Remind me: when did Hemos lose his masterpiece pr0n directory?

Re:Priorities

[pellaeon]

That's what I thought as well :-)

But basically, if they're _anything_ like the other Dutch universities, it's the latter. We're doing a pretty thourough job of dismantling our knowledge infrastructure.

Re:Priorities

[brianvan]

It's a shame I remember these things. I really need to get out more.

Re:Priorities

[MonoSynth]

They can't (and won't) pay the profits of the insurance-companies. Having your own disaster-budget is less expensive if you have the facilities to manage it.

Re:Priorities

[MonoSynth]

...like cutting 358 million Euro's in the next four years, while there is already a shortage of 700 million for the maintenance of the buildings only :(

Re:Priorities

[NoOneInParticular]

Phrasing of the grandparent was a bit wrong, it should be:

Traditionally, they were able...

Now they're broke...

And yes, the Dutch government has decided that the Dutch educational system can be top of the line with spending less money on education (in percentage points of the GNP) than Uganda.

The truly sad thing is that the statistic above was from last year. Now they're cutting another 5 percent. Ah well, nothing bad, my university is coping by not renewing any temporary contracts. So that will get rid of all postdocs funded by the universities directly.

Now I'm going to continue writing that grant proposal.

Re:Priorities

[ReelOddeeo]

Let's hope that they rebuild with something better and newer.

There is a problem with this. Longhorn is not out yet. We know it will at least be newer than XP. As for better, well, Microsoft has promised that it will be.

last syslog messages from SURFnet routers

[ahu]

lo0.ar5.enschede1.surf.net 3613: Nov 20 07:20:50.927 UTC: %ENV_MON-2-TEMP: Hotpoint temp sensor(slot 18) temperature has reached WARNING level at 61(C)

few seconds later on the local side:
lo0.cr2.amsterdam2.surf.net 1146: Nov 20 07:20:56.458 UTC: %CLNS-5-ADJCHANGE: ISIS: Adjacency to ar5.enschede1 (POS2/0) Down, interface deleted(non-iih)

Re:last syslog messages from SURFnet routers

[Jugalator]

I wonder if they got the Linux "easter egg" error message "Printer on fire"? I guess it would be appropriate for once.

Re:last syslog messages from SURFnet routers

[Anonymous Coward]

ping lo0.ar5.enschede1.surf.net
PING lo0.ar5.enschede1.surf.net (145.145.255.14): 56 data bytes
--- lo0.ar5.enschede1.surf.net ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss :-(((

Re:last syslog messages from SURFnet routers

[CoolVibe]

Those are Cisco syslog messages. No linux easter eggs.

Re:last syslog messages from SURFnet routers

[Jugalator]

Those are Cisco syslog messages. No linux easter eggs.

Of course, but surely they must have had *some* Linux box there that noticed strange things going on in the printer room. :-)

Re:last syslog messages from SURFnet routers

[CoolVibe]

Maybe one with a LM chipset in there perhaps? Who knows. :) A fire in the room would surely increase the mobo temperature. Allright, who's got MRTG or RRDtool graphs?

Re:last syslog messages from SURFnet routers

[Anonymous Coward]

temperature has reached WARNING level at 61(C)

FYI- 61 degrees C = 141 degrees F.

Sprinkler systems kick in around 150 to 160 degrees; temperature at the top of the room was probably higher.

My bet is that router died(along with everything else) when the sprinkler system came on and power was cut to the room(most fire suppression systems in datacenters, water or gas, cut power to the entire room.)

It dated back to the earliest systems- "big red switches" usually have what are called "Molly guards", which are supposedly named for the daughter of a Univ. of Calif. programmer whose daughter hit the button 3 times in one day.

The same jargon dictionary mentions that in the 'early dates', said BRS fired a shell into the mains bar of the mainframe to sever it.

That said, 60 degrees is about the max operating temperature for most electronics. So take your pick :-|

Re:last syslog messages from SURFnet routers

[Coke in a Can]

Man. I dunno why, but I'm crying at that. It's like the server's last cry for help. And then the connection is lost. :-(

Re:last syslog messages from SURFnet routers

[jelle]

Network traffic rrdtool graph.

Hope they ...

[zensonic]

... got backups. Only fools ditch the backup procedure!

If the actually have backups it's only a matter of money/time before sites will be up'n'running again.

Re:Hope they ...

[devilkin]

Well, let's also hope the backups are stored OFF-SITE! Otherwise they'd also be burned to crisps... DK

Re:Hope they ...

[Martin Blank]

I imagine they were well-insured. On the greed side of things, some major server company is going to report a small spike in Q4/02 server sales in Europe.

"Hi, we need a quote for a few hundred new servers and a couple dozen new routers and switches. Oh, and we need them *now*. Hello? Hello...? Dammit! Another sales rep fainted!"

Re:Hope they ...

[mezzin]

What I heard they were off-site

Re:Hope they ...

[ntp]

Of course. Daily backups to /dev/null. It's much faster that way.

Another useful tip from the BOfH.

Re:Hope they ...

[LinuxHam]

nah, we can get 'em up with a coupla z800's sometime next week. A few thousand virtual servers in a single 19" rack. They'll get tons of floor space back and maybe this time they'll consider DR to another z800 in another building on campus.

Re:Hope they ...

[HuibertAlblas]

Remember All says: "Save time, save often!" (tm)

From Leisure Suit Larry I, II and III

But that's no problem...

[jmo_jon]

But that's no problem, all i need to do is apt_get...

Does this mean that there will be no more apt_get interesting +5 posts in a while?

Halon dumps?

[wiredog]

The last time I worked in a NOC, it ran Vaxes, but we had a halon dump. A Big Red Button that got smacked by the last person out of the room. The halon would smother any fire by replacing all the oxygen in the room (which was why the last guy out hit the button). Why wasn't there a halon dump in this NOC? Or, if there was one, what happened?

I hope Debian practices good management principles by having offsite backup.

Re:Halon dumps?

[Entrope]

The blurbs imply ("The fire department has given up every hope on protecting the server area") that the fire originated elsewhere and spread -- fire supression systems are good for fires that start in server rooms, but if somebody puts tinfoil in the microwave next door, the fire may be just too big by the time it gets to the server room.

Re:Halon dumps?

[Martin Blank]

Depending on whether there was a recent overhaul of the system, Halon is now illegal in most countries. Existing systems were grandfathered in, but new systems are prohibited from using it. I'm not sure which gas replaced it, but I understand that it's incredibly expensive, and not quite as effective. Of course, as has been pointed out, if the fire originated (and thus was fed from) elsewhere, the gas would have been useless once the storage tanks were empty.

Re:Halon dumps?

[lamj]

Halon DO NOT replace oxygen in the room to extinguish the fire. It breaks the chain reaction of fire, basically stop the elements of fire to react with each other.

Most scenario would only require a less than 8% of concentration to take out the fire. Under 10% and you can still breath.

Problem of Halon is when over 900 degree C, it breaks down into hydrogen fluoride, hydrogen bromide and bromine - stuff that are toxic. So, run!

Re:Halon dumps?

[Get Behind the Mule]

Halon DO NOT replace oxygen in the room to extinguish the fire. It breaks the chain reaction of fire, basically stop the elements of fire to react with each other.

Most scenario would only require a less than 8% of concentration to take out the fire. Under 10% and you can still breath.

Really? Well, I worked in a server room with a Halon system back in the 80's, and the big scary warning signs said that you wouldn't be able to breathe within n seconds after the stuff fell. So always be prepared to dash for the door.

It was pretty ominous, looking at those two big ol' red buttons on opposite walls with the all-caps, white-on-red warnings next to them, imagining that you could just go wham!, and maybe kill a few people, if they couldn't think & run fast enough. And imagining that in case of a catastrophe, you just might have to do it, and then scream at everybody to get the hell out, wondering whether you saved some lives or ended a few.

Re:Halon dumps?

[VirtualAdept]

Wow. It sounds like all that place needed to be like Star Trek was a large, heavy emergency door that closes just slow enough that the last person can roll out of the room just in time for the door to slam shut.

Re:Halon dumps?

[kriston]

In the US it's being replaced with FM-200 gas and some places are using a new concept called "dry pipe" where the sprinkler systems aren't pressurized so you have time to turn things off before escaping (assuming that if the fire goes on long enough for the water to reach the computer room before turning off, the room is lost).

Or something.

Kris

Re:Halon dumps?

[nolife]

Halon DO NOT replace oxygen in the room to extinguish the fire.

It cools, replaces O2 and works to break the chemical chain. At least according to this

There is a fire triangle. The points of the triangle represent heat, oxygen, and a combustable material. If you take any one of the three away a fire will go out. Halon and other devices (Baking Soda, AFFF, PKP, etcc) replaces or prevents introduction of O2 causing the fire to go out. Use of water removes the heat, CO2 replaces the O2 and has a slight cooling effect. Halon also breaks the chemical reaction. A problem with O2 removing substances is if the temperature has not gone down, reintoduction of O2 will cause a reflash (like opening the door to early).

http://www.nifc.gov/pres_visit/whatisfire.html
http://www.usbr.gov/power/data/fist/fist5~2/5~2_1. htm
http://www.princeton.edu/~ehs/theater/Appendi xA.ht ml

Re:Halon dumps?

[mrjive]

Don't forget the heroic grabbing of the hat that fell off on the other side of the door, Indiana Jones style.

Re:Halon dumps?

[mmol_6453]

Dry-pipe sprinklers aren't new...I've got a ten-year old Firefighter's training book at home that describes their effectiveness and how to turn them off after the fire is knocked down.

Re:Halon dumps?

[mmol_6453]

Considering that the pyrolysis reaction for each compound is different, I find it's rather difficult to know how to break down each reaction without displacing oxygen--the common component in nearly all fires.

(I say "nearly" because, once it's started, acetylene doesn't require oxygen to continue burning. They get away with using it for welding because the gas escapes the cylinder faster than the fire can creap up the stream. At least, that's how it was explained to me.)

Re:Halon dumps?

[oh]

Again, this is from memory, but isn't acetylene the gas commonly used in whats commonly called and oxy torch? Every acetylene torch I've looked at had two cylenters, and two hoses. One was a black cylender with a white top for the oxygen, and a slightly bigger, red-brown cylender for the acetylene.

see this

Re:Halon dumps?

[mmol_6453]

It requires oxygen to ignite, but does not require it to continue burning. See the firefighting section of the MSDS ("Right To Know") sheet. Look at the entry under "Flammability Limits in Air."

It can only be 2.2% of the gas in a given volume of normal atmosphere, to ignite, but, once started, it can continue to burn even after it is the only volatile gas left.

I predict...

[bLanark]

That in the next year, most new OS documenation and projects will have several URLs built in for all online stuff (gentoo's Portage; TLDP; having a sourceforge page and a mirror, for example).

OS people learn quickly from mistakes like this.

Re:I predict...

[gmack]

I predict they will use their backups and repoint the URL to another location.

It's not as if the debian project didn't have the domain properly setup with 3 diffrent nameservers in 3 completely different locations.

They may even have a working mirror...

Re:I predict...

[Wolfrider]

--Yep, this explains the problems I was having yesterday with apt-get update / upgrade. However, today things are working fine, so they must have brought 'security' back up somewhere else and either propogated the DNS or kept the same address.

--I'm really getting to love Debian. This from a longtime SuSE 7.3 user (even bought the DVD).

Re:why does this matter?

[brianvan]

They're not gone. They're just experiencing a Distributed Combustion Denial-of-Service attack.

Debian Security mirror

[lemmen]

In case one might be interested, Essent mirrored security.debian.org.

You can use debian.essentkabel.com to download the latest security updates (in case you haven't already). Please note this is NOT an official mirror.

Re:Debian Security mirror

[Tottori]

I don't want to besmirch Essent or anyone else, but please don't install security-critical software from a mirror some guy posted on Slashdot. apt-get doesn't check signatures, so if you put a compromised or malicious mirror in your sources.list, it's game over.

Debian haven't released a security advisory since yesterday, so it's deeply unlikely you'll need an update before they get a new security.debian.org online tomorrow.

OK, folks, admit it!

[matija]

Who posted a link to UT's webserver on slashdot?

Re:OK, folks, admit it!

[McFly69]

Sheee... don't tell anyone.. it was me. *hides under bed*

Re:OK, folks, admit it!

[erpbridge]

And why would you do something like that? Think, McFly, think!

Re:OK, folks, admit it!

[FFNieko]

Right, /.ing a 9.6Gbps backbone link? ;)

More Pictures from the webcam

[CuriousGeorge113]

Here's some statics taken from their webcam. Someone might want to mirror these before they get slash'd. Webcam PicsPics

Also, here's what seems to be the only close up I could find of the fire. pics

last syslog entry

[Anonymous Coward]

the last syslog entry from AR5.Enschede1.surf.net to SARA:
Nov 20 08:20:51 lo0.ar5.enschede1.surf.net 3613: Nov 20 07:20:50.927 UTC: %ENV_MON-2-TEMP: Hotpoint temp sensor(slot 18) temperature has reached WARNING level at 61(C) :-)

Halon is probably illegal today

[mangu]

Effective as it was, Halon was also a major ozone-hole cause, so it has been outlawed in most countries by now.

Re:Halon is probably illegal today

[Porsche_Pilot]

I beleve that they have stopped production of halon, but their is still stock, just like R12 that used to be used in car A/C. Many people I know have halon extinguishers in their cars, an when I get another air-cooled car it will have a installed halon system .

Re:Halon is probably illegal today

[phil reed]

Wouldn't have helped. A halon system in the server room doesn't do much when the entire building goes up.

Oh Dear....

[Zech Harvey]

Is there anything that could cause this naturally? I mean, judging from the pictures it looked fairly large and out of control...I hope no one caused this purposefully. I've never had to deal with a catastrophe like this, luckily...I send my best wishes of luck and hope to those involved and pray no one got hurt. =(

Re:Oh Dear....

[BabyDave]

I hope no one caused this purposefully.

Cue another round of "Bill Gates/MS evil anti-Linux conspiracy" posts ...

Re:Oh Dear....

[CH-BuG]

> Is there anything that could cause this naturally?

A match?

Re:Oh Dear....

[SagSaw]

At my university the computer center is housed in the oldest building on campus. This building also contains a foundry, welding shop, E.E. labs, etc. While I have no idea what the circumstances in this instance were, I imagine many acedemic institutions have similar set-ups. It's not overly difficult so see how a fire could start and quickly spread in such a building.

Dangerous server rooms

[Bazman]

Lets hope the place didnt look like this beforehand:

Dangerous server rooms

Slashdot effect?

[lyonsden]

Is that what happens to a web server when an article housed on it gets posted on Slashdot?

Re:why does this matter?

[smnolde]

Time for them to switch to FreeBSD. With mirrors all over the world I don't have to worry about where to go for the source.

However, any fire destroying the building is a *bad* thing and if it's arson I hope they pin his nuts to the wall. I hope the data in the NOC is restored quickly.

The truth is right here

[nukey56]

Now, you see, this didn't happen through some random electrical spark or from some warez-eliminating conspiracy, the truth is that this building was just fine -- until it was slashdotted. Those pictures, well, those were posted later on. I mean, seriously, the link probably said "mad warez now!" for the first few minutes, then they changed it and put the fire story up.

Moral of this story: never underestimate the power of the slashdot effect.

Re:The truth is right here

[CoolVibe]

The roof!
The root!
The roof is on fire!
We don't need no water, let the warezkiddies burn!
burn! burn!

Allright, that was cheap... *ducks*

Update from SARA (SURFnet NOC)

[mdav]

Here's an update from SARA (that's where I work), the network operator for SURFnet. SURFnet is very busy ordering new equipment and fixing the 2 x 10 Gbit/s lamda's to Enschede. We hope to restore connectivity a.s.a.p. Greetings, Marco

Re:why does this matter?

[The Dobber]

Boy Scouts Of America? Which merit badge would that be?

It must have been Microsoft

[micaiah]

"Witnesses saw a large balding man monkey dancing from the scene and a slim geeky man with glasses trailing behind continuously adjusting his glasses." An in other news....

Re:It must have been Microsoft

[$rtbl_this]

Am I the only one who just pictured the Gentlemen and their henchmen from the Buffy episode Hush? **Shudder** I'll never take the piss out of Steve Ballmer again.

Re:It must have been Microsoft

[plugger]

A *slim* geeky man? It couldn't have been Gates then.

Re:It must have been Microsoft

[Kashif Shaikh]

Yeah, microsoft is that desparate to take out Debian. Or...

It may be a university that didn't want M$ 'donation' so that they could have more windoze courses.

Either way, I think the monkey is trying to do a rain dance!

LFS mirror affected too

[decarelbitter]

The Dutch LFS FTP mirror was also hosted at the University of Twente, which means it's also down. The Dutch HTTP mirror should work properly, since it's outside utwente.nl space.
Last news is that HP (Who supplied most of the UT backbone equipment) is on its way with emergency equipment to have things up and running somewhere tomorrow.

At least the fire...

[l0wland]

...was good for something. My DSL d/l-speed doubled after the Cisco's at UT got smoked. :-)

Warning Sounds

[Anonymous Coward]

If you ever smell smoke in a computer enviroment, take it serious! I once was in a computer pool room and only the smell of burning paper made me aware of the fire. Only then i realized a beeping noise, simlar to the noises some computers or cardreaders make to draw maintainace attention towards them.
In that environment an old fashion siren would have been a much better solution.

Oh, well...

[larien]

Time to find out if their disaster recovery procedures work... There'll be a heck of a lot of running around trying to get some kind of infrastructure back in place now, I'd imagine.

Re:Lazy commie bastard

[zztzed]

I have yet to see any machine translators that will do Dutch to English.

Re:Halon dumps? -- not if the whole building is in

[earthy]

The fire did not start in the server rooms. What happened was that a fire started in one of the lecture rooms (and a smallish one at that, number A108) that just happened to be in the same wing of the TWRC building that also housed the server rooms (yes, multiple). It then proceeded to take out two entire wings and threaten other buildings nearby as well as the library.

Now, I would *love* to see a halon system capable of stopping that...

Owh, and the fire seems to be under control by now, as evidenced on http://webcam.traserv.com/
(which you can contrast to http://www-infstud.sci.kun.nl/~arthurvl/ispy.jpg (taken at about 09:40 CET this morning)).

thank god for mirrors

[dspeyer]

Let this be a lesson to all system archetechts: don't put stuff in only one place -- it could catch fire. With all the debian mirrors out there, I don't expect this to cause all that much trouble (obviously, if people had been hurt, that would be another matter, but apparently everyone got out in time).

PGP Keyserver root was hosted by SURFNet

[Erik_]

I believe, the open and distributed network of Keyserver.net (distibuted network of PGP keyservers) was hosted by SURFNet. This network is a distributed network holding PGP and OpenPGP keys. The loss the to UT NOC could have an impact on the updating of key-rings across the keyserver.net network.

Re:PGP Keyserver root was hosted by SURFNet

[arjennienhuis]

Yes, but was it at UTwente? SURFNet works just fine without Twente.

Re:PGP Keyserver root was hosted by SURFNet

[dwaallicht]

The keyserver hosted by SURFnet is called 'horowitz.surfnet.nl' and is physically located in Tilburg, several 100 kms from Enschede. It is of course possible there was an important piece of keyserver.net infrastructure hosted at utwente.nl that I am not aware of, what I do know is that that part would not have been hosted by SURFnet. Jan

Re:why does this matter?

[Angry White Guy]

The arson and destruction of private property badges. If a car burnt as well, they're well on their way to achieving the elusive 'inciting riots' and 'urban terrorist' badges. These last two were proudly held by a select few: Ted Kazinsky, Che Guevera, and Jimmy Carter

The registers take on things...

[ContemporaryInsanity]

El Reg have an article up (http://www.theregister.co.uk/content/6/28204.html ) paying particular note to the warez / pr0n side of things.

Re:The registers take on things...

[SpaceJunkie]

Hmmm- Maybe the MPAA and RIAA could be implied here.... The register artical pointed out how much of a major P2P hub this segment was.

Re:why does this matter?

[mezzin]

I agre with this with security patches you need to have them as fast as you can. Shame the servers are gone :-( hopefully they find a good new spot for them.

security updates mirrored worldwide.

[novakreo]

Actually, Debian security updates are usually also added to the proposed-updates section, which means that they are available on nearly all Debian mirrors worldwide.

Something like
deb ftp://ftp.XX.debian.org/pub/debian/ proposed-updates main contrib non-free
deb ftp://ftp.XX.debian.org/pub/debian-non-US/ proposed-updates/non-US main contrib non-free
(replace XX with your ISO country code) in your /etc/apt/sources.list should work well.

You can find mirrors on Debian's website.

OK

[wiredog]

It started elsewhere. What about other fire supression systems? Sprinklers and the like?

Re:OK

[Hektor_Troy]

Well, not knowing what's in room 108A, I can think of several types of fires, where sprinklersystems won't help.

Magnisium catching fire.
Na (which I can't remember the English name for) stock in the room would also spell disaster.

Re:OK

[Tet]

Na (which I can't remember the English name for)

Sodium.

Re:OK

[CoolVibe]

Na == Sodium

Glad to help :)

Re:OK

[bartjan]

The building in question is the TW/RC building (Technische Wiskunde / Rekencentrum = Technical Mathematics / Datacentre) which is a 3 floor high building consisting of 4 wings (A, B, C and D) and partially built on water. A Google Image search on 'tw utwente' will show some pictures from the outside of the building.
The building is almost 30 years old and the two destroyed wings have just been renovated a year or so ago. Because of this renovation the 2 wings are believed to pass current fire code (the other wings don't).
The hallways are a bit narrow and are laid out in what can be described as a maze, so I'm not surprised that a large enough fire managed to spread through the entire building.
I'm almost sure A108 is one in a series of 'normal' classrooms with only chairs and tables and a blackboard, so I'm a bit puzzled why the fire would have started there.

Osama

[tha_european]

I wonder if CNN's gonna broadcast something like "CIA has found links suggesting, that al-Qaida is responsible for this act of terrorism." ;-)

Re:why does this matter?

[mezzin]

For security patches. and the non-us section. And a lot of other mirror are down now ;-((((

Re:Lazy commie bastard

[novakreo]

worldlingo.com has one.

Re:The importance of professional hosting...

[decarelbitter]

When a fire starts outside of the systemsroom you can have all the Halon gas you want, but it won't help. And if it would, the water used by the firedepartment would drown all machines. This is not just a datacenter which went up in flames, it's two wings of a whole building, which also happened to host the utwente.nl datacenter.

BACKUP!!!

[Beliskner]

Now we can find out how secure and hardened Debian really is. You are as good as your latest backup.
BACKUPS BACKUPS BACKUPS Off-site! I've had enough of people who are talking about RAID-5 because 5TB tape drive arrays are too slow. Always keep your BACKUPS!

Re:BACKUP!!!

[noahm]

Umm, what does "secure and hardened" have to do with backups? From my experience, secure and hardened is often counter to backups. Every additional copy of the data you have is an additional security concern.

That said, people are currently working on restoring the machine now, and the rough ETA is given as tomorrow. So backups are not an issue.

noah

Re:BACKUP!!!

[Beliskner]

Umm, what does "secure and hardened" have to do with backups?

Secure is meant in the "safe and secure" sense, not the "securely encrypted" sense. Hardened is used in the sense that the Internet is hardened, having been designed to survive a nuclear war.

Dual language is also used in various religous texts - e.g. "convince the non-believers" can mean talk to them, or it can mean kill them all.

Translation of the Twentsche Courant Tubantia

[TeeWee]

[Apologies for the poor translation, no time for a better attempt]

The burning building contains the IT department and a part of the faculty of Business Administration [Closest I could come up with: Bestuurskunde]. The building has three storeys.

The university fears the loss of its network facilities and is trying to save the main computer. According to a spokesman this network is amongst the fastest in Europe. Most classes are expected to resume as normal today. [According to other sources, this is an exam week, meaning few classes anyway. This is also a reason that few students were around so chances were that this also reduced potential casualties]

The fire department is fighting the fire with 25 firefighters and expects to need the entire day to extinguish the fire. No dangerous materials have been released by the fire at this point. At the moment nothing is known about the cause of this fire.

Re:Translation of the Twentsche Courant Tubantia

[CoolVibe]

No dangerous materials have been released by the fire at this point.

Oh? I heard lots of asbestos was released into the air because of this fire.

Re:Translation of the Twentsche Courant Tubantia

[TheLink]

Someone tried to cancel an exam and succeeded?

Re:The importance of professional hosting...

[nochops]

Hmmm...
It looks like your REAL datacenter's web server just went down in flames, and the link's only in a comment, not the main story!

Halon systems aren't illegal, but....

[mks113]

You can't install new halon systems, but existing ones are still legal (in Canada, anyway).

However, I wouldn't want to have to fill out all the paperwork involved with a discharge! We had an accidental discharge (a leak, I believe) and they decided it was enough impetus to remove the system.

I think they are using CO2 now. The advantage of Halon is that you can breathe quite comfortably in an atmosphere that will not sustain fire. CO2 works just about as effectively but will not sustain life.

OTOH, more recent studies have shown that just because you can still breath in a Halon infiltrated environment doesn't mean that there are no health effects!

I expect there are more CO2 systems going in now, with lots of alarms to make sure people get out before the atmosphere gets unlivable.

Michael

Re:Halon systems aren't illegal, but....

[srpatterson]

We've got a couple of server rooms at work with big (5ft tall) CO2 extinguishers, you have 10 seconds to leave the server room after they activate.

Strangely enough, there are no servers in there, just hubs, routers and comms gear.

Re:Halon systems aren't illegal, but....

[914]

Halon will kill you just as dead, just as quickly, as CO2.

Both gasses extinguish fire by excluding oxygen... which isn't good for people either.

The difference is that one cannot comfortably breath CO2, an involuntary physiological reaction makes it impossible. (next time you open a bottle of Coke, try sniffing the little cloud of CO2 that forms in the top of the bottle)

Halon, otoh, is perfectly comofortable to breathe in and out, but will provide no oxygen. This is similar to breathing helium (recommended as the nicest way to commit suicide, after N20) in that one would be quite comfortable right up until one passed out from hypoxia. Death follows soon after.

Also, i've read in boating magazines that undersized Halon systems used in engine compartments can be dangerous. Apparently, if the diesel engine is running when the system fires and there isn't enough Halon to kill the engine, the burned Halon/air/diesel mixture produces some really nasty toxic gasses.

Anyhow.. enough rambling!

Re:Halon systems aren't illegal, but....

[ComaVN]

Because, obviously, there will never be a false alarm, will there...

Re:Halon systems aren't illegal, but....

[Dahan]

Halon, otoh, is perfectly comofortable to breathe in and out, but will provide no oxygen.

Why does it need to provide oxygen? The air provides oxygen. How come you're not worried that air is about 80% nitrogen, which also provides no oxygen? See, humans don't need to breathe 100% oxygen... we do fine with much less. The advantage of Halon over CO2 is that it does not extinguish a fire by displacing oxygen. It will put out a fire at concentrations of about 5%, leaving plenty of oxygen to breathe.

the burned Halon/air/diesel mixture produces some really nasty toxic gasses.

And a fire doesn't? If you have an undersized system installed, you're gonna have problems in a fire anyways. With a proper system, the small quantity of toxic gasses produced by the Halon decomposition before the fire is extinguished (which is a fraction of a second--Halon systems have been used for explosion suppression) is much preferable to the large quantity of toxic gasses and heat produced by a fire.

Re:Halon systems aren't illegal, but....

[haligan]

One reason why Halon works i sthat it displaces the air to remove that side of the fire tetrahedron. You cannot breathe in an atmosphere filled with Haoln, this is why there are big red buttons that will delay the discharge of the Halon. That way you have more time to escape the area that is about to be filled with Halon. And you cannot buy nor recharge a Halon system in the U.S. anymore. A sprinkler system should have been in place instead. It might not have been required by the building code, but there has never been a multi-fatality fire in a building with a properly installed and maintained sprinkler system and statistics have shown there is a greater chance of recover from a fire as well. The DOE has placed many sprinkler systems in their Server rooms.

Re:Halon systems aren't illegal, but....

[914]

It will put out a fire [tpub.com] at concentrations of about 5%, leaving plenty of oxygen to breathe.

I stand corrected! I was under the impression that it works by displacing the ambient air (and it's oxygen).

As for the engine-room thing, I think the point was that an engine, being essentially a large positive-displacement air pump, could actually pump the halon out of the engine room, creatng toxic gasses as it did so, and also lowering the overall amount of halon in the room to the point of rendering it ineffective.

There is some controversy (according to this article in PassageMaker) as to whether these halon systems should be installed with an automatic engine-kill or not. On one hand, it would suck to have your engine suck all of the halon out of the engine room.... otoh, it would suck worse if you needed that engine to maneuver during the emergency (to avoid collision etc) and it had been killed by the fire system.

Anyhow, you make good points and I'm glad I know better now.

Re:Halon systems aren't illegal, but....

[SillySlashdotName]

I have never understood the thinking behind changing the 'fire triangle' to the 'fire tetrahedron.' A triangle has three points - coresponding to fuel, oxygen, and heat - while a tetrahedron has four points - with the fire tetrahedron points being fuel, oxygen, heat, and "chemical reaction". BUT "The chemical chain reaction know[sic] as fire occurs when fuel, oxygen and heat are present in the right conditions and amounts." (The Fire Tetrahedron) If you replace "The chemical chain reaction know[sic] as fire" with "Fire", the tetrahedron reverts back to a triangle with "fire occurs when fuel, oxygen and heat are present in the right conditions and amounts."

In other words, if you have fuel, oxygen and heat in correct amounts, AND YOU HAVE FIRE PRESENT (fourth point of the tetrahedron), you will have fire.

Well duh!

About as good as it gets with only two sites...

[Kjella]

I mean seriously, each tower collapsed because it was hit by its own plane. If one tower had been in NY, one in California and both were still hit by a plane each, the result would be exactly the same.

The lesson should be: Primary back-up is a very good start, but secondary/tertiary back-up is the thing if it's that critical.

Kjella

Re:About as good as it gets with only two sites...

[Anonymous Coward]

The chances of two buildings standing side by side being taken down by planes is ridiculously low.

It apparently can happen, though.

In any case, the chances of two buildings standing side by side being taken down is much greater than two random buildings on either coast.

Keep in mind - there's enough nuclear weaponry out there to make the United States into a glass desert. What if that goes off? I hope you had secondary+ backups in Canada and Mexico!

(Frankly, I wouldn't be concerned with backups at that point.)

The point is that backup is simply another game of risk.

Re:About as good as it gets with only two sites...

[operagost]

Considering how several other buildings in the WTC were damaged or destroyed just from debris, I'd say that even if only one tower had fallen, there's a good chance everything in the second would have been trashed.

Really, not having your backups in close proximity to the data center IS good policy.

Re:About as good as it gets with only two sites...

[Fencepost]

No, you want two geographically diverse sites in case of natural disasters or other events that keep you from reaching (physically or electronically) one.

I'm sure that there's been lots of study of this, but I suspect that a good distance is 1-2 hours drive. Far enough to avoid most of the impact from things like a big chunk of the city being shut down for weeks, close enough to get to (with some inconvenience) if necessary.

In the Chicago area it might be something like downtown Chicago and Schaumburg, Naperville, Aurora or even Rockford (at the 2-hour mark).

Re:About as good as it gets with only two sites...

[0x69]

Bullcrap. BOTH towers had been exposed as very vulnerable to ONE bomb in the underground parking garage long before. BOTH towers were connected to the same small pieces of the electrical, telecom, water, sewer, gas, etc. systems, and Chicago demonstrated the dangers of that a few year ago.

The only [gag] good [retch] reasons to have the backup in the other tower are spelled "lazy" and "stupid".

Re:About as good as it gets with only two sites...

[gaudior]

I wish some of the downtown outfits would put some data centers out here in Rockford. I need a job.

Re:About as good as it gets with only two sites...

[madsen]

Spreading them out is definately a good thing to do but you also have to think about in what kind of a building you put the NOC. There is a worldwide banking network that has its NOCs on three continents and in towns that are not in any way on anyones radar (well I guess they are because of these NOCs). The buildings are built as bombshelters and with faraday cages aso. They also have at least "double everything" in all NOCs.
This is ofcourse extreme but if you want no single point of failure and have the need to do it right, this is the way to do it.

WARNING: Above post a GOATSE!

[Cletus the yokel]

I just lost my breakfast. Thanks ever so much.
If you want to read about the fire, go here. Apparently UT was a major node for KazAA, and'a primo source for warez and pr0n.

Re:WARNING: Above post a GOATSE!

[ealar dlanvuli]

Apparently UT was a major node for KazAA, and'a primo source for warez and pr0n.

And the firefighters will determine it was lit by a professionaly arsonist, but never be able to figure out what orginization funded him..

Re:WARNING: Above post a GOATSE!

[ReelOddeeo]

>>Apparently UT was a major node for KazAA...

And the firefighters will determine it was lit by a professionaly arsonist, but never be able to figure out what orginization funded him..

....but never be able to figure out which of the *AA's funded him.

Re:WARNING: Above post a GOATSE!

[Cletus the yokel]

>(Score:0, Flamebait) ...
> Apparently UT was a major node for KazAA, and'a primo source for warez and pr0n.

Flamebait?! I didn't say it, The Register did!

Bah... moderators on crack again.

Re:WARNING: Above post a GOATSE!

[Yottabyte84]

set goatse.cx to 127.0.0.1 in your hosts file. I realized it befor the rediredt hit, but my hosts file would have redirected it to my local http server.

Re:No Halon?

[lamj]

Depends, if the fire starts inside the NOC then there's a high chance that Halon would extinguish the fire but if the fire started elsewhere then spread to the NOC, most likely the fire suppression systems are not designed to handle that.

Moreover, Halon system are no longer installed (globally) since 1987 (Montreal Protocol) due to its CFC damaging effect. Most systems already installed are replaced by FM-200. Water, Argon, FE-13, Inergen and a few others are all possible replacement.

Re:English, you insensitive clod!

[Jugalator]

Hey how about a link in English for those who are unfortunately unilingual?!?

There are some news at The Register.

The importance of only posting crap after READING

[mdb31]

Yeah, yeah, whatever. In case you were not paying attention: the entire *building* (which happened to contain most core routers etc. - there are servers in other buildings as well, but they kinda lost connectivity) burnt down. This building not only contained the IS department (which, I'm sure, has active fire protection in its most data centers), but also normal offices, classrooms, etc. Once an entire structure is on fire and collapsing in flames, there's only so much your fire suppression system can do. In this case, it will most likely have evaporated... Fire suppression systems only help if a fire is *starting* in a protected area, not if the entire structure the system is in is collapsing in flames

Re:why does this matter?

[cjwatson]

Yes, security.debian.org, non-us.debian.org, qa.debian.org, and nm.debian.org were hosted on our machine there. Work on restoring all of them to another machine is already in progress. Packages aren't too much of a problem; the infrastructure running there will take a little longer to rebuild.

Fortunately the majority of the critical infrastructure was kept in CVS.

Disaster Recovery plans

[ACK!!]

This brings to the forefront of my mind the fact my organization is currently devising disaster recovery plans for our new building.

The NIH Computer Center Disaster Recovery Plan is available online somewhere in pdf format and provided a good deal of insite.

Does anyone else have good hints on texts and outline for good disaster recovery plans?

___________________________________________

Re:Disaster Recovery plans

[wiredog]

The Gartner Group put some stuff up after 9/11. Most of it is common sense.

Do full backups weekly, store copies offsite. Incremental backups daily, copies offsite also. If you can afford it (or can't afford any downtime), have emergency backup hardware (enough for minimal operations) in an offsite storage facility. Old hardware that would otherwise be thrown out is good for this (remember, it's for an emergency). Have a supplier who can get replacement hardware to you in a hurry (so you can get off of those old 90 MHz Pentium servers).

The most vital part of the plan, after backups, is good insurance. If the building burns to the ground Monday morning, you want to be able to call the insurer Monday Noon, and have the check in hand Tuesday morning at the latest.

These recommendations do not cover disasters such as 767s flying into the building and killing all the sysops. Earthquakes dropping the building on the same. Etc. The people are the most important part of any company and, if too many of them are lost at once, the company probably is lost too.

Unless you have really good (and expensive)insurance which can provide enough funds for you to hire new people, get them trained, and keep the company solvent while you do so.

Re:Disaster Recovery plans

[guacamolefoo]

The most vital part of the plan, after backups, is good insurance. If the building burns to the ground Monday morning, you want to be able to call the insurer Monday Noon, and have the check in hand Tuesday morning at the latest.

Despite all the horseshit about "State Farm is there" and "Nationwide is on your side", you will not have a check in hand the next day. When you make a claim, you are a cost center to the insurer, not a policy premium paying buddy.

Expect to have your ass turned inside out a la goatse.cx before seeing so much as a nickel from an insurer. An "expensive" policy doesn't mean that they will be any nicer about it -- it just means you overpaid for the privilege of being dejected, suspected, inspected, and rejected by a greedy little monkey-fucking insurance company.

Re:Disaster Recovery plans

[ReverendRyan]

These recommendations do not cover disasters such as 767s flying into the building and killing all the sysops. Earthquakes dropping the building on the same. Etc. The people are the most important part of any company and, if too many of them are lost at once, the company probably is lost too.

Perhaps if the disaster recovery plan included dummy instructions for getting basic services back online. A manual that was as easy to follow as "..insert the tape labeled 'usrbin20021119-6' and type 'tar -xzfv /dev/st0'. Now change directories to /home and insert the tape labeled 'home20021119-3' and type 'tar -xzfv /dev/st0'..."

With a system like that, the entire IT department could be killed and as long as someone knew where the disaster plan was, the site could be recovered.

Re:Disaster Recovery plans

[noz]

"...good insurance. If the building burns to the ground Monday morning, you want to be able to call the insurer Monday Noon, and have the check in hand Tuesday morning at the latest."

This is good advice. Sites need insurance, but demanding payment in under 24 hours is not reasonable. If insurance for fires is unconditional, then within one to five days is reasonable, but then again, many people will have conditions in their insurance contract that may either be void in certain circumstances, or simply require investigation under others.

Re:How is "Twente" pronounced??

[devilkin]

As you write it - Twente. Take 'Twenty' and replace the 'y' with the sound of the 'e' in the same word. DK

Yeah

[wiredog]

It might have been a chemistry lab with willy pete (white phosphorus), sodium (Na), or something like that. Or a gas main could've blown.

In Maryland a couple days ago several businesses in the same building burned to the ground after a medical supply company caught fire. The company supplied oxygen tanks.

The world's most warped error message...

[meringuoid]

... finally gets to make an appearance in earnest.

'lp1 on fire'

I wondered what was going on

[BenjyD]

There I am wondering why my Debian machine's update script won't work, I bring up slashdot and find this story.

Re:Lazy commie bastard

[spiny]

there's this:

http://www.worldlingo.com/products_services/worl dl ingo_translator.html ....

I swear...

[flaez]

...I was updating my debian & was checking
slashdot out of boredom because there seemed
to be a network problem.

[screenshot]
http://flaez.ch/scratch/twente_bur ns.png

Ah HAH!!!

[wiredog]

Some damn fool wrote an assembly program that used the dreaded HCF instruction, didn't they?

Re:Ah HAH!!!

[Ziviyr]

The 64 bit HCF must be pretty powerful.

(though I really am a bit sad to hear such well purposed hardware combust)

Oops.

[Effugas]

Maybe I shouldn't have released that code after all...

Yours Truly,

Dan Kaminsky
DoxPara Research
http://www.doxpara.com

Comment removed

[account_deleted]

Comment removed based on user account deletion

More info...

[citizenkeller]

...straight from the horse's mouth here.
There are some pictures, official news and infos.

Debian is working on recovering

[StormCrow]

Quote from debian-devel-announce

At around 8 this morning (local time) a fire started in the computing facilities of Twente University. This affects Debian, since one of our servers (satie) is hosted there. At this moment it seems very likely that the machine can not be recovered from the fire.

The following services are currently down as a result of this:

security.debian.org
non-us.debian.org
nm.debian.org
qa.debian.org

We are working to restoring these services on another machine and hope to have things in mostly working order by tomorrow. Security advisories are still available at http://www.debian.org/security/

Wichert.

it was the printers!!!

[manual_overide]

lp0 on fire!!!

(sorry, couldn't help myself...)

Breathe Quite Comfortably?

[Guido69]

The advantage of Halon is that you can breathe quite comfortably in an atmosphere that will not sustain fire. CO2 works just about as effectively but will not sustain life.

You've obviously never been in a Halon dump. Either that or you consider burning in your lungs to be "quite comfortable". Not to mention that if you're standing under one of the discharge nozzles at a dump you can get a nasty case of frostbite.

I've personally been through two 1211 dumps and had to enter a computer room and drag staff out after an FM200 dump. It takes about two days to completely stop coughing.

Re:Breathe Quite Comfortably?

[Wolfrider]

--You're an idiot. He said he was dragging STAFF out - that means PEOPLE. Read the friggen text carefully before you post like a numbnuts.

--Dumbass.

See what happens....

[Cap'n Canuck]

See what happens when you get a beowulf cluster of those things? Ka-boom!

conspiracy

[antonsthlm]

it's obvious that this fire was really caused by the RIAA/MPAA terrorists trying to target 'piracy terrorism'. It was only a matter of time before this new pre-emptive strike policy targetted the Really Free World: Western European Universities (The american ones don't count - they are just brands).

there are dead people, they won't tell you though.

welcome to the real world.

Not funny...

[distributed.karma]

Looks like somebody forgot to patch the firewall. *ducks*

Insert "don't overclock in your noc" joke here

[otis wildflower]

.. and they could have halt -n'd with a legitimate reason..

Still from what I can see with my nonexistent Dutch nobody was hurt and backups were fairly solid, which is about all you can ask for at a time like this.. Still, you know what this means.. NEW TOYS!!!!

Cause? Injuries?

[nurb432]

Just curious if everyone is ok, and if they have found a cause yet..

*insert M$ terrorist joke here*

Students!

[flippet]

The Register has a story about it. I like the paragraph at the end...

Twente's high-speed network was originally constructed to provide students with access to high-speed Internet access for their studies. It soon became one of the major hubs for peer-to-peer exchange programs like KaZaA. This perceived misuse of Internet resources caused former Dutch education minister Loek Hermans to comment: "It would be nice if the students at Twente University would use their fast connections for information and education purposes, instead of downloading huge amounts of porn."

Phil, just me

Grrr

[Zathruss]

You know, fuckit, one day someone, somewhere, may come up with something really cool like cold fusion, zero point energy, whatever... and no one will know about it.. Why is that? Because it'll be bloody slashdot'ed right out of existence within five minutes of being posted. goddamnit.

Re:Grrr

[Zathruss]

Hmm, I posted this to the "fan wing" story. Damn gremlins....

Re:Grrr

[Zathruss]

..."it would probably get laughed out of slashdot anyway by the majority of the readers.

I know, and it has many times before. I was basically venting my frustration at the fact that everytime something nifty comes along, I have to wait hours till I can read about it. Its not like the blurb tells us anything useful, (mostly).

Final job

[bullgod]

Request failed: printer on fire.

Terrorism

[SamMichaels]

I'm surprised noone has screamed "OH NO TERRORISM" yet.

Seems like everytime something happens we have to blame the evil terrorists first...then find out what actually happened.

Re:Terrorism

[osu-neko]

Piss of the USA? No problem. But piss off hackers? The terrorists know better...

[More likely, technoculture remains below their radar. They'll go after Hollywood before Silicon Valley...]

Re:Terrorism

[Koos]

I'm surprised noone has screamed "OH NO TERRORISM" yet.

It did not happen in the US ;-)

Won't someone PLEASE think of the servers...

[Nonillion]

At least no one lost their life in this fire. Well, unless you are a server. Hopefully this will turn out to be an accident and not arson or botched terrorist act. It always bums me out when inocent servers lose their lives ;)

Re:English, you insensitive clod!

[CoolVibe]

Learn dutch then.

The slashdot posting says it all really, the Utwente NOC burned down. I myself am dutch, so I grokked the links quite fine :)

Many friends of mine study at utwente. I was already wondering why they weren't on IRC, but since I heard the news I knew why.

Oh, AFAIK, utwente kept off-site backups of all data, so all is probably not lost.

Re:moderators on crack

[Zathruss]

Ug... will someone please shoot me now? The "grandparent" is indeed a goatse.cx link. And I need some sleep. Sheesh, I never thought I'd have to apologise to an AC...

burn

[pr0nbot]

Is this what happens when you're slashdotted by God?

Twente University targeted in piracy raids 1 year

[Megasphaera Elsdenii]

Might this fire have something to do with this, one cannot escape wondering?

Al Gore already took that idea...

[DanEsparza]

What a stunning idea. Perhaps they should call it the internet

Nah. Al Gore already took that name. Or was it the 'Information superhighway'?

:-)

Sad

[ledow]

Maybe it was a flame war between students? Were they overclocking? The dangers of using FireWire. Were their harddrives Quantum Fireballs? (They are now) Is this what you get when you try to hot-swap them?

Other sad jokes will no doubt follow.

Re:Sad

[ledow]

Were the students playing on Heat.net too much? Did they do too many warm reboots?

translation..

[mikevdg]

This is a translation from the telegraaf (which, btw, isn't the best newspaper in holland, but anyway..)

"University network down from fire"

"ENSCHEDE - A fire in the computing center from the university of Twente (UT) in Enschede has caused a few 10's of millions of euros damage on Wednesday.

The fire, which broke out about 8:00am, has disabled the universities network. There were no injuries. The fire department has cleared out several buildings in the nearby facinity. The IT departments of the UT and a part of the business studies department were housed in the building where the fire broke out. The building was three stories high.

The fire department fought the fire with several dozen people and estimated that they would need the whole day to get the fire under control. There are, until now, no dangerous substances released. The cause of the outbreak is not yet known.

According to Van Vught, a backup of the network's data has been made. "All data have been safely stored". Because of exams, there were no classes at the time. There were also few students on the campus on wednesday. Exams on wednesday have been cancelled.

Staff at the UT want to set up a temporary network, which can take several days. Thousands of staff and students can not make use of the campus network until a temporary network has been installed.

One on-looking business studies student is shocked (?): "incredibly sad. It means that I can't do my assignments. Doubtless, a temporary building will need to be set-up to help the situation". The Saxion Polytechn in Enschede has made room available.

The web-site from the Enschede city council is not available because of the fire. The council used the servers from the UT. The Saxion Polytech is also without internet access for the same reason."

Okay, this translation is a wee-bit too literal, but it's understandable.

Michael.

The cause of the fire?

[Lethyos]

So wait, the University of Twente NOC caught fire. Why? Was there a story posted on Slashdot's front page that linked to a server at their location? Or was this fire caused by something other than a hardy slashdotting?

Re:The cause of the fire?

[SeanAhern]

Merchant of Venice, Act 2, Scene 7, Line 66?

Re:The cause of the fire?

[poot_rootbeer]

They should have just let Milton have his stapler...

Re:The cause of the fire?

[pmind]

Is this the guy in that famous movie "Office space" ? :)

I like the way they destroy that printer hehe...

Could this be...?

[fruity1983]

Microsoft said they were looking for new strategies to counter the rise of linux, but who woulda thought they'd go this far?

Slashdot Coincidence?

[Destoo]

Seeing this happen just a few hours after the story about Which desktop distro will die first? is quite scary.

</sarcasm>

Donations?

[Akardam]

I understand from a post a ways down that the University is currently having financial difficulties. Does anyone know if there's been something setup, whether for the University in general or Debian in specific to accept donations? I know (even though I don't use Debian) that I would contribute a few dollars to help them pay for all the new equipment, and I suspect that there's more'n a few geeks out there who feel the same way.

Re:Donations?

[jpetts]

I understand from a post a ways down that the University is currently having financial difficulties.

Did anybody see the Dean beforehand sneaking round with a can of petrol?

clicky clicky (was Re:The registers take on thing)

[CoolVibe]

Did you know you could use a subset of HTML on slashdot? Check it out:

click

Next time use the allmighty <a> tag.

Utwente - The proof that admins shouldn't smoke

[fluor2]

Utwente - The proof that admins shouldn't smoke

security.debian.org backup online

[m0i]

Apparently, klecker.debian.org (194.109.137.218) is now hosting security.debian.org (as seen in NS updates propagating).

Re:Halon dumps? -- not if the whole building is in

[Salamander]

I would *love* to see a halon system capable of stopping that...

I wouldn't. Any halon system capable of doing that would literally be a killer in its own right.

You mean HiveCache (formerly MojoNation)?

[Fencepost]

That's exactly what the Mojo Nation folks are doing now. Info at http://www.mojonation.net/.

Conspiracy Theory!!!

[fantomas]

Come on guys, where are the conspiracy theories???

UT hosted HAL2001 last year, surely it was the FBI / CIA / KGB / SMERSH / Mysterons / [enter your bad guys of choice...] getting back at those pesky linux hackers?

Uh-oh

[NBrooke271]

Did we slashdot these guys latley?

fire supression systems?

[eclectric]

sure, they kill everyone in the room, but they're also pretty good at putting out fires. What are they using... water?

Re:fire supression systems?

[kylegordon]

What has a higher chance of succeeding? An entire burning building, or a room full of Halon? I know which one I would put my money on.

You name it

[complexmath]

For one thing, the NOCs and the switching centers of the major switched carriers aren't always even in the same building. The only people in the switching centers are people that need to be there and switching centers of major carriers all even occasionally share a building (maybe bad from a single-point-of-failure standpoint, but it means that they all get to exploit a single well-designed and well-placed building for a slightly more reasonable cost). The hardware rooms are all temperature controlled, are on replenishable power backup systems (to account for theoretically unlimited time off-grid), and protected from fire by gas systems (like Halon). They also generally don't have anything flammable anywhere (how did this fire get started anyway?). You've got to remember, in the US anyway, phone companies are considered utilities and governed by special rules as a result. If a city were being firebombed and police had ordered that civilians not be allowed on the streets, switch operators would be an exception to that rule. It's highly unlikely any of our switching centers would fail from much besides a natural disaster (floods have interrupted service before) or a deliberate attack.

If a man asks you for his stapler back...

[HaloZero]

...here's an example of why you should give it to him.

Gee whillickers. That sucks. I've heard great things about their infrastructure, mostly about HAL and how it went.

I can't read Dutch, or whatever the hell those articles were posted in, but, does anyone know how exactly the fire started? As we speak, I'm getting dressed with one hand and typing this with the other, on my way over to my own Uni's ops room. Just have to make sure my babies are ok.

Twente Hard NOCs?

[digitac]

Oh come on, where's all of the jokes about the "School of Hard NOCs"?

"The School of Twente Hard NOCs offers course on Fire Safety"

42

Re:Well Damn... Nobody Hurt!

[The J Kid]

..wel, there was nobody hurt or killed.

There has been a lot of asbestos freed, but otherwise it's ok. Everything is backuped and HP will have a small part of the network up by tomorrow.
To get the whole network up will take 3 weeks, insides say.

The main point is, is that the warez servers are not affected (I'm NOT kidding), that's because they are mostly located in the dormitories of the campus and not in the computer center.

security, qa & non-us.debian.org ARE down...but there are mirrors, mentioned elsewhere in this topic.

OpenSource Lightning?

[Diesel Dave]

How about this one:

Write free software
????
Insure Everything
Light a Match
PROFIT!

So, when does the pledge drive start?

[bADlOGIN]

Since Debian doesn't have a 40 Billion dollar monopoly warchest to draw from, I'm assuming that some funds will need to be raised to get new boxes. This is a perfect time for uses to step up and make a contribution back.

More Photo's: (Re:Photo's )

[The J Kid]

can be found at the official backup utwente site:

http://srv1ut.utwente.virtu.nl/nieuws/items/brand/

This isn't homeowner's insurance

[wiredog]

Business insurance is more costly, but has better service. If the word gets out that a certain company isn't able/willing to pay claims on policies with high quarterly premiums expeditiously, then all the companies that used to buy from that agency will go elsewhere. Fast.

Re:This isn't homeowner's insurance

[bad-badtz-maru]

Have you ever filed a claim against a business insurance policy? I was involved in the filing of a $1.1M USD claim involving hurricane damage and I assure you, the check is not there the next day... or the day after... or the day after... even though our annual premium, for more than seven years, had been more than four times the amount of the claim. And this was a major insurance carrier.

maru

OhNO! More.. or Less.. lemmings..

[ixxologic]

OMG.. Arnold.c64.org wasnt hosted at UTWENTE was it?

Cause of fire?

[Kakarat]

Perhaps they didn't replace a recalled battery?

On second thought...

[rholland356]

Maybe it wasn't such a good idea to replace the consoles with laptops...

http://www.computerproblems.com/answer.cfm?Answe rI D=14867&QuestionID=14929&CatID=40

"Dude, your datacenter got Delled!"

Re:Perens-kabob

[osu-neko]

Thanks for the reminder of how depressingly low a human being can sink...

Class IV construction

[mmol_6453]

I don't doubt it. That's a Class IV construction; I.e. mostly concrete and steel. It's really hard for fire to spread in buildings like that; You'd have to have notably flammable materials in any direction the spreads.

An excellent example of Class IV construction would be any recently build NFL stadium. I haven't been in one often, but I was struck by how far apart any flammable material (food carts, etc.) is. Even the hallway ceiling I saw was actually the bottom of the concrete steps above.

Re:Class IV construction

[haligan]

As a former fire fighter and fire inspector I must comment that yes the construction may have been type IV , but like most people , Architects included, you are forgetting that the contents are mostly flammable, (ie carpet, paint, some ceiling tiles). Mike B.

Re:Class IV construction

[mmol_6453]

As a former fire fighter and fire inspector

Explains your username. :)

...forgetting that the contents are mostly flammable, (ie carpet, paint, some ceiling tiles).

I thought about that, but I presumed that the materials would be somewhat spaced out; distance between couches for comfort, room between rows of computers (in labs) for safety. Things like that.

All in all, though, good point.

They Needed Low-Tech Fire Protection

[0x69]

Our ISP bought an old legal office building for their HQ and colo facility. The place was built with file rooms to safeguard tons of irreplaceable paper documents - imagine thick concrete walls & ceilings, with heavy steel fire doors, rated to preserve the contents through an EVERYTHING-else-burned-to-the-ground fire.

Critical stuff is spread between the file rooms, with metal conduit, etc. protecting the few small holes they added for wiring.

Steel & reinforced concrete aren't quite obsolete.

more information

[Ruliz Galaxor]

I live about 5km away and at 11:00 CET I could still see a lot of brown/grey smoke in the air.

frontpage.fok.nl (a dutch news site) reports the following, well at least not the article itself but in a reaction to it: "The D-wing was recently renovated to make it more 'fireproof', so there are no harmful gasses and such.
Every morning a backup is made from all the data. Unfortunetely, before this backup could be taken, it was already destroyed, which means data from yesterday morning is lost.

Tonight (CET of course) there will probably be an emergency network running at 10kb/s and the server will probably be online in 3 days. Until then Teletop (some kind of information system for communication between tutors and students) and the utwente email will be down. Something about the building: The TWRC building some kind of monument which included that the outside of the building cannot be changed. This means the renovation plans have to be the same as the orignal building."

At least I hope they got rid of the air-problem, because it really stinked in that building anyways. Too bad everything to lost. They were just about to move all the servers to another location.
I was just printing my report which I had to hand in at 12:00 CET when I heard it. I worked the last 24 hours to complete the damn thing and then this. :(
Lucky for me I don't have internet from the university, but it still sucks bigtime.

firewall

[Ruliz Galaxor]

So much about a firewall...

sig(h)

Re:+5 OT: Karma

[Stonehead]

Your reply is indeed off-topic (just like this one) and may even be a troll. The text is inconsistent and even flamebait - why would schools not teach knowledge? But I find this one of the most beautiful -1-rated comments I have seen in time.
Thanks.

this guy probably did it...

[h4x0r-3l337]

Jos Boersema, usenet cook

(yeah, I know physics department != NOC, but this guy isn't very bright)

Insurance money... not much

[erpbridge]

Back the end of Sept 2001 we had a lightning strike on campus where I work. It twisted around a tree, then went into the ground.

The UPS's worked great, and none of the power was affected at all. Howerver, the problem was when the lightning hit the ground, it traveled in all directions... eventually hitting the buildings themselves, and found the bundles of CAT-5 running below the floors, in the walls.

Being a small campus, we only lost 14 workstation network cards, 4 printer network cards, and 1 workstation. However, one of those buildings was where our main fiber / copper switch was for the campus. We lost two blades on that, as well as a few lesser switches.

The moral being: Insurance company only gave us about 40% of what everything was worth. In something like the fire we're all talking about, they may give some more, but nowhere near enough to bring the campus back up to where it was. There is going to be some significant cost to this university.

Re:Insurance money... not much

[rosewood]

Someone was under insured then or not properly insured

You can be covered so that everything ruined from that would be COVERED

but hey, that costs MONEY

No! MPAA/RIAA

[bsd-mon]

It soon became one of the major hubs for peer-to-peer exchange programs like KaZaA

DMCA enforcement in full effect

Hey! Those are in order!

[mmol_6453]

Interesting that almost all the pics there are in time-sequence...You could make an animation out of it.

I don't know how, but I encourage other people to try.

Two sites don't cost much more than one

[billstewart]

Instead of putting all the computers in one building, put half of them in two buildings. The main cost difference is some extra real-estate preparation. And if you've got a multi-gigabit network, it's going to multiple buildings, so spread out your routers. Huge centralized computers were really useful back when mainframes were the size of dozens of refrigerators, but if you're using modern machines, they're either rack-mounted or pizza-boxes, and or else they're PC-shaped. But you've got multiples of them and can spread them around.

Re:Two sites don't cost much more than one

[pellaeon]

Yeah, you could do that, but like a countryman of mine (apparently) remarked, there's a shortage of about 700 million euros in real estate (both maintenance and new) in the higher education echelons.

Spread most of that among 14 universities and you can imagine the space shortages...
On top of that: server rooms aren't exactly as cheap as offices.

And if you've got a really expensive piece of equipment that you simply can't split up, what do you do?

Re:Two sites don't cost much more than one

[billstewart]

No, but while you probably need more total air conditioning if you split between two buildings, you need only about half as much space per building as you did before, because you're putting about half the stuff in each. So instead of one building with N square meters of server rooms and another building with N square meters of cheap low-quality graduate student desk space (:-), you need two buildings, each one with N/2 square meters of computers and N/2 square meters of desks. Obviously some pieces of equipment can't be split, so you have to decide how badly you need two of them for reliability - but for almost everything in the computer business, prices are much lower than in past years, comparable to expensive physics or chemistry equipment (which is more likely to catch fire.)

While it's too bad that the equipment was lost, and much worse that data was lost, one advantage of replacing old computers with new ones is that the same amount of computer capacity will cost you less money with new equipment, and will use less floor space, electricity, and air conditioning, and will cost less to maintain than the old equipment. Depending on your campus wiring infrastructure, it may be possible to take advantage of some other new architectural developments, like peer-to-peer storage. It's not just for pirate music - the cost of PC data storage is almost down to $1000 per terabyte, and some of the applications can be reliable enough to use it effectively.

Re:Two sites don't cost much more than one

[pellaeon]

But if you do it that way and one of the server areas gets devastated you lose half your capacity. You still lose out from an operational point of view.

It all comes down to: what do you do when you _don't_ have the money to duplicate everything? Sure, I'd like to have everything quadrupled and in multiple buildings, but with university budgets that's not going to happen any time soon (i.e. 100 years =)

I agree completely with your second paragraph, btw.

The cause is clear

[Maskirovka]

I guess the their boss fired the bofh.

more bad puns

[Linuxthess]

The HotLine Server crashed and burned, whilst the CD-R Burner was spittin out toasty-coasters. Thats what you get for playing with Alcohol 120% and Nero. Damn pyros.

----------

Enschede ??

[fred666]

Enschede ?? Remember: about 4 years ago, a fireworks factory ended up in flames. throwing out *huge blocks of concrete* in the sky. Many houses in the neightborhood were gone in smoke.

Enschede is a really hot place. :-)

Do NOT stay at ground level with HALON!

[billstewart]

Halon is heavier than air, so if you're somewhere there's been a Halon dump, DO NOT HIT THE GROUND! Walk/run as upright as you comfortably can. The usual "stay low" advice is because hot air and smoke are lighter than regular air, so the regular air stays near the floor.

Halon (or at least most of the Halons used for fire suppression; not sure about all of them) is non-toxic, though it'll get you a bit high, like nitrous oxide or most solvents, so being stuck in the stuff won't injure you quickly (except from flying objects that were blown around by the gas pressure.) However, it's no substitute for oxygen, and you'll probably be wanting some oxygen real soon now. If you can remember not to breathe in the stuff, try not to breathe deeply, because there's more oxygen left in your lungs than the stuff you'll be breathing in, and unfortunately, while your body can generally figure out not to breathe in water or hot smoke, it's not as good at realizing that near-room-temperature inert gases aren't very useful. Mostly, don't worry about it - find a safe door to run for and run for it, and do some deep breathing once you're outside, and try to close the door behind you.

OT: dutch (was Re:English, you insensitive clod!)

[CoolVibe]

So many people in Holland know English that they won't even speak Dutch to you unless you're completely fluent, or a local.

Favourite prank I tend to play on foreigners:

Foreigner: (trying really hard) "Hoe gaat het?"
Me: "What did you just say about my mother?"
F: "Uh, uh, sorry... I didn't mean to.."
M: "Gotcha! Just kidding!"

Also, teaching foreigners lude words in dutch is lots of fun.

As for the "grokking" "understanding" and "enlightenment", well I tend to use all of those those loosely. It's just my whacky sense of humor I guess...

Uplink back

[MiDS]

Thanks to Progress and Essent we have an uplink again.

Re:Uplink back

[Venix]

To explain this post a bit. Thanks to the great guys of SNT, CIV and the ones named above, the uplink is back here. This is realized by setting setting up 4 sattelite dishes, and now there is a uplink through these dishes. It still isn't as fast as it used to be, but at least we're back online and much faster than was foreseen this morning.

Back online... kinda

[smeagol415]

Hi, this is a message from an Utwente student living on the university campus. We're back online.

A couple of guys set up a connection from one of the student buildings to the BB building which has a new uplink, using transmitter dishes.

It will take quite a while before things get back to normal. But at least we can check email and surf the web now.

Re:The really important thing...

[lhdentra]

Or if you've already got it... warez.slashdot.org.

Things are being restored slowly

[wichert]

Lots of people are helping to restore the lost network and computing facilities. People from XS4ALL drove from one side of the country to bring spare junipers and other equipment. The campus is already back online thanks to Virtu. Mail for student is being stored at a new backup MX server courtesy of Terena and zedz.net

Debian is restoring the lost services on klecker. At this moment qa.debian.org is up and running and the non-US and security archives are available as well, although their backend systems have not been restored yet.

Valuable lessons have been learned though: it is very useful to have machines on standby where you can switches services to when needed. Having backups of important data is also really useful (and we could have done a bit better at that. UTwente apparently has good off-site backups of its own data though). And having good insurance is also definitely useful.

Backup strategies vary widely

[billstewart]

Some companies run their own backup data centers, some use backup / disaster recovery vendors, some don't. It varies a lot by customer and industry. Another strategy that's been emerging rapidly is the use of Internet colo centers as backup sites (primarily the rackspace-oriented colo rather than managed hosting colo.) Engineering companies and companies that formed from mergers of similarly-sized companies often have two data centers anyway, so they'll use one to back up the other.

The big value of many of the disaster recover vendors, beside consulting skills, was traditionally that they owned mainframes, and assuming not too many people had disasters at the same time, they could mount your data on their mainframe without you having to spend a few million dollars to buy a spare mainframe - so they were basically selling part-time use of the mainframes. Transaction-based online services companies could sometimes live with this kind of environment, but not usually - if it takes a day to rebuild your database from backup tapes and journalling, that's probably ok for a manufacturing company, and maybe ok for a traditional bank, but it's a total disaster for an airline ticket sales system or an online stockbroker; even a couple of hours may be too expensive. So those companies often have to run their own backup sites, or at the very least will have a dedicated backup mainframe at the backup vendor.

One of the interesting technologies that's starting to affect this business is metropolitan-distance fiber optics for Storage Area Networks. Somebody else mentioned iSCSI, but for the mainframe world, Fibre Channel lets you connect processors and disk farms at distances up to about 30km, and for real computers (:-) Gigabit Ethernet on fiber also has a long range (and eventually 10gigE will be mature.) This is especially useful in some niche markets - Wall Street trading and banking firms can put disk farms in New Jersey, partly for physical diversity and partly for real estate cost reasons. They're also good for other downtown/suburban connections. Sometimes these are managed by the customer, but the internet data center business and local fiber providers are going hog-wild trying to sell this. (I work for AT&T, which is selling this.)

security.debian.org down

[slifox]

And in unrelated news, a huge hole was uncovered in Debian systems today, allowing anyone, from anywhere, to gain root access simply by connecting to port 80 on the machine.

Bad Jokes?

[duncf]

If I hear another bad joke about "flame wars" or "firewalls" in reference to this, I think I might puke. :-)

Re:Nice Moderation

[scott_evil]

... I hope nobody was hurt.

After that, I wish them luck getting back online.

Someone explain how the FUCK that can be considered insightful?

Yet again we see the need for moderators to pass an IQ test.

Re:+5 OT: Karma

[DancingSword]

Knowledge, in the sense intended, means "wisdom", not facts.

There is difference.

The substance of realization is immeasureable-compassion, and the means of realization is wisdom. Facts are just objects that attach one's mind to matter and ignosises.

karma and /. "karma" aren't identical: karma is cause-and-effect law, in mind rather than Newtonian mindless matter.

Eastern understanding is, to our culture's prejudice, beneath dirt, but is oft eleganter than everything we've come up with, and oft enough declares quantum stuff ( see Alexandra David-Neel and Lama Yongden's "Secret Oral Teachings in Tibetan Buddhist Sects" CityLights.com where the discreteness of "quantum" reality is outright declared, but ignored by us... ).

Damned if I remember the code

[CaptainZapp]

DECs Zurich office, which - aside that it was inbetween nothing and nowhere - was a great place to work.

What stunned me at that time was that they had sprinkler in a fairly modern building. I understood better when I got a hang of that companies general attitude towards its environment and its employees at that time.

Re:The world's most warped error message... gone

[rastos1]

That one may be gone by this time:
patch-2.4.20.log:

Alan Cox :

• replace end user confusing "on fire" joke with real info

Last Post!

[alpg]

No man is an Iland, intire of it selfe; every man is a peece of the
Continent, a part of the maine; if a Clod bee washed away by the Sea,
Europe is the lesse, as well as if a Promontorie were, as well as if
a Mannor of thy friends or of thine owne were; any mans death diminishes
me, because I am involved in Mankinde; And therefore never send to know
for whom the bell tolls; It tolls for thee.
-- John Donne, "No Man is an Iland"

- this post brought to you by the Automated Last Post Generator...