Slashdot Mirror
Network Blackout
An anonymous reader writes "Renesys put together a special report on the effects of the recent blackout on routing and network reachability on the Internet. It includes a cool animation of networks dropping off the internet (presumably as a result of the power outage). It is interesting to see how localized some of the outage was--networks in New York state right up to the Vermont border go dark while everything on the other side of the border is quiet. New York City obviously gets clobbered."
That little red dot at the tip of LI is my home LAN going down...
Everyone could have switched over to ConMars power. Being a Russian installation, they would have just strung about 17million Home Depot Heavy Duty extension cords all plugged together from here to Mars (the outdoor version of course) to access it.
Is that a red circle on the webserver hosting that gif?
Banaaaana!
I'm sorry. I had apparently text'd most of my friends saying electricity was poor, and almost immediately word spread around the north east..
When modding "Informative", please make sure it both has a source and IS actually informative.
Don't these backbone routers have backup? I was in an ISP server house in the UK which had a full backup system. In the case of a power failure, it had a UPS that kicked in for 10 seconds while the generator was booting up, which then provided power for the infrastructure of the building. I would find it hard to believe that in the USA they don't have similar systems?
As the link says - http://www.socio.demon.co.uk/mphil/appendix1.html [Quote] The Internet is a web of several thousand computer networks that now extends to just about every region of the world and has 50 to 100 million users. The Internet of today has it's origins in a networking project called ARPAnet which was run by the Advanced Research Projects Agency, a science research body set up in 1957 by the Pentagon. (Hafner & Lyon, 1996, 19) The popular belief is that the military created the ARPAnet, the precursor of today's Internet, so that data held on Pentagon computers could survive a nuclear attack by the Soviet Union. Upon attack, data from computers at the Pentagon and other military installations could be uploaded (sent electronically) to other remote computers not affected by such an attack. [/quote] It's always been my understanding that the Internet would continue working after a Nuclear war, at least that was the plan. If this blackout had effected all of US / Canada like a Nuclear attack would, would any of the Net worked ?
Apparently not around my neck of the woods... I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.
Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.
I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.
Seriously, you've got to get out more.
http://use.perl.org
oh yeah, the cell network here was down for a good while after the lights went out. well not down, just full. i thought they learned on sept 11, that there wasnt enough capacity on the cell networks. but you know, i could be wrong.
I want 2D games back.
Woo hoo! Notice how all the outages are south of the border? We are so good! ;-)
Would love to see an animation of a webserver being slashdotted.
14th 4:20pm - Power goes out, our building's generators for some reason dont kick in.
14th 4:35 - Most of us decide to call it a day and go home
14th 4:30 - I'm in my car, I realize the blackout is bad when only 2 or 3 radio stations are working and the no traffic lights are. I know it was going to be a fun drive home - thank god i live 15 mins away.
14th 4:45 - I hear the blackout expands to parts of the States. me: OH FUCK@#$@#$
14th 4:50 - My sister sms's me on my cell telling me her and my mom are stuck in the subway - they need help. Like I care, I have my own problems , traffic is a mess and there are hundreds of psychos out.
14th 5:00 - I get home, look for a battery powered radio and listen in.
14th 5:30 - I get a call from my sister - they're stuck somewhere downtown. I just wish them luck.
14th 7:00 - I realize there are no candles in the freaking house, time to look for those puppies.
14th 8:00 - Sister & mom arrive home. me: LOL
14th 11:30 - I go to bed & pray the blackout lasts until the middle of the next day, that way I get an extended weekend -wohooo, back to bed
15th 7:30am - Wake up, lights are still out - no work, home free! wohoo, back to bed.
15th 11:30 - Receive a call from my boss, asking me where the fuck i was since they got power at the office but there is a lil "issue".
15th 12:30m - Got to the office, problem: No Internet connection, seems one of the ISP's switches went bye bye after the black out. Our main app server is down. No power you think? Nope The colo company hasnt been paying their bills and WorldCom used the blackout to pull the plug on them. Server is up and running but no outside world connection. FUCK@#*$.
15th 1:00pm - We think , no biggy let's use one of our other servers and restore apps and data from backup. HA! yah right - Turns out a DNS servers for our backup machines had died and the backup script had stopped working 10 days earlier. Great.
15th 2:30 - We wait to see if Worldcom is nice enough to plug the box back in.
15th 4:30 - Yah, it ain't happening - 20+ clients are without website and apps.
15th 5:00pm - Boss and I drive downtown to the WorldCom building to download data physically off the Box.
15th 5:30 - Stop for gas - HA! huge lineup.
15th 7:30 - Get into the server room, ha! the fucking cage where our box is is locked and the key is not working. One of us climbs the cage and goes into it, runs an ethernet cable from the box to the laptop. So picture these, 4 geeks inside a server room, three sitting on the floor , one inside a cage like some wild animal. I should have brought a camera. Let the tar'in begin.
15th 9:00 - Download is completed, our asses are sore from sitting on concrete, necks hurting, and WorldCom employee happy that we're finally leaving.
15th 9:30 - We're downtown wondering how the fuck we're going to upload 2gig+ worth of data and source code to our spare server.
15th 10:30 - Since there is no inet at the office and our home's cable is too fucking slow (Rogers cable sucks!) , we decide we bring out the ghetto in us. We walk up and down Yonge street asking Internet cafe's if they could lend us some bandwith!!!. Yes, you hear me right, we were begging for bandwith in internet cafes.
15th 10:45 - We decide we're hungry, so we stop at a sushi bar. After we're done we realized it might not be a good idea to eat fish after a blackout. Fridges not working aand all. Too fuckin late.
15th 11:00 - Found an internet cafe that will let us connect the laptop to upload.
15th 11:30 - Realize we can't do shit since the computer is in Korean , have all Win settings locked and the guy taking care of the place has no clue.
16th 1:30am - I'm at my boss's house uploading 2gig+ data , will take about 27 hours. Ask me if I cared about the clients at this point.
16th 1:35 - I leave and head to my friend's place where they're having beers & bbq'in on my friend's balcony. - My weekend Begins.
Network Blackouts? Yah they suck.
[alk]
Apparently not around my neck of the woods... I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.
Yeah, same up here in Ottawa, Canada... I was awakened early on Friday morning to the sound of my servers POSTing; my power was back in under 12 hours. I was lucky. :) (Made sure to double-check that hdparm was set to spin down the drives, that and killing the A/C were my contributions to energy efficiency.)
Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours.I don't have a UPS (well, I do, I got one free, but it's broken and I haven't had time to troubleshoot it - anyone got schematics for an APC Back-UPS Pro 280?), so your mileage may vary. If the UPS runs off 12V batteries, you might be able to:
Note that I don't know how the UPS's inverter will handle running at rated load for longer than the internal battery is capable, nor do I expect that the UPS will have much noise suppression on the battery leads - after all, batteries themselves are pretty much noise-free electrical sources and alternators are not.
Fire and Meat. Yummy.
You've got to be kidding....
No.
I hope that it was someone elses negligence & not yours!Yes. In fact, I did learn something: there's a very good reason that I hate KVM switches.
Solution (probably falling on deaf ears, but I shall try anyway):
- Keyboard and monitor on all servers. Monitors should be LCD to save UPS batteries. This way, if the UPS shutdown sequence fails, it can be done manually. (A distressing number of these machines run Redmond's cancer, so having one central "shutdown console" is less practical.)
- Serial shutdown outputs from UPS will be "broadcast" to all hosts connected to that UPS. Will require making a custom cable with a few MAX232 line-driver ICs and hang it off a machine's PS/2 port for power.
- Backup generator is rated for 90kW and showed only about 10kW load running emergency lights around the building. Time to tap out that extra 80kW of capacity. Why this wasn't checked before is an absolute mystery to me.
- Reorganize server interconnections. The network switch for the users' LAN wasn't on UPS (but neither are the users, so it didn't seem like a big deal). However, over the years, some stuff has come to rely on mapped drives... fortunately in this case, we're not so lucky to have hard-mounted NFS anywhere.
:)
- Have occasional practice power outages before long weekends. Who doesn't test their UPS by unplugging it from time to time? [grin]
I guess I'll stop complaining about the 50 l-users who ignored my emails on Jul.28 - Aug.1 warning them of the impending RPC vulnerability worm that would destroy their data.... (well, it didn't destory their data, but they did ignore my warnings & they did get the worm!)Heheh... Yeah, I know the type.
Along those lines, how's this for bitter irony? The Canadian government's "Office of Critical Infrastructure Protection and Emergency Preparedness" runs IIS on Windows 2000. (Note also the subjects of a couple of the bulletins on their site...) Somehow, this reminds me of doctors who smoke, or mechanics who don't change their oil.
My tax dollars at work. [sigh]
Fire and Meat. Yummy.
You wouldn't be that guy from the redgreen show by any chance ?
Steve Smith? No. But we are on first name basis (no kidding!) and I wear lots of flannel. He once did something on the show which really reminded me of snowblower couch races with friends, and I can't remember if I told him about it or not...
(Snowblower couch racing? You scoop sofas, mattresses and box-springs from the garbage and store them in your backyard. You buy beer and sharpen the ice-cutters on your snowblower. Then you invite over some friends with their own snowblowers and see who can demolish the upholstered furniture the fastest. Truly a good reason to own a snowblower in a warm climate.)
Fire and Meat. Yummy.
The geolocation of networks is never a precise thing. Networks can be registered in the city of their corporate offices but deployed anywhere in the world (it's how the Internet works--cool, huh?).
There are some interesting, precise tricks that you can play by sending various kinds of packets (usually UDP) and using detailed latency information about each hop of those packets, along with a provider network map, to get closer to the physical location of a particular IP address. We didn't do that for these maps.
At my last job we had similar problems. The system was heavily dependent on JMS, so there were rundandant JMS servers. Unfortunately, the first time the primary one on the production network went down under load all the client systems had to send tons of JMS messages around as part of the recovery process, which created a snowball effect that took down the secondary server and many of the other clients too. And then of course the clients started coming back up and sending out JMS messages to announce the fact.... (It turned out to be a bug in the JMS client that eventually got fixed, but it wasn't pretty while it was happening.)
Moral: if you haven't tested the redundancy / failover / power failure mechanism, it might as well not exist.