Slashdot Mirror
Network Blackout
An anonymous reader writes "Renesys put together a special report on the effects of the recent blackout on routing and network reachability on the Internet. It includes a cool animation of networks dropping off the internet (presumably as a result of the power outage). It is interesting to see how localized some of the outage was--networks in New York state right up to the Vermont border go dark while everything on the other side of the border is quiet. New York City obviously gets clobbered."
That little red dot at the tip of LI is my home LAN going down...
The blaming and finger pointing began almost as quickly as the lights went out. First it was the U.S. and Canada blaming each other for causing this particular blackout, but inevitably the blame conversation turned to larger issues of policy, and how something like this could happen in such a heavily regulated industry.
Some of the finger pointing in the national press has been at deregulation -- if it weren't for deregulation, we would be better able to control and manage the grid. This misguided contention is incorrect in a number of ways.
First, the "deregulation" that has occurred in electricity has primarily been in opening up wholesale markets for power generators and their customers (i.e., utilities), enabling people in Manhattan to continue consuming power (and clamoring now for more regulation) without Con Edison having to build more power plants on the island itself. The existence and growing vitality of wholesale electricity markets has created substantial value in the past decade, through encouraging generation where it is cheapest and sales of power to where it is most needed.
But this limited amount of market liberalization has left the industry in an awkward place. Generation is largely governed by market processes, but transmission and retail distribution remain heavily regulated. The investment decisions of transmission owners and the retail rates that they can charge to their end customers all hinge on rate cases that are decided by state-level regulators. The rates that regulators allow take into account changes in costs, required investments, and the payment to the utility of a rate of return on the assets they own. For much of the past decade this rate of return has been substantially lower than what utilities could earn from doing other things with their money, so they did not invest in building much new transmission capacity or in upgrading existing lines. Nor did a regulatory environment that is a relic from the 1930s, constructed to govern and control local, vertically integrated utilities, either have the incentive or the wherewithal to force the utilities to invest in transmission assets that would carry power to customers in other states.
This lack of investment in the infrastructure that carries the product exchanged in growing, vibrant wholesale electricity markets has become a problem -- not an overnight problem, as those who follow the industry have been concerned about transmission capacity for at least five years. The numbers offered this weekend suggest that electricity volume has increased 30 percent while transmission carrying capacity has increased only 15 percent. This fact illustrates the mismatch between the dynamic markets for wholesale power and the rigid, maladaptive set of state-level regulations and incentives that govern transmission investment decisions.
Markets adapt to changing conditions. The existing electricity regulations do not, and because of that, the transmission infrastructure has not adapted to the increased demand on it from the increasing vibrancy of wholesale electricity markets.
So how do we proceed to ensure that a blackout of this magnitude does not happen again? There are four things that can relieve the strain on the grid. The knee-jerk reaction of many people is "build more wires!" More wires will increase the carrying capacity of the system, and in some cases transmission owners can add lines to existing paths. But this approach faces some serious obstacles -- such construction is expensive and time-consuming. Most importantly, though, getting new lines and towers sited is increasingly difficult, as people and communities object to having such large structures near them or strung overhead.
A second option is to use new technologies, such as high-temperature superconductors and sophisticated computer switching, to upgrade the capacity of the existing power lines. While also expensive, this option gets around the NIMBY issues that accompany the siting of new lines.
A third option is to build
Everyone could have switched over to ConMars power. Being a Russian installation, they would have just strung about 17million Home Depot Heavy Duty extension cords all plugged together from here to Mars (the outdoor version of course) to access it.
Is that a red circle on the webserver hosting that gif?
Banaaaana!
I'm sorry. I had apparently text'd most of my friends saying electricity was poor, and almost immediately word spread around the north east..
When modding "Informative", please make sure it both has a source and IS actually informative.
Don't these backbone routers have backup? I was in an ISP server house in the UK which had a full backup system. In the case of a power failure, it had a UPS that kicked in for 10 seconds while the generator was booting up, which then provided power for the infrastructure of the building. I would find it hard to believe that in the USA they don't have similar systems?
As the link says - http://www.socio.demon.co.uk/mphil/appendix1.html [Quote] The Internet is a web of several thousand computer networks that now extends to just about every region of the world and has 50 to 100 million users. The Internet of today has it's origins in a networking project called ARPAnet which was run by the Advanced Research Projects Agency, a science research body set up in 1957 by the Pentagon. (Hafner & Lyon, 1996, 19) The popular belief is that the military created the ARPAnet, the precursor of today's Internet, so that data held on Pentagon computers could survive a nuclear attack by the Soviet Union. Upon attack, data from computers at the Pentagon and other military installations could be uploaded (sent electronically) to other remote computers not affected by such an attack. [/quote] It's always been my understanding that the Internet would continue working after a Nuclear war, at least that was the plan. If this blackout had effected all of US / Canada like a Nuclear attack would, would any of the Net worked ?
Apparently not around my neck of the woods... I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.
Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.
I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.
Seriously, you've got to get out more.
http://use.perl.org
oh yeah, the cell network here was down for a good while after the lights went out. well not down, just full. i thought they learned on sept 11, that there wasnt enough capacity on the cell networks. but you know, i could be wrong.
I want 2D games back.
Woo hoo! Notice how all the outages are south of the border? We are so good! ;-)
Would love to see an animation of a webserver being slashdotted.
Anyone else notice that node in Florida go down as well???
Donald 'Duck' Dunn: We had a band powerful enough to turn goat piss into gasoline.
You don't have backup power on your home LAN? Pffft, and you call yourself a geek.
From what I know, this failure happened from New York City to Ohio in a matter of under a minute. So, I guess the dots are more representative of the average lifespan of the (formerly) fully-charged batteries in one's UPS.
As for me, I was dead in the water. At home, it was instantaneous (I'm too cheap to buy a UPS for a site which is just for my personal amusement); at work, it was 10 minutes of standing there in the server room listening to the frantic beep-beep-beep of UPSes all around me, and then rushing around to connect low-power LCD monitors to the servers that someone else forgot to connect to the UPS's shutdown signals... (of course, this after the realization that the emergency generator is running, but not actually even connected to the servers... [grumble])
Fire and Meat. Yummy.
14th 4:20pm - Power goes out, our building's generators for some reason dont kick in.
14th 4:35 - Most of us decide to call it a day and go home
14th 4:30 - I'm in my car, I realize the blackout is bad when only 2 or 3 radio stations are working and the no traffic lights are. I know it was going to be a fun drive home - thank god i live 15 mins away.
14th 4:45 - I hear the blackout expands to parts of the States. me: OH FUCK@#$@#$
14th 4:50 - My sister sms's me on my cell telling me her and my mom are stuck in the subway - they need help. Like I care, I have my own problems , traffic is a mess and there are hundreds of psychos out.
14th 5:00 - I get home, look for a battery powered radio and listen in.
14th 5:30 - I get a call from my sister - they're stuck somewhere downtown. I just wish them luck.
14th 7:00 - I realize there are no candles in the freaking house, time to look for those puppies.
14th 8:00 - Sister & mom arrive home. me: LOL
14th 11:30 - I go to bed & pray the blackout lasts until the middle of the next day, that way I get an extended weekend -wohooo, back to bed
15th 7:30am - Wake up, lights are still out - no work, home free! wohoo, back to bed.
15th 11:30 - Receive a call from my boss, asking me where the fuck i was since they got power at the office but there is a lil "issue".
15th 12:30m - Got to the office, problem: No Internet connection, seems one of the ISP's switches went bye bye after the black out. Our main app server is down. No power you think? Nope The colo company hasnt been paying their bills and WorldCom used the blackout to pull the plug on them. Server is up and running but no outside world connection. FUCK@#*$.
15th 1:00pm - We think , no biggy let's use one of our other servers and restore apps and data from backup. HA! yah right - Turns out a DNS servers for our backup machines had died and the backup script had stopped working 10 days earlier. Great.
15th 2:30 - We wait to see if Worldcom is nice enough to plug the box back in.
15th 4:30 - Yah, it ain't happening - 20+ clients are without website and apps.
15th 5:00pm - Boss and I drive downtown to the WorldCom building to download data physically off the Box.
15th 5:30 - Stop for gas - HA! huge lineup.
15th 7:30 - Get into the server room, ha! the fucking cage where our box is is locked and the key is not working. One of us climbs the cage and goes into it, runs an ethernet cable from the box to the laptop. So picture these, 4 geeks inside a server room, three sitting on the floor , one inside a cage like some wild animal. I should have brought a camera. Let the tar'in begin.
15th 9:00 - Download is completed, our asses are sore from sitting on concrete, necks hurting, and WorldCom employee happy that we're finally leaving.
15th 9:30 - We're downtown wondering how the fuck we're going to upload 2gig+ worth of data and source code to our spare server.
15th 10:30 - Since there is no inet at the office and our home's cable is too fucking slow (Rogers cable sucks!) , we decide we bring out the ghetto in us. We walk up and down Yonge street asking Internet cafe's if they could lend us some bandwith!!!. Yes, you hear me right, we were begging for bandwith in internet cafes.
15th 10:45 - We decide we're hungry, so we stop at a sushi bar. After we're done we realized it might not be a good idea to eat fish after a blackout. Fridges not working aand all. Too fuckin late.
15th 11:00 - Found an internet cafe that will let us connect the laptop to upload.
15th 11:30 - Realize we can't do shit since the computer is in Korean , have all Win settings locked and the guy taking care of the place has no clue.
16th 1:30am - I'm at my boss's house uploading 2gig+ data , will take about 27 hours. Ask me if I cared about the clients at this point.
16th 1:35 - I leave and head to my friend's place where they're having beers & bbq'in on my friend's balcony. - My weekend Begins.
Network Blackouts? Yah they suck.
[alk]
Apparently not around my neck of the woods... I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.
Yeah, same up here in Ottawa, Canada... I was awakened early on Friday morning to the sound of my servers POSTing; my power was back in under 12 hours. I was lucky. :) (Made sure to double-check that hdparm was set to spin down the drives, that and killing the A/C were my contributions to energy efficiency.)
Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours.I don't have a UPS (well, I do, I got one free, but it's broken and I haven't had time to troubleshoot it - anyone got schematics for an APC Back-UPS Pro 280?), so your mileage may vary. If the UPS runs off 12V batteries, you might be able to:
Note that I don't know how the UPS's inverter will handle running at rated load for longer than the internal battery is capable, nor do I expect that the UPS will have much noise suppression on the battery leads - after all, batteries themselves are pretty much noise-free electrical sources and alternators are not.
Fire and Meat. Yummy.
You've got to be kidding....
No.
I hope that it was someone elses negligence & not yours!Yes. In fact, I did learn something: there's a very good reason that I hate KVM switches.
Solution (probably falling on deaf ears, but I shall try anyway):
- Keyboard and monitor on all servers. Monitors should be LCD to save UPS batteries. This way, if the UPS shutdown sequence fails, it can be done manually. (A distressing number of these machines run Redmond's cancer, so having one central "shutdown console" is less practical.)
- Serial shutdown outputs from UPS will be "broadcast" to all hosts connected to that UPS. Will require making a custom cable with a few MAX232 line-driver ICs and hang it off a machine's PS/2 port for power.
- Backup generator is rated for 90kW and showed only about 10kW load running emergency lights around the building. Time to tap out that extra 80kW of capacity. Why this wasn't checked before is an absolute mystery to me.
- Reorganize server interconnections. The network switch for the users' LAN wasn't on UPS (but neither are the users, so it didn't seem like a big deal). However, over the years, some stuff has come to rely on mapped drives... fortunately in this case, we're not so lucky to have hard-mounted NFS anywhere.
:)
- Have occasional practice power outages before long weekends. Who doesn't test their UPS by unplugging it from time to time? [grin]
I guess I'll stop complaining about the 50 l-users who ignored my emails on Jul.28 - Aug.1 warning them of the impending RPC vulnerability worm that would destroy their data.... (well, it didn't destory their data, but they did ignore my warnings & they did get the worm!)Heheh... Yeah, I know the type.
Along those lines, how's this for bitter irony? The Canadian government's "Office of Critical Infrastructure Protection and Emergency Preparedness" runs IIS on Windows 2000. (Note also the subjects of a couple of the bulletins on their site...) Somehow, this reminds me of doctors who smoke, or mechanics who don't change their oil.
My tax dollars at work. [sigh]
Fire and Meat. Yummy.
You wouldn't be that guy from the redgreen show by any chance ?
Steve Smith? No. But we are on first name basis (no kidding!) and I wear lots of flannel. He once did something on the show which really reminded me of snowblower couch races with friends, and I can't remember if I told him about it or not...
(Snowblower couch racing? You scoop sofas, mattresses and box-springs from the garbage and store them in your backyard. You buy beer and sharpen the ice-cutters on your snowblower. Then you invite over some friends with their own snowblowers and see who can demolish the upholstered furniture the fastest. Truly a good reason to own a snowblower in a warm climate.)
Fire and Meat. Yummy.
The geolocation of networks is never a precise thing. Networks can be registered in the city of their corporate offices but deployed anywhere in the world (it's how the Internet works--cool, huh?).
There are some interesting, precise tricks that you can play by sending various kinds of packets (usually UDP) and using detailed latency information about each hop of those packets, along with a provider network map, to get closer to the physical location of a particular IP address. We didn't do that for these maps.
anyone? Anyone? Bueller?
(sigh)
MjM
Groovy. Gear. Mod.
XKCD:Xeric Knowledge Comically Dispen
Why was there an outage in Boston, MA and Springfield, MA -- Massachusetts did not loose any power?!