Slashdot Mirror
Network Blackout
An anonymous reader writes "Renesys put together a special report on the effects of the recent blackout on routing and network reachability on the Internet. It includes a cool animation of networks dropping off the internet (presumably as a result of the power outage). It is interesting to see how localized some of the outage was--networks in New York state right up to the Vermont border go dark while everything on the other side of the border is quiet. New York City obviously gets clobbered."
That little red dot at the tip of LI is my home LAN going down...
In case the site is slow, for whatever reason, here are a couple mirrors for link 2 and link 3.
A programmer is a machine for converting coffee into code.
The blaming and finger pointing began almost as quickly as the lights went out. First it was the U.S. and Canada blaming each other for causing this particular blackout, but inevitably the blame conversation turned to larger issues of policy, and how something like this could happen in such a heavily regulated industry.
Some of the finger pointing in the national press has been at deregulation -- if it weren't for deregulation, we would be better able to control and manage the grid. This misguided contention is incorrect in a number of ways.
First, the "deregulation" that has occurred in electricity has primarily been in opening up wholesale markets for power generators and their customers (i.e., utilities), enabling people in Manhattan to continue consuming power (and clamoring now for more regulation) without Con Edison having to build more power plants on the island itself. The existence and growing vitality of wholesale electricity markets has created substantial value in the past decade, through encouraging generation where it is cheapest and sales of power to where it is most needed.
But this limited amount of market liberalization has left the industry in an awkward place. Generation is largely governed by market processes, but transmission and retail distribution remain heavily regulated. The investment decisions of transmission owners and the retail rates that they can charge to their end customers all hinge on rate cases that are decided by state-level regulators. The rates that regulators allow take into account changes in costs, required investments, and the payment to the utility of a rate of return on the assets they own. For much of the past decade this rate of return has been substantially lower than what utilities could earn from doing other things with their money, so they did not invest in building much new transmission capacity or in upgrading existing lines. Nor did a regulatory environment that is a relic from the 1930s, constructed to govern and control local, vertically integrated utilities, either have the incentive or the wherewithal to force the utilities to invest in transmission assets that would carry power to customers in other states.
This lack of investment in the infrastructure that carries the product exchanged in growing, vibrant wholesale electricity markets has become a problem -- not an overnight problem, as those who follow the industry have been concerned about transmission capacity for at least five years. The numbers offered this weekend suggest that electricity volume has increased 30 percent while transmission carrying capacity has increased only 15 percent. This fact illustrates the mismatch between the dynamic markets for wholesale power and the rigid, maladaptive set of state-level regulations and incentives that govern transmission investment decisions.
Markets adapt to changing conditions. The existing electricity regulations do not, and because of that, the transmission infrastructure has not adapted to the increased demand on it from the increasing vibrancy of wholesale electricity markets.
So how do we proceed to ensure that a blackout of this magnitude does not happen again? There are four things that can relieve the strain on the grid. The knee-jerk reaction of many people is "build more wires!" More wires will increase the carrying capacity of the system, and in some cases transmission owners can add lines to existing paths. But this approach faces some serious obstacles -- such construction is expensive and time-consuming. Most importantly, though, getting new lines and towers sited is increasingly difficult, as people and communities object to having such large structures near them or strung overhead.
A second option is to use new technologies, such as high-temperature superconductors and sophisticated computer switching, to upgrade the capacity of the existing power lines. While also expensive, this option gets around the NIMBY issues that accompany the siting of new lines.
A third option is to build
Everyone could have switched over to ConMars power. Being a Russian installation, they would have just strung about 17million Home Depot Heavy Duty extension cords all plugged together from here to Mars (the outdoor version of course) to access it.
Is that a red circle on the webserver hosting that gif?
Banaaaana!
I'm sorry. I had apparently text'd most of my friends saying electricity was poor, and almost immediately word spread around the north east..
When modding "Informative", please make sure it both has a source and IS actually informative.
Don't these backbone routers have backup? I was in an ISP server house in the UK which had a full backup system. In the case of a power failure, it had a UPS that kicked in for 10 seconds while the generator was booting up, which then provided power for the infrastructure of the building. I would find it hard to believe that in the USA they don't have similar systems?
Note: the dots only represent 5% of the actual NUMBER of routers downed. (though I bet their locations are based off of an average)
"Sorry Im not more user-friendly."
All that power outage, and not a single network outage. Woohoo!
Pah! You dont need a power grid to run computers. Just hook up the computer to your handy Mr. Fusion, toss in an old banana peel or two, and get cracking!
Seriously though, this has to be one of the stupidest articles ever. It's like putting together a report on "the effects of a gas shortage on cars". Duh! Then again, micheal submitted the article, so I guess we shouldn't be too surprised.
Internal corporate networks may often use third parties to provide links between different offices. In this case, the office in Florida may have been connected to the New York office via a satellite link. Any outgoing traffic would have gone from Florida to New York to the rest of the world.
As the link says - http://www.socio.demon.co.uk/mphil/appendix1.html [Quote] The Internet is a web of several thousand computer networks that now extends to just about every region of the world and has 50 to 100 million users. The Internet of today has it's origins in a networking project called ARPAnet which was run by the Advanced Research Projects Agency, a science research body set up in 1957 by the Pentagon. (Hafner & Lyon, 1996, 19) The popular belief is that the military created the ARPAnet, the precursor of today's Internet, so that data held on Pentagon computers could survive a nuclear attack by the Soviet Union. Upon attack, data from computers at the Pentagon and other military installations could be uploaded (sent electronically) to other remote computers not affected by such an attack. [/quote] It's always been my understanding that the Internet would continue working after a Nuclear war, at least that was the plan. If this blackout had effected all of US / Canada like a Nuclear attack would, would any of the Net worked ?
Apparently not around my neck of the woods... I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.
Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.
I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.
Seriously, you've got to get out more.
http://use.perl.org
I've yet to see a datacenter that didn't have some sort of backup power... I've got backup power in my house, for pity sake.
Anyone in the affected areas care to comment on what happened? Did you guys just exhaust your UPS capacity, or do you just have it for orderly shutdown?
There's boku generators still floating around from all that Y2K kerfluffle... you could probably purchase some cheap failover power just about anywhere...
Even if a man chops off your hand with a sword, you still have two nice, sharp bones to stick in his eyes.
oh yeah, the cell network here was down for a good while after the lights went out. well not down, just full. i thought they learned on sept 11, that there wasnt enough capacity on the cell networks. but you know, i could be wrong.
I want 2D games back.
Woo hoo! Notice how all the outages are south of the border? We are so good! ;-)
Would love to see an animation of a webserver being slashdotted.
Anyone else notice that node in Florida go down as well???
Donald 'Duck' Dunn: We had a band powerful enough to turn goat piss into gasoline.
Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.
Why would you want it all in batteries? Use the UPS to tide you over until you can fire up the gas/diesel generator. Those you can get pretty cheaply (well, compared to 300lbs of batteries) and are useful for other things too - such as going camping.
You don't have backup power on your home LAN? Pffft, and you call yourself a geek.
From what I know, this failure happened from New York City to Ohio in a matter of under a minute. So, I guess the dots are more representative of the average lifespan of the (formerly) fully-charged batteries in one's UPS.
As for me, I was dead in the water. At home, it was instantaneous (I'm too cheap to buy a UPS for a site which is just for my personal amusement); at work, it was 10 minutes of standing there in the server room listening to the frantic beep-beep-beep of UPSes all around me, and then rushing around to connect low-power LCD monitors to the servers that someone else forgot to connect to the UPS's shutdown signals... (of course, this after the realization that the emergency generator is running, but not actually even connected to the servers... [grumble])
Fire and Meat. Yummy.
i'm up in canada and when the power went out our internet conenction didn't. i know this because one of the guys from our offices out west called and asked me what was going on. tho our network did go down due to our ups running out of juice.
the only thing that did NOT go down during the power outage up here was our telephone system. i'm not sure about our other net conenctions tho, i know my home cable connection went down, that's cause cable was out in certain areas, tho in other areas the cable was fine, our bell dsl was fine because bell's authenication servers are in quebec, who would have guessed the french were useful?
i'm guessing that all of our networks were up, i guess the diesel generators a lot of the big data centers have installed came in handy.
14th 4:20pm - Power goes out, our building's generators for some reason dont kick in.
14th 4:35 - Most of us decide to call it a day and go home
14th 4:30 - I'm in my car, I realize the blackout is bad when only 2 or 3 radio stations are working and the no traffic lights are. I know it was going to be a fun drive home - thank god i live 15 mins away.
14th 4:45 - I hear the blackout expands to parts of the States. me: OH FUCK@#$@#$
14th 4:50 - My sister sms's me on my cell telling me her and my mom are stuck in the subway - they need help. Like I care, I have my own problems , traffic is a mess and there are hundreds of psychos out.
14th 5:00 - I get home, look for a battery powered radio and listen in.
14th 5:30 - I get a call from my sister - they're stuck somewhere downtown. I just wish them luck.
14th 7:00 - I realize there are no candles in the freaking house, time to look for those puppies.
14th 8:00 - Sister & mom arrive home. me: LOL
14th 11:30 - I go to bed & pray the blackout lasts until the middle of the next day, that way I get an extended weekend -wohooo, back to bed
15th 7:30am - Wake up, lights are still out - no work, home free! wohoo, back to bed.
15th 11:30 - Receive a call from my boss, asking me where the fuck i was since they got power at the office but there is a lil "issue".
15th 12:30m - Got to the office, problem: No Internet connection, seems one of the ISP's switches went bye bye after the black out. Our main app server is down. No power you think? Nope The colo company hasnt been paying their bills and WorldCom used the blackout to pull the plug on them. Server is up and running but no outside world connection. FUCK@#*$.
15th 1:00pm - We think , no biggy let's use one of our other servers and restore apps and data from backup. HA! yah right - Turns out a DNS servers for our backup machines had died and the backup script had stopped working 10 days earlier. Great.
15th 2:30 - We wait to see if Worldcom is nice enough to plug the box back in.
15th 4:30 - Yah, it ain't happening - 20+ clients are without website and apps.
15th 5:00pm - Boss and I drive downtown to the WorldCom building to download data physically off the Box.
15th 5:30 - Stop for gas - HA! huge lineup.
15th 7:30 - Get into the server room, ha! the fucking cage where our box is is locked and the key is not working. One of us climbs the cage and goes into it, runs an ethernet cable from the box to the laptop. So picture these, 4 geeks inside a server room, three sitting on the floor , one inside a cage like some wild animal. I should have brought a camera. Let the tar'in begin.
15th 9:00 - Download is completed, our asses are sore from sitting on concrete, necks hurting, and WorldCom employee happy that we're finally leaving.
15th 9:30 - We're downtown wondering how the fuck we're going to upload 2gig+ worth of data and source code to our spare server.
15th 10:30 - Since there is no inet at the office and our home's cable is too fucking slow (Rogers cable sucks!) , we decide we bring out the ghetto in us. We walk up and down Yonge street asking Internet cafe's if they could lend us some bandwith!!!. Yes, you hear me right, we were begging for bandwith in internet cafes.
15th 10:45 - We decide we're hungry, so we stop at a sushi bar. After we're done we realized it might not be a good idea to eat fish after a blackout. Fridges not working aand all. Too fuckin late.
15th 11:00 - Found an internet cafe that will let us connect the laptop to upload.
15th 11:30 - Realize we can't do shit since the computer is in Korean , have all Win settings locked and the guy taking care of the place has no clue.
16th 1:30am - I'm at my boss's house uploading 2gig+ data , will take about 27 hours. Ask me if I cared about the clients at this point.
16th 1:35 - I leave and head to my friend's place where they're having beers & bbq'in on my friend's balcony. - My weekend Begins.
Network Blackouts? Yah they suck.
[alk]
Apparently not around my neck of the woods... I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.
Yeah, same up here in Ottawa, Canada... I was awakened early on Friday morning to the sound of my servers POSTing; my power was back in under 12 hours. I was lucky. :) (Made sure to double-check that hdparm was set to spin down the drives, that and killing the A/C were my contributions to energy efficiency.)
Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours.I don't have a UPS (well, I do, I got one free, but it's broken and I haven't had time to troubleshoot it - anyone got schematics for an APC Back-UPS Pro 280?), so your mileage may vary. If the UPS runs off 12V batteries, you might be able to:
Note that I don't know how the UPS's inverter will handle running at rated load for longer than the internal battery is capable, nor do I expect that the UPS will have much noise suppression on the battery leads - after all, batteries themselves are pretty much noise-free electrical sources and alternators are not.
Fire and Meat. Yummy.
Those were all the old nodes that retired and moved south to Florida.......
Oklahomah had a node go down too..WTF? Ya got me on that one...Can't even make a joke about that..
..........FULL STOP.
Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :(
You want fuel cells, not batteries.
$x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
$x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
You've got to be kidding....
No.
I hope that it was someone elses negligence & not yours!Yes. In fact, I did learn something: there's a very good reason that I hate KVM switches.
Solution (probably falling on deaf ears, but I shall try anyway):
- Keyboard and monitor on all servers. Monitors should be LCD to save UPS batteries. This way, if the UPS shutdown sequence fails, it can be done manually. (A distressing number of these machines run Redmond's cancer, so having one central "shutdown console" is less practical.)
- Serial shutdown outputs from UPS will be "broadcast" to all hosts connected to that UPS. Will require making a custom cable with a few MAX232 line-driver ICs and hang it off a machine's PS/2 port for power.
- Backup generator is rated for 90kW and showed only about 10kW load running emergency lights around the building. Time to tap out that extra 80kW of capacity. Why this wasn't checked before is an absolute mystery to me.
- Reorganize server interconnections. The network switch for the users' LAN wasn't on UPS (but neither are the users, so it didn't seem like a big deal). However, over the years, some stuff has come to rely on mapped drives... fortunately in this case, we're not so lucky to have hard-mounted NFS anywhere.
:)
- Have occasional practice power outages before long weekends. Who doesn't test their UPS by unplugging it from time to time? [grin]
I guess I'll stop complaining about the 50 l-users who ignored my emails on Jul.28 - Aug.1 warning them of the impending RPC vulnerability worm that would destroy their data.... (well, it didn't destory their data, but they did ignore my warnings & they did get the worm!)Heheh... Yeah, I know the type.
Along those lines, how's this for bitter irony? The Canadian government's "Office of Critical Infrastructure Protection and Emergency Preparedness" runs IIS on Windows 2000. (Note also the subjects of a couple of the bulletins on their site...) Somehow, this reminds me of doctors who smoke, or mechanics who don't change their oil.
My tax dollars at work. [sigh]
Fire and Meat. Yummy.
Actually, parts of Ontario East of Ottawa are actually hooked up to the Quebec grid.
The line is actually right beside my house. The block on the left was dark, humid, black Ontario power. On the right (and in my house)... bright, cool, soothing Quebec power.
So I strung up a couple hundred feet of electrical cable, pulled out the coolers, invited over all the neighbours, and drank cold beer all night.
On another note.. did anyone else notice how many more of the stars they could see that night?!?
You wouldn't be that guy from the redgreen show by any chance ?
Steve Smith? No. But we are on first name basis (no kidding!) and I wear lots of flannel. He once did something on the show which really reminded me of snowblower couch races with friends, and I can't remember if I told him about it or not...
(Snowblower couch racing? You scoop sofas, mattresses and box-springs from the garbage and store them in your backyard. You buy beer and sharpen the ice-cutters on your snowblower. Then you invite over some friends with their own snowblowers and see who can demolish the upholstered furniture the fastest. Truly a good reason to own a snowblower in a warm climate.)
Fire and Meat. Yummy.
The geolocation of networks is never a precise thing. Networks can be registered in the city of their corporate offices but deployed anywhere in the world (it's how the Internet works--cool, huh?).
There are some interesting, precise tricks that you can play by sending various kinds of packets (usually UDP) and using detailed latency information about each hop of those packets, along with a provider network map, to get closer to the physical location of a particular IP address. We didn't do that for these maps.
I thought that the internet was supposed to be redundant (like power)... i thought they learned on sept 11, that there wasnt enough capacity on the cell networks
We had a presentation last night from Tom O'Rourke on the critical infrastructure affected by Sept 11 2001.
The main point to come out of it is that most critical infrastructure is commercially run. They are designed to run and handle typical peak loading.
And there is a difference between redundancy and surge capacity. Redundancy allows you to continue operating at a reduced or normal level. Surge capacity allows you to operate at increased levels due to an unexpected event - such as handling increased mobile phone demand by increasing available spectrum bandwidth as was also done after Sept 11 2001.
The problem that you have in these sort of failures is that not only have you lost capacity, but you see increased demand at the same time. One slide he had detailed cell phones in NYC. Typical block rate (no tone) is about 4% - ie one in 25 calls you can't make. After losing all the capacity after the collapse, combined with everyone wanting to talk on their phones, the blocked call rate jumped to 92%. The expense required handle these sort of extreme events cannot be justified.
Utilities can handle ordinary spikes in their systems, but it is not economic to design surge capacity into most systems.
You'll find this problem across a number of sectors... telecommunications, power, and hospital beds tend to provide the best examples. You'd be suprised at how few hospital beds are generally available at any given point in time.
If we had a mass casualty event in New Zealand, it is quite possible we would be sending victims to hospitals in Australia because we don't have the surge capacity.
Cheers Gav
anyone? Anyone? Bueller?
(sigh)
Whereas a 200HP car engine can put out nearly 150KW of power at peak, which is enough to run several suburban homes, the alternator can probably only put out around 1KW max. Be careful not to overload and burn it out, costing you more money than a couple extra batteries would have. }:)
Cars are designed to haul themselves and your ass around, not keep your little server farm running.
Sender: jm
... it becomes a bit more likely if one considers what the authors of
0 1/
To: dave
Hi Dave -- for IP.
There's an article from Heise Security in Germany at [1], which raises
some interesting questions about whether W32.Blaster could be to blame for
the blackout. Some translated points are at [2] -- quote:
that article found out:
- The Niagara Mohawk power grid which seemed to got overloaded first
is owned by National Grid USA.
- National Grid is listed as an important customer of Northern
Dynamic who call themselves the "OPC Experts".
- OPC is an acronym for OLE for Process Control and is used for
communications between control systems.
- OPC is based on DCOM, exactly that Windows technology attacked by
W32.Blaster.
- One symptom of a W32.Blaster attack is that a crashing DCOM
service (not only under Windows), often taking down the whole
server.
One usage of OPC is the coupling of so-called SCADA (Supervisory Control
and Data Acquisition) systems. Among other things is SCADA used in power
plants and grids to exchange data between some central instance and
external measuring units. And for some reason did the monitoring system
which should prevent snowball effects like the one on thursday from
happening.
So the questions the authors of the article have are:
- For which processes does National Grid utilise OPC?
- Were there any problems regarding OPC when the power went down?
- If yes, were they related to W32.Blaster?
1. http://www.heise.de/newsticker/data/ju-15.08.03-0
2. http://msquadrat.de/archive/03/08/16/02
--j.
MjM
Groovy. Gear. Mod.
XKCD:Xeric Knowledge Comically Dispen
Why was there an outage in Boston, MA and Springfield, MA -- Massachusetts did not loose any power?!
Here is a nice satellite pic comparison of the Northeast before and during the outage from Natural Hazards
Altenators in new cars aren't really good for this:
An altenator is different from a generator in that the supplied field is electromagnetic...it depends on another voltage source. This source is supplied (initially) by the battery or residual magnetic field when you start up the car, and is varied by the regulator. When your car needs recharging, the regulator ups the current or voltage in the altenators field wire, and your car cranks out more volts and recharges the battery. When the battery is mostly charged the regulator trims the field down so the alternator produces about 13V.
This has the advantage of not robbing your engine of power and gas milage when the battery has a full charge. However, to use it as a generator, you're going to have to hard wire the field to a constant (and not to the altenator output-- this creates a infinite voltage loop that will kill the altenator).
The reason new car altenators are not ideal is because the voltage regulator is internal to the altenator. This was done for simplicity and to make sure the regulator was always grounded, but it really complicates what we're trying to do. Older cars (until late 70's, early 80's) have external regulators and cars older than that (early to mid 60's ?) had generators (field was a permenant magnet).
Also, you do NOT want to bypass the car battery. It is an excellent power conditioner for the not-so-even generation of the altenator/generator
- Sig