Slashdot Mirror


Network Blackout

An anonymous reader writes "Renesys put together a special report on the effects of the recent blackout on routing and network reachability on the Internet. It includes a cool animation of networks dropping off the internet (presumably as a result of the power outage). It is interesting to see how localized some of the outage was--networks in New York state right up to the Vermont border go dark while everything on the other side of the border is quiet. New York City obviously gets clobbered."

183 comments

  1. I can see my house from here. by Purosesuchi-Zu · · Score: 5, Funny

    That little red dot at the tip of LI is my home LAN going down...

    1. Re:I can see my house from here. by seinman · · Score: 0, Funny

      You don't have backup power on your home LAN? Pffft, and you call yourself a geek.

    2. Re:I can see my house from here. by Purosesuchi-Zu · · Score: 4, Funny

      Ironically I do, but my ISP doesn't

    3. Re:I can see my house from here. by Anonymous Coward · · Score: 0
      That little red dot at the tip of LI is my home LAN going down...

      Pardon me, but where is LI?

    4. Re:I can see my house from here. by Anonymous Coward · · Score: 0

      Long Island. It is, well, that Long Island east of New York City on the map.

    5. Re:I can see my house from here. by Purosesuchi-Zu · · Score: 0

      Long Island (LI) is off the coast of New York

    6. Re:I can see my house from here. by Purosesuchi-Zu · · Score: 0

      Oops, guess I shouldn't reply as soon as I get e-mail notification. -1 Redundant :P

  2. Mirror by inertia187 · · Score: 2, Informative

    In case the site is slow, for whatever reason, here are a couple mirrors for link 2 and link 3.

    --
    A programmer is a machine for converting coffee into code.
  3. For the "Deregulation was the problem" nutjobs by Anonymous Coward · · Score: 3, Informative

    The blaming and finger pointing began almost as quickly as the lights went out. First it was the U.S. and Canada blaming each other for causing this particular blackout, but inevitably the blame conversation turned to larger issues of policy, and how something like this could happen in such a heavily regulated industry.

    Some of the finger pointing in the national press has been at deregulation -- if it weren't for deregulation, we would be better able to control and manage the grid. This misguided contention is incorrect in a number of ways.

    First, the "deregulation" that has occurred in electricity has primarily been in opening up wholesale markets for power generators and their customers (i.e., utilities), enabling people in Manhattan to continue consuming power (and clamoring now for more regulation) without Con Edison having to build more power plants on the island itself. The existence and growing vitality of wholesale electricity markets has created substantial value in the past decade, through encouraging generation where it is cheapest and sales of power to where it is most needed.

    But this limited amount of market liberalization has left the industry in an awkward place. Generation is largely governed by market processes, but transmission and retail distribution remain heavily regulated. The investment decisions of transmission owners and the retail rates that they can charge to their end customers all hinge on rate cases that are decided by state-level regulators. The rates that regulators allow take into account changes in costs, required investments, and the payment to the utility of a rate of return on the assets they own. For much of the past decade this rate of return has been substantially lower than what utilities could earn from doing other things with their money, so they did not invest in building much new transmission capacity or in upgrading existing lines. Nor did a regulatory environment that is a relic from the 1930s, constructed to govern and control local, vertically integrated utilities, either have the incentive or the wherewithal to force the utilities to invest in transmission assets that would carry power to customers in other states.

    This lack of investment in the infrastructure that carries the product exchanged in growing, vibrant wholesale electricity markets has become a problem -- not an overnight problem, as those who follow the industry have been concerned about transmission capacity for at least five years. The numbers offered this weekend suggest that electricity volume has increased 30 percent while transmission carrying capacity has increased only 15 percent. This fact illustrates the mismatch between the dynamic markets for wholesale power and the rigid, maladaptive set of state-level regulations and incentives that govern transmission investment decisions.

    Markets adapt to changing conditions. The existing electricity regulations do not, and because of that, the transmission infrastructure has not adapted to the increased demand on it from the increasing vibrancy of wholesale electricity markets.

    So how do we proceed to ensure that a blackout of this magnitude does not happen again? There are four things that can relieve the strain on the grid. The knee-jerk reaction of many people is "build more wires!" More wires will increase the carrying capacity of the system, and in some cases transmission owners can add lines to existing paths. But this approach faces some serious obstacles -- such construction is expensive and time-consuming. Most importantly, though, getting new lines and towers sited is increasingly difficult, as people and communities object to having such large structures near them or strung overhead.

    A second option is to use new technologies, such as high-temperature superconductors and sophisticated computer switching, to upgrade the capacity of the existing power lines. While also expensive, this option gets around the NIMBY issues that accompany the siting of new lines.

    A third option is to build

    1. Re:For the "Deregulation was the problem" nutjobs by tmu · · Score: 2, Interesting
      Security focus is saying that a previous outage wasn't deregulation, wasn't transmission, it was the slammer worm hitting Ohio's Davis-Besse nuclear power plant and disabling a safety monitoring program.

      Interesting story. CNN and a bunch of people on NANOG (www.nanog.org) were speculating that this outage was caused by the msblaster worm. This story backs up at least the feasibility of that.

  4. Florida? by soliaus · · Score: 0, Offtopic

    What the hell is a router in florida doing going out when it isnt even in the blackout? On the day of the blackout, in Las Vegas (NV), I was in Best Buy. I dont know how or why, but the power there went off too for about 15 minutes. Weird.

    --
    Speaking at Defcon 12 - Credit Card Networks Revisted: Pen
    1. Re:Florida? by Anonymous Coward · · Score: 0

      Likewise the router outage in Nebraska. A fluke?

    2. Re:Florida? by SmackCrackandPot · · Score: 2, Informative

      Internal corporate networks may often use third parties to provide links between different offices. In this case, the office in Florida may have been connected to the New York office via a satellite link. Any outgoing traffic would have gone from Florida to New York to the rest of the world.

    3. Re:Florida? by tmu · · Score: 4, Informative

      The geolocation of networks is never a precise thing. Networks can be registered in the city of their corporate offices but deployed anywhere in the world (it's how the Internet works--cool, huh?).

      There are some interesting, precise tricks that you can play by sending various kinds of packets (usually UDP) and using detailed latency information about each hop of those packets, along with a provider network map, to get closer to the physical location of a particular IP address. We didn't do that for these maps.

  5. This is why we need that Martian Nuclear PP by binaryDigit · · Score: 4, Funny

    Everyone could have switched over to ConMars power. Being a Russian installation, they would have just strung about 17million Home Depot Heavy Duty extension cords all plugged together from here to Mars (the outdoor version of course) to access it.

    1. Re:This is why we need that Martian Nuclear PP by Anonymous Coward · · Score: 0

      I know youre trying to be funny but:
      Ever heard of voltage drop?
      And sun?! (sometimes its between earth and mars..)

    2. Re:This is why we need that Martian Nuclear PP by Anonymous Coward · · Score: 1, Insightful

      I know you're trying to be serious/relevant, but:

      Ever heard of HUMOR??

    3. Re:This is why we need that Martian Nuclear PP by aleph+ · · Score: 2, Funny

      You're off by a factor of at least 100. The minimum distance between Earth and Mars is about 1,500 million extension cords (assuming 50m cords). Right now the two planets are pretty close, but later we'll need as many as 7,500 million cords, plus a few extra so we can swing the cable over the sun.

    4. Re:This is why we need that Martian Nuclear PP by Sivaram_Velauthapill · · Score: 1

      1 million.. 5 million... 1500 million... 7500million... it's all the same. If the Russians can build a power plant on Mars, this is nothing ;)

      Sivaram Velauthapillai, waiting patiently for Russians to power the earth :)

      --
      Sivaram Velauthapillai
      Seeking the meaning of life... @slashdot of all places ;)
    5. Re:This is why we need that Martian Nuclear PP by erpbridge · · Score: 1

      American components, Russia Components...

      they're all MADE IN TAIWAN!

      --Russian cosmonaut, from Armageddon

  6. What's that I see? by Exiler · · Score: 5, Funny

    Is that a red circle on the webserver hosting that gif?

    --
    Banaaaana!
    1. Re:What's that I see? by Anonymous Coward · · Score: 0

      Har har har!!!

      Yeah, that's a red circle on the webserver hosting that gif alrighty!!!

  7. Apology by ticklemeozmo · · Score: 5, Funny

    I'm sorry. I had apparently text'd most of my friends saying electricity was poor, and almost immediately word spread around the north east..

    --
    When modding "Informative", please make sure it both has a source and IS actually informative.
    1. Re:Apology by Anonymous Coward · · Score: 0

      Very funny. I wish I could mod you higher than 5.

  8. backup? by killermal · · Score: 5, Insightful

    Don't these backbone routers have backup? I was in an ISP server house in the UK which had a full backup system. In the case of a power failure, it had a UPS that kicked in for 10 seconds while the generator was booting up, which then provided power for the infrastructure of the building. I would find it hard to believe that in the USA they don't have similar systems?

    1. Re:backup? by soliaus · · Score: 5, Funny
      I would find it hard to believe that in the USA they don't have similar systems?

      We prefer the rat on a wheel approach to power backup. It usually works, we just forgot to feed them so it failed on such a wide spread application

      --
      Speaking at Defcon 12 - Credit Card Networks Revisted: Pen
    2. Re:backup? by Anonymous Coward · · Score: 1, Informative

      Well...I don't think this is limited to backbone routers, and there are a LOT of routers out there...

    3. Re:backup? by doorman · · Score: 1

      Some did, and a few generators failed. We lost our OC-3 from New York to London because of a generator that failed about 4 hours after the blackout started.

      --
      -G "We love to buy books, because we are buying the belief we have time to read them" - Warren Zevon
    4. Re:backup? by Anonymous Coward · · Score: 0

      Shit, your allowed a rat on a wheel? Luxury! Oh how I wish for a rat on a wheel. In my office, I have to get butt naked, grease myself up with cleaning fluid, ram my head against the wall until blood begins to expunge from my cranium, then jump into our aquarium, filled with pirhanas, then swim round and round in circles fast enough to create the required tidal energy to power our generators. And then I have to flog myself to death for the pure gratification of my CFO.

    5. Re:backup? by evilviper · · Score: 2, Informative

      Backbone routers were fully functional. The problem was than many, many smaller networks don't.

      I know it's too much to ask here, but I would suggest you read the story.

      --
      Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
    6. Re:backup? by ChrisCampbell47 · · Score: 3, Informative
      Don't these backbone routers have backup? I was in an ISP server house in the UK which had a full backup system. In the case of a power failure, it had a UPS that kicked in for 10 seconds while the generator was booting up, which then provided power for the infrastructure of the building. I would find it hard to believe that in the USA they don't have similar systems?

      Uh, yeah, thanks, we never thought of that.

      UPS's run out of juice. Generators run out of fuel. Generators turn out to be less than perfectly maintained and fail after a couple hours. Budgets get trimmed, maintenance gets overlooked, blah blah blah it's never a perfect world. If it was, engineers would be replaced by algorithms.

      Companies that are dead serious about power reliability run generator tests every day, and when lightning is detected within miles we automatically start up all the generators and run off them. Yes, we actually go OFF THE GRID every time a thunderstorm rolls in, and in Atlanta that's many times a week. Of course, we've got millions of dollars an hour running through our facility so heads would roll if we weren't this paranoid.

      I believe we keep our tanks fueled up for 3 days of continuous service, and we pay a premium to guarantee that when the shit hits the fan, WE get refueled first.

      Of course, hardcore multiply-redundant (and *tested*) systems are something that elude the typical IT crew staffed by DeVry grads.

    7. Re:backup? by PiratePTG · · Score: 1
      >Yes, we actually go OFF THE GRID every time a thunderstorm rolls in, and in Atlanta that's many times a week. Of course, we've got millions of dollars an hour running through our facility so heads would roll if we weren't this paranoid.

      We did the same thing when I used to work for Home Shopping Network. If the forcast predicted rain, the generators would fire up. If it actually started raining in the Tampa Bay area, we would go off-grid. I heard someone throw out a figure once, I don't know how accurate it is, but supposedly HSN potentially looses $365,000 per minute that it is off the air...

      Guess that diesel fuel bill isn't as bad as it seems...

      As an final thought, when HSN was on a load-sharing basis, prior to going off-grid, Florida Power and Light had to buy the excess power back at retail rates... Making money all around...

      --
      The number 1 problem of working in a cubicle - 23 power cords, 1 outlet...
    8. Re:backup? by ChrisCampbell47 · · Score: 1

      > > Of course, we've got millions of dollars an
      > > hour running through our facility so heads
      > > would roll if we weren't this paranoid.

      > I heard someone throw out a figure once, I don't
      > know how accurate it is, but supposedly HSN
      > potentially looses $365,000 per minute that it
      > is off the air...

      Yeah, that was actually a typo on my part -- I
      meant millions of dollars a minute :)
      [puffs up chest]

  9. keep in mind by citizen6350 · · Score: 2, Interesting

    Note: the dots only represent 5% of the actual NUMBER of routers downed. (though I bet their locations are based off of an average)

    --
    "Sorry Im not more user-friendly."
  10. Woohoo Toronto by Streiff · · Score: 2, Interesting

    All that power outage, and not a single network outage. Woohoo!

    1. Re:Woohoo Toronto by Erick+the+Red · · Score: 1

      That means that although you can't work, you can still read slashdot! Wait, that's what you were doing anyways :P

      --

      DO NOT WRITE IN THIS SPACE

      ok
    2. Re:Woohoo Toronto by killermal · · Score: 0

      Yay for Canada!

    3. Re:Woohoo Toronto by malloc · · Score: 2, Interesting

      I have a friend who runs a small ISP here in Toronto. Today he told me that he never lost network connectivity during the blackout. He's with WorldCom and apparently because of W2K they built facilities downtown so they can last a whole month without power.

      -Malloc

      --
      ___________________ I want to be free()!
    4. Re:Woohoo Toronto by Anonymous Coward · · Score: 1, Informative

      The map only shows US outages, not Canadian network outages

    5. Re:Woohoo Toronto by mnmn · · Score: 1

      Yeah Sympatico here did well. My servers did not have UPSes though and did go out, but they booted back online by themselves so no issue.

      The Toronto Public Library systems were out for a while though. Libraries here are still closed. Those were I suppose the only major servers that did go out.

      --
      "Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
  11. Re:Summary by alexre1 · · Score: 2, Funny

    Pah! You dont need a power grid to run computers. Just hook up the computer to your handy Mr. Fusion, toss in an old banana peel or two, and get cracking!

    Seriously though, this has to be one of the stupidest articles ever. It's like putting together a report on "the effects of a gas shortage on cars". Duh! Then again, micheal submitted the article, so I guess we shouldn't be too surprised.

  12. Add One More... by theGreater · · Score: 1

    Add one more server to that infographic, since I'm fairly certain a pic that large is going to get fragdotted quite speedily.

    -theGreater Pessimist.

  13. looks more like by Anonymous Coward · · Score: 0

    soviet missile strike plan

    1. Re:looks more like by Anonymous Coward · · Score: 0

      or... slashdotters farting... same result. ... hmm, better post this as anonymous...

    2. Re:looks more like by killermal · · Score: 1

      ...from Mars!

    3. Re:looks more like by clf8 · · Score: 1

      Hey now, Russia is no longer a threat. Couldn't be North Korea, they've only got 2-3. China maybe, or a whole buncha Iraqi SCUDS wif chem weapons. They did find those, right?

    4. Re:looks more like by korgull · · Score: 1

      Well, I guess paranoid americans can think of any stupid reason to satisfy their needs.....

  14. just got off the phone with verizon by gimpboy · · Score: 1, Offtopic

    i told them my dsl had been choking off and on for the last day and that my ip address had been changing on the hour. now i get to see pittsburghs own special dot on the map. it's quite comforting :).

    the good news is that they told me it should be fixed in 4 hours (that was two hours ago).

    --
    -- john
    1. Re:just got off the phone with verizon by jmobley · · Score: 1

      Verizon is up to their eyeballs in broken circuits. We keep calling them to fix downed T1's caused by the power outage (I work for a CLEC) and when we ask if the circuits will get looked at today they start laughing.

      I think they've all lost their shit over there. :)

    2. Re:just got off the phone with verizon by MShook · · Score: 1

      Funny you said that. I work in Manhattan and our T1 (the provider is DTI but Verizon owns the line) is still down...

    3. Re:just got off the phone with verizon by Anonymous Coward · · Score: 0

      We're using WorldCom with Verizon providing the fibre into our office in New Brunswick, NJ. Still offline as of this posting and working from home tomorrow to avoid getting lynched by our clients. Neither Verizon or WorldCom will say anything other than 'some areas are still experiencing outages'. /z

  15. Origins of the Internet - no power, no work ? by PaulBartlet7 · · Score: 5, Informative

    As the link says - http://www.socio.demon.co.uk/mphil/appendix1.html [Quote] The Internet is a web of several thousand computer networks that now extends to just about every region of the world and has 50 to 100 million users. The Internet of today has it's origins in a networking project called ARPAnet which was run by the Advanced Research Projects Agency, a science research body set up in 1957 by the Pentagon. (Hafner & Lyon, 1996, 19) The popular belief is that the military created the ARPAnet, the precursor of today's Internet, so that data held on Pentagon computers could survive a nuclear attack by the Soviet Union. Upon attack, data from computers at the Pentagon and other military installations could be uploaded (sent electronically) to other remote computers not affected by such an attack. [/quote] It's always been my understanding that the Internet would continue working after a Nuclear war, at least that was the plan. If this blackout had effected all of US / Canada like a Nuclear attack would, would any of the Net worked ?

    1. Re:Origins of the Internet - no power, no work ? by silas_moeckel · · Score: 2, Informative

      Define work? With current routing topologies you take down all the tier ones and your not getting out of the USA and will have trouble getting much farther than that. Contract wise the tier ones have been applying a lot of presure on the tier 2 guys not to advertise interconnects and often have good reasons not to. Add to this the massive ammounts of long haul centralization take out a few MAE points and things would be bad VPN's are replaceing the long haul circut and as it gets nastier and naster out there firewalls are the norm blocking trafic through corps private backbones and satalite links.

      --
      No sir I dont like it.
    2. Re:Origins of the Internet - no power, no work ? by four12 · · Score: 2, Insightful
      computers could survive a nuclear attack by the Soviet Union.

      Well, yeah, the computers survived but the power grid that runs them and their environmental support got hosed.

    3. Re:Origins of the Internet - no power, no work ? by Ralph+Wiggam · · Score: 5, Informative

      The popular belief that ARPAnet was designed to survive a nuclear war was created by a Time magazine writer who didn't know what he was talking about. ARPAnet was created so that people doing military research could share thier work and the DOD wouldn't have to pay for the same research twice. That's why the first nodes were universities and not military bases. My alma mater, Univ. of Illinois, was supposed to get one of the first nodes outside of CA, but hippie protestors delayed it for a while. Fucking hippies.

      -B

    4. Re:Origins of the Internet - no power, no work ? by Anonymous Coward · · Score: 0

      Hail Alma Mater,
      Ever so true.

    5. Re:Origins of the Internet - no power, no work ? by SmallFurryCreature · · Score: 3, Insightful
      You might be suprised to learn this, but army bases tend to have their own power generators. Silly but for some reason they object to being depended for power on an installation hundreds of miles away.

      So yes the original setup of the internet was to survive stuff like this. As indeed it did. In areas not nuked it continued to work just fine.

      The entire point after all was for the network to survive even if a big hole was punched into it. We just saw that happen. And talked about on the net while it happened showing that the bits around the hole kept working.

      --

      MMO Quests are like orgasms:

      You may solo them, I prefer them in a group.

    6. Re:Origins of the Internet - no power, no work ? by jonbrewer · · Score: 1

      With current routing topologies you take down all the tier ones and your not getting out of the USA and will have trouble getting much farther than that. Contract wise the tier ones have been applying a lot of presure on the tier 2 guys not to advertise interconnects and often have good reasons not to.

      Silly question: Will IPv6 make the Internet more stable? Will it allow the tier 2 providers to trade routes more easily?

    7. Re:Origins of the Internet - no power, no work ? by IncohereD · · Score: 1

      Silly question: Will IPv6 make the Internet more stable? Will it allow the tier 2 providers to trade routes more easily?

      I think his point is the decision is political, not technical. It doesn't matter if IPv6 makes it easier, because the Tier 1 guys don't want it to happen. But, you know, privatized telecom is good for us, ain't it?

    8. Re:Origins of the Internet - no power, no work ? by Ctrl-Z · · Score: 1


      If this blackout had effected all of US / Canada like a Nuclear attack would, would any of the Net worked ?

      If a tree falls in the forest ...

      --
      www.timcoleman.com is a total waste of your time. Never go there.
  16. Re: Don't backbone routers have backup? by Baron_Yam · · Score: 4, Interesting

    Apparently not around my neck of the woods... I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.

    Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.

  17. It stops on Saturday! by sharkey · · Score: 1

    We don't get to see ask.slashdot.org fall off the Internet from this morning.

    --

    --
    "Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
  18. Re: Don't backbone routers have backup? by notque · · Score: 4, Funny

    I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.

    Seriously, you've got to get out more.

    --
    http://use.perl.org
  19. Animation by Anonymous Coward · · Score: 0

    I don't see what's so cool about that animation.

  20. Agreed by The+Tyro · · Score: 2, Interesting

    I've yet to see a datacenter that didn't have some sort of backup power... I've got backup power in my house, for pity sake.

    Anyone in the affected areas care to comment on what happened? Did you guys just exhaust your UPS capacity, or do you just have it for orderly shutdown?

    There's boku generators still floating around from all that Y2K kerfluffle... you could probably purchase some cheap failover power just about anywhere...

    --
    Even if a man chops off your hand with a sword, you still have two nice, sharp bones to stick in his eyes.
  21. redundancy by paradesign · · Score: 4, Interesting
    Im in detroit, and after power went out, so did the network. I thought that the internet was supposed to be redundant (like power)! i thought telcos invested bigtime to keep in the 9's. its good to know it didnt do squat. at work today, we finaly got a stable connection from our upstream routers. our tech department was furious because all of our stuff was on backup, why wasnt theres?

    oh yeah, the cell network here was down for a good while after the lights went out. well not down, just full. i thought they learned on sept 11, that there wasnt enough capacity on the cell networks. but you know, i could be wrong.

    --
    I want 2D games back.
    1. Re:redundancy by Anonymous Coward · · Score: 1, Informative

      I thought that the internet was supposed to be redundant (like power)! i thought telcos invested bigtime to keep in the 9's.

      No. The telcos invested big time to keep the phone system in the 9's. I'm going out on a limb here, but I'll bet anyone who had an old-fashion, line-powered telephone still had a dial tone (and I'm not talking about one of those whiz-bang cordless phones).

    2. Re:redundancy by evilviper · · Score: 1
      after power went out, so did the network. I thought that the internet was supposed to be redundant

      What went out, where, when? Just because your cable modem couldn't connect, doesn't mean the internet was out ;-). I suspect you had something better than that to test, but what?

      Maybe your ISP/hosting provider is just an economy service, and doesn't have any long-term power supply.

      Although, I really don't know why so few had power... A car/truck is a 12-volt generator, and an uninteruptable power supply is a 12-volt inverter. If you can go buy gas, you should be able to power everything you need, indefinately.
      --
      Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
    3. Re:redundancy by IncohereD · · Score: 1

      No. The telcos invested big time to keep the phone system in the 9's. I'm going out on a limb here, but I'll bet anyone who had an old-fashion, line-powered telephone still had a dial tone (and I'm not talking about one of those whiz-bang cordless phones).

      Damn straight. Although to be fair my cable modem worked just fine once the power came back on. Unfortunately My roommate f'ed up his UPS so we couldn't test it during the outage.

    4. Re:redundancy by digitalsushi · · Score: 1

      our tech department was furious because all of our stuff was on backup, why wasnt theres?

      if you're complaining about it, you should consider getting a second upstream so that your network doesnt have a single point of failure :)

      --
      slashdot: where everyone yells sarcastic metaphors to themselves to understand the issue
    5. Re:redundancy by Anonymous Coward · · Score: 0

      Yeah, plain old telephone service worked fine during the blackout (in Brooklyn at least)

    6. Re:redundancy by YZman · · Score: 1

      Detroit also; Compuserve (dialup), no problem. Cable out though.

  22. Canada did not fail by hopbine · · Score: 1

    Cool, Ottawa stayed up, too bad I lost my power so I couldn't enjoy it.

    --
    Semper ubi sub ubi
    1. Re:Canada did not fail by kick_in_the_eye · · Score: 1

      Yup, we did not fail. We do not seem to have any provincial borders either.

    2. Re:Canada did not fail by Seek_1 · · Score: 1

      Actually, Ottawa went down. Quebec (Gatineau which is across the river) stayed up. :( pretty pathetic eh?

    3. Re:Canada did not fail by broken.data · · Score: 2, Interesting

      Actually, parts of Ontario East of Ottawa are actually hooked up to the Quebec grid.

      The line is actually right beside my house. The block on the left was dark, humid, black Ontario power. On the right (and in my house)... bright, cool, soothing Quebec power.

      So I strung up a couple hundred feet of electrical cable, pulled out the coolers, invited over all the neighbours, and drank cold beer all night.

      On another note.. did anyone else notice how many more of the stars they could see that night?!?

    4. Re:Canada did not fail by IncohereD · · Score: 1

      Apparently Quebec runs on 60.25 Hz, in typical freaky frenchmen fashion, so they're in a crazy isolated power zone.

    5. Re:Canada did not fail by Nalmar · · Score: 1
      Apparently Quebec runs on 60.25 Hz, in typical freaky frenchmen fashion, so they're in a crazy isolated power zone.

      I don't know where you get your facts but you're wrong. It's standard 60Hz power here but we didn't fail because are on an asynchronous network. Only texas and us have the newer async networks in north america.

      in typical freaky frenchmen fashion

      Pffff !!

      --
      It's not because we laugh that it's funny
    6. Re:Canada did not fail by IncohereD · · Score: 1

      Ahhh, thank you. I have very little faith in the guy who advanced that point to me, and it kind of sounded like bullshit. Thanks for setting me straight.

  23. Renesys Corporation... by notetoi · · Score: 1

    "Providing real-time internet connectivity monitoring and reporting worldwide"

    Wouldn't "real-time...monitoring" = DoS?

  24. Go Canada!!! by akorvemaker · · Score: 5, Funny

    Woo hoo! Notice how all the outages are south of the border? We are so good! ;-)

    1. Re:Go Canada!!! by Anonymous Coward · · Score: 0

      Gee, I wonder why a map of outages in the US doesn't show any outages in Canada? Duh.....

  25. Slashdotting animation by psylent · · Score: 5, Funny

    Would love to see an animation of a webserver being slashdotted.

    1. Re:Slashdotting animation by Anonymous Coward · · Score: 1, Funny

      > Would love to see an animation of a webserver being slashdotted.

      It probably would look something like this the entire time....

    2. Re:Slashdotting animation by korgull · · Score: 1

      Ah, I'd love to see the whole net getting /.d for a whole week. I guess that would be the most relaxing week of the past few years.

    3. Re:Slashdotting animation by tmu · · Score: 2, Informative

      And yet, the webserver's fine (famous last words).

      Did y'all notice *how* *small* that animation is? Someone else here put it together. I didn't know that animated gifs could compress that much.

    4. Re:Slashdotting animation by W1BMW · · Score: 1

      Send us your IP address and wait a few minutes.

  26. Florida??? by advocate_one · · Score: 3, Interesting

    Anyone else notice that node in Florida go down as well???

    --
    Donald 'Duck' Dunn: We had a band powerful enough to turn goat piss into gasoline.
    1. Re:Florida??? by twilight30 · · Score: 1

      Yeah, I thought that was strange too. As a poster below noted, don't forget that the animation was a representative sample of only 5% of the total outages. Which doesn't go any further in explaining it, but it stands to reason that other nodes in Florida (or general vicinity) went out as well, though obviously not to the same extent as in the Northeast US.

      --
      ========================================
      Death will come, and will have your eyes
      -- Pavese
    2. Re:Florida??? by celerityfm · · Score: 1

      Yeah I thought the map was just a map of outtages in general, not just because of the power outtage.

      --
      ...unfortunately no one can be told what The Mat^H^H^HGoatse is...they must experience it for themselves...
  27. Even FULL BACKUP POWER is no guarantee... by Anonymous Coward · · Score: 0

    You might have a network with full redundant backup power that stays powered throughout the entire blackout, yet still go down. How? If you are connected elsewhere by fiber, that signal has to be regenerated ever so often. Some of those fiber regeneration facilities have limited battery backup, or if they do have generator backups available, there could be failures. So unless you control and manage and test and maintain ALL the equipment from your facility to your end customer sites, your network could still go down. There were networks during the outage that had this very problem. Their routers and servers were powered and online with backup generators, but the telco long haul circuits one by one blinked out as remote regeneration sites lost power. Fun, isn't it!

  28. Re: Don't backbone routers have backup? by RollingThunder · · Score: 2, Insightful

    Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.

    Why would you want it all in batteries? Use the UPS to tide you over until you can fire up the gas/diesel generator. Those you can get pretty cheaply (well, compared to 300lbs of batteries) and are useful for other things too - such as going camping.

  29. Random Noise? by Anonymous Coward · · Score: 0

    What the hell happened in Tampa, FL?

    1. Re:Random Noise? by El · · Score: 1

      The router was using a voting algorithm.

      --

      "Freedom means freedom for everybody" -- Dick Cheney

    2. Re:Random Noise? by Anonymous Coward · · Score: 0

      I was asking the same thing. I live in Tampa and was on the net for most of that day at work and at home, didn't notice anything.

  30. Map of UPS battery exhaustions by BigBlockMopar · · Score: 3, Interesting

    You don't have backup power on your home LAN? Pffft, and you call yourself a geek.

    From what I know, this failure happened from New York City to Ohio in a matter of under a minute. So, I guess the dots are more representative of the average lifespan of the (formerly) fully-charged batteries in one's UPS.

    As for me, I was dead in the water. At home, it was instantaneous (I'm too cheap to buy a UPS for a site which is just for my personal amusement); at work, it was 10 minutes of standing there in the server room listening to the frantic beep-beep-beep of UPSes all around me, and then rushing around to connect low-power LCD monitors to the servers that someone else forgot to connect to the UPS's shutdown signals... (of course, this after the realization that the emergency generator is running, but not actually even connected to the servers... [grumble])

    --
    Fire and Meat. Yummy.
    1. Re:Map of UPS battery exhaustions by LauraW · · Score: 4, Insightful
      At a company I used to work for the network operations folks were supposedly bragging about how redundant the servers were, with backup power, automatic network failover, and so on. So an executive decided to test them on it: he grabbed a power cord from one of the switches and pulled it out of the wall. Oops. It turned out that one wasn't redundant. Or rather, it technically was -- there were two of them -- but the failover mechanism didn't work.

      At my last job we had similar problems. The system was heavily dependent on JMS, so there were rundandant JMS servers. Unfortunately, the first time the primary one on the production network went down under load all the client systems had to send tons of JMS messages around as part of the recovery process, which created a snowball effect that took down the secondary server and many of the other clients too. And then of course the clients started coming back up and sending out JMS messages to announce the fact.... (It turned out to be a bug in the JMS client that eventually got fixed, but it wasn't pretty while it was happening.)

      Moral: if you haven't tested the redundancy / failover / power failure mechanism, it might as well not exist.

    2. Re:Map of UPS battery exhaustions by MrDelSarto · · Score: 1

      On the topic of emergency generator stories. I used to work at a helpdesk, and arrived very early one Saturday morning to about 5 fire trucks and the HAZMAT (hazardous materials) team. Turns out a blackout the night before had triggered the generator, which had split a fuel line. Of course the fuel was above the split so pushed out a couple of thousand litres of diesel into the surrounding environment before the operator 3 floors up said "what's that smell" ...

  31. Imagine... by ChopSocky · · Score: 1, Funny

    A beowulf cluster of network blackouts... Wait... Somebody shoot me.

    --

    "Joan of Arc, up top!" - Ghandi, Clone High
  32. Canada was still alive by thexdane · · Score: 2, Funny

    i'm up in canada and when the power went out our internet conenction didn't. i know this because one of the guys from our offices out west called and asked me what was going on. tho our network did go down due to our ups running out of juice.

    the only thing that did NOT go down during the power outage up here was our telephone system. i'm not sure about our other net conenctions tho, i know my home cable connection went down, that's cause cable was out in certain areas, tho in other areas the cable was fine, our bell dsl was fine because bell's authenication servers are in quebec, who would have guessed the french were useful?

    i'm guessing that all of our networks were up, i guess the diesel generators a lot of the big data centers have installed came in handy.

  33. Script generated response by OneIsNotPrime · · Score: 0, Redundant

    Stupid MS and their NETWORK BLACKOUT. Man, Linux users never experience NETWORK BLACKOUT. What is SCO's plan?

    1. NETWORK BLACKOUT
    2. ????
    3. Profit!

    In Soviet Russia, the NETWORK BLACKOUT NETWORK BLACKOUTs you!

    _______________________

    --

    ---

    WARNING:Slashdot karma not redeemable in the afterlife.

  34. Slashdotting animation-Server smoke. by Anonymous Coward · · Score: 1, Funny

    "Would love to see an animation of a webserver being slashdotted."

    Hot enough for you?

  35. WorldCom by Anonymous Coward · · Score: 0

    WorldCom was pretty good about keeping their servers running. I was with SkyTel, Wcom's paging company, and we survived satellite outages, twin-tower collapes, and stock falls. Heh.

  36. Surfing at work by halo8 · · Score: 1

    I work for IBM and our building had an hours worth of generating power left.

    45 minutes after the power went out we were still surfing the web listing to what we could from winamp radio stations, our phones are standard call center VOIP lucent stuff.. they worked fine i was calling friends in ottawa (im in toronto) letting them know what cnn.com said.

    nice to know that if the power goes out i have to walk down 11 flights of stairs, but i can still work (in the dark, no AC) on my computer and use my phone.. gee.. i guess i know where my employeres priorities are.

    --
    The More Knowledge you have the Luckier you Get- J.R. Ewing
    1. Re:Surfing at work by Phroggy · · Score: 1

      ice to know that if the power goes out i have to walk down 11 flights of stairs, but i can still work (in the dark, no AC) on my computer and use my phone.. gee.. i guess i know where my employeres priorities are.

      Should be obvious, no?

      --
      $x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
      $x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
  37. Philadelphia? by Punchinello · · Score: 1

    Although this diagram shows an extended network outage in the Philadelphia area, I do not recall any network outages in or near Philadelphia. I have clients all around the 5 counties of southeastern Pennsylvania and none of them reported an outage.

    --

    Remember... ZG9uJ3QgZm9yZ2V0IHRvIGRyaW5rIHlvdXIgb3ZhbHRpbmU=

  38. Stories, anybody? by nighty5 · · Score: 1

    Admins (ect) from NY post your funny/heart breaking/serious stories from the outcome that was the black out.

    Stuff like:
    1. Systems that went down and stayed down.
    2. Pointy haired bosses that realised that investing in Solaris instead of those "Linux Hobby" boxes was a bad move because none of the Solaris came back to life
    3. You were just about to go-live with a product after 2 years of development, and then the power went out
    4. Everyone went home because no-one could work, except your hot looking system admin babe couldnt resist you in the data center

    ect ect

    We, from the other side of the globe are interested :)

    1. Re:Stories, anybody? by bmantz65 · · Score: 1

      The hot (blonde, curvy, 23) executive asst. and I were stuck in a dark and hot elevator for an hour by ourselves. Fill in the rest ;) Oh wait, another of my dreams.

  39. speaking of network blackouts by loconet · · Score: 5, Interesting

    14th 4:20pm - Power goes out, our building's generators for some reason dont kick in.
    14th 4:35 - Most of us decide to call it a day and go home
    14th 4:30 - I'm in my car, I realize the blackout is bad when only 2 or 3 radio stations are working and the no traffic lights are. I know it was going to be a fun drive home - thank god i live 15 mins away.
    14th 4:45 - I hear the blackout expands to parts of the States. me: OH FUCK@#$@#$
    14th 4:50 - My sister sms's me on my cell telling me her and my mom are stuck in the subway - they need help. Like I care, I have my own problems , traffic is a mess and there are hundreds of psychos out.
    14th 5:00 - I get home, look for a battery powered radio and listen in.
    14th 5:30 - I get a call from my sister - they're stuck somewhere downtown. I just wish them luck.
    14th 7:00 - I realize there are no candles in the freaking house, time to look for those puppies.
    14th 8:00 - Sister & mom arrive home. me: LOL
    14th 11:30 - I go to bed & pray the blackout lasts until the middle of the next day, that way I get an extended weekend -wohooo, back to bed
    15th 7:30am - Wake up, lights are still out - no work, home free! wohoo, back to bed.
    15th 11:30 - Receive a call from my boss, asking me where the fuck i was since they got power at the office but there is a lil "issue".
    15th 12:30m - Got to the office, problem: No Internet connection, seems one of the ISP's switches went bye bye after the black out. Our main app server is down. No power you think? Nope The colo company hasnt been paying their bills and WorldCom used the blackout to pull the plug on them. Server is up and running but no outside world connection. FUCK@#*$.
    15th 1:00pm - We think , no biggy let's use one of our other servers and restore apps and data from backup. HA! yah right - Turns out a DNS servers for our backup machines had died and the backup script had stopped working 10 days earlier. Great.
    15th 2:30 - We wait to see if Worldcom is nice enough to plug the box back in.
    15th 4:30 - Yah, it ain't happening - 20+ clients are without website and apps.
    15th 5:00pm - Boss and I drive downtown to the WorldCom building to download data physically off the Box.
    15th 5:30 - Stop for gas - HA! huge lineup.
    15th 7:30 - Get into the server room, ha! the fucking cage where our box is is locked and the key is not working. One of us climbs the cage and goes into it, runs an ethernet cable from the box to the laptop. So picture these, 4 geeks inside a server room, three sitting on the floor , one inside a cage like some wild animal. I should have brought a camera. Let the tar'in begin.
    15th 9:00 - Download is completed, our asses are sore from sitting on concrete, necks hurting, and WorldCom employee happy that we're finally leaving.
    15th 9:30 - We're downtown wondering how the fuck we're going to upload 2gig+ worth of data and source code to our spare server.
    15th 10:30 - Since there is no inet at the office and our home's cable is too fucking slow (Rogers cable sucks!) , we decide we bring out the ghetto in us. We walk up and down Yonge street asking Internet cafe's if they could lend us some bandwith!!!. Yes, you hear me right, we were begging for bandwith in internet cafes.
    15th 10:45 - We decide we're hungry, so we stop at a sushi bar. After we're done we realized it might not be a good idea to eat fish after a blackout. Fridges not working aand all. Too fuckin late.
    15th 11:00 - Found an internet cafe that will let us connect the laptop to upload.
    15th 11:30 - Realize we can't do shit since the computer is in Korean , have all Win settings locked and the guy taking care of the place has no clue.
    16th 1:30am - I'm at my boss's house uploading 2gig+ data , will take about 27 hours. Ask me if I cared about the clients at this point.
    16th 1:35 - I leave and head to my friend's place where they're having beers & bbq'in on my friend's balcony. - My weekend Begins.

    Network Blackouts? Yah they suck.

    --
    [alk]
    1. Re:speaking of network blackouts by korgull · · Score: 2, Funny

      oh man, you gotta let us know how the sushi worked out !!!

  40. Quick and Dirty LIVE UPS Recharging Ideas by BigBlockMopar · · Score: 4, Insightful

    Apparently not around my neck of the woods... I had fun doing traceroutes as the power came back up and seeing how far I could get as more and more routers along the way were returning to service.

    Yeah, same up here in Ottawa, Canada... I was awakened early on Friday morning to the sound of my servers POSTing; my power was back in under 12 hours. I was lucky. :) (Made sure to double-check that hdparm was set to spin down the drives, that and killing the A/C were my contributions to energy efficiency.)

    Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :( Still, I was up and running before connectivity in my area was restored.

    I don't have a UPS (well, I do, I got one free, but it's broken and I haven't had time to troubleshoot it - anyone got schematics for an APC Back-UPS Pro 280?), so your mileage may vary. If the UPS runs off 12V batteries, you might be able to:

    • cobble a set of binding posts onto the side of its case, in parallel with the battery, and connect them to the battery in a running car. (Essentially, "jump-start" your UPS. Start the car first.)
    • Replace the 12V gel-cel battery with a good old-fashioned car battery. Even a weak used one should run it for a lot longer, but I haven't seen how charge current is regulated when the UPS is on AC, so I don't know how well the UPS's charging circuits will tolerate it.
    • Scoop an old gas lawnmower out of the garbage (Briggs and Stratton or Techumseh 4-stroke motors are preferable and very reliable if you keep them well tuned). Fix it, and install a pulley where the blades were (usually a 3/4" or 1" keyed shaft, and you want to take a 4L belt of the required length). Cut a hole in the deck, install a bracket, and hang a GM 1-wire alternator (1975-1985 models) in there. The lawnmower's deck is ground, the plastic-insulated bolt on the back is the positive. Mount a car battery onto it, and you have a portable jump-starter and 12V generator. Good also to weld on a perch for your toolbox. (Built one for myself, works *great* in junkyards when you want to test compression or oil pressure in an old car.)

    Note that I don't know how the UPS's inverter will handle running at rated load for longer than the internal battery is capable, nor do I expect that the UPS will have much noise suppression on the battery leads - after all, batteries themselves are pretty much noise-free electrical sources and alternators are not.

    --
    Fire and Meat. Yummy.
    1. Re:Quick and Dirty LIVE UPS Recharging Ideas by balbeir · · Score: 3, Funny

      You wouldn't be that guy from the redgreen show by any chance ?

    2. Re:Quick and Dirty LIVE UPS Recharging Ideas by ahaning · · Score: 2, Funny

      I see no mention of duct tape in the grandparent's post, so it's pretty safe to say that it's not him.

      --
      Withdrawal before climax is very ineffective and those who try this are usually called "parents."
    3. Re:Quick and Dirty LIVE UPS Recharging Ideas by tconnors · · Score: 1

      Note that I don't know how the UPS's inverter will handle running at rated load for longer than the internal battery is capable

      I picked up a Sola 300 UPS for $10 with 2 dead batteries, so bought 2 12v car batteries. I killed the first UPS after about an hour of testing (at the rated 300 W load), because the main transformer got too hot and burnt out one of the windings. The transformer retained a heckload of heat for several hours.

      So I bought another $10 UPS, and this time drilled holes all through the case, and put a large fan that was fed 12V when power off, and a few volts when power on (so it would still keep the charging circuit cool -- keeps the noise down though)

      I also only run the computer, switch and other low power important things off this, and the monitors and other less important things off an inverter/charger/power-available-tester/relay circuit. This way the UPS has been able to stay up as long as the batteries hold out (at least a few hours), and the transient created by turning on the monitors don't kill the UPS when under battery (it just kills the inverter, which can be rebooted).

    4. Re:Quick and Dirty LIVE UPS Recharging Ideas by Anonymous Coward · · Score: 0

      I have a similar set-up at my home, but a little different.
      First in line (starting at the wall outlet) I have a small auto battery charger (one with auto shut-off or trickle charge), next in line are two 12 volt rideing lawn-mower batteries ($20 each new) wired in paralell, After that is a automotive power inverter 12volt to 115-120volt, and finaly my surge protector to which my tower, moniter, dual dj type cd player and mixer (I dj for a shoutcast station)are conected to this setup has kept me going for over 24 hours at times, ofcourse the moniter was turned on anly when needed. But its cheaper than an ups that last the same time and is automatic and always connected...For large systems you should use deep cycle marine batteries and possibly extra batteries and a larger charger, If you use used batteries, you should make sure to fully charge them prior to use.

    5. Re:Quick and Dirty LIVE UPS Recharging Ideas by Baron_Yam · · Score: 1

      I like the lawnmower idea, and would only add one thing to it - if you run the 12V output of the alternator through an inverter to create AC, then plug the UPS into the inverter you get line conditioning at the inverter inputs (because they're usually designed to run in a car, which has an alternator, which has the noisy DC), plus you don't have to hack open a perfectly good UPS.

      You do lose a bit of power converting AC-DC-AC-DC-AC (Mower Alternator - Car Inverter - UPS Inverter - UPS Battery - UPS Inverter), but I think it's worth it... I'm going to build one of these things as soon as I hunt down the parts.

    6. Re:Quick and Dirty LIVE UPS Recharging Ideas by nmos · · Score: 1

      I like the lawnmower idea, and would only add one thing to it - if you run the 12V output of the alternator through an inverter to create AC, then plug the UPS into the inverter you get line conditioning at the inverter inputs (because they're usually designed to run in a car, which has an alternator, which has the noisy DC), plus you don't have to hack open a perfectly good UPS.

      Except that UPSs (at least some of them) don't like the power produced by inverters. I have a customer with an alternative powered home that has been through a couple of different brands without success.

  41. Old Nodes by spineboy · · Score: 2, Funny

    Those were all the old nodes that retired and moved south to Florida.......

    Oklahomah had a node go down too..WTF? Ya got me on that one...Can't even make a joke about that..

    --
    ..........FULL STOP.
  42. Is this a good time for Power Over Ethernet???!?! by Anonymous Coward · · Score: 0

    Man, I did all I could to help...I plugged my cable modem into an outlet (imagine the look on the guy at radio shack's face when I told him I wanted to go from RG68 to 110VAC). It didn't keep it up, eh? I need to call someone and bitch..

  43. hey wait, I can still get to... by spid101 · · Score: 1

    oh wait, deamn cache.

  44. Dude.... by calebb · · Score: 1


    You've got to be kidding....

    I hope that it was someone elses negligence & not yours! I guess I'll stop complaining about the 50 l-users who ignored my emails on Jul.28 - Aug.1 warning them of the impending RPC vulnerability worm that would destroy their data.... (well, it didn't destory their data, but they did ignore my warnings & they did get the worm!)

  45. Nebraska, eh? by Trepidity · · Score: 1

    "I swear, it's the Northeastern power outage that caused the Omaha routers to go down."

  46. Re: Don't backbone routers have backup? by Anonymous Coward · · Score: 0

    Funnily enough, I calibrated my APC Smart-UPS 1400RMXL on the day of the power outage. It has 4 Costco 12V deep cycle batteries hooked up in series using a Trimetric battery monitor from Bogart Engineering (popular with the solar folks) and shunt so I can watch the amp-hours/voltage/etc. come and go. I took out the stock batteries (4 12V 17Ah VRLA batteries wired in series). The Costco batteries give me 48VDC 110Ah and with about 45% load I went 8 hours and only reached ~24% capacity as indicated on the Trimetric. The Costco batteries are a great deal -- assuming you have a good inverter that can run for hours at a time. Only $240 or so for all four batteries. So far I haven't needed to water the batteries, though I expect that I will need to do that eventually due to the high float voltage that the APC uses (ok for VRLA, flooded batteries are slightly less tolerant).

    Oh yes, the magic connnector that all of the APC units use is from Anderson Power. You can get those at solar places too or just sacrifice an existing APC battery so that you can plug in your own batteries to the external battery connector on the back of your APC 'XL' model.

    word to the wise: do NOT use a cheap UPS and add more battery capabity to it and expect to get hours of runtime. The inverter on nearly all UPS models isn't designed to run for more than 10-15 minutes!

    I also use lcd-nut from http://www.webbastards.com/projects/ to keep track of the UPS from upstairs (the batteries are too heavy for the floor). A cool little utility for your LCD screen.

    I also have a Generac Beacon whole-house generator with Sola ferroresonant transformers in case of extended blackouts. California power isn't quite that bad (though I've noticed that the *quality* of the utility power here is getting worse, requiring me to invest in some isolation transformers (just Oneac units, not ferroresonant) and better TVSS units).

    Another rant: APC UPS units have meager surge protection. Get a better surge protector (ideally without any MOVs) for the UPS -- you'll need it!

  47. MIDS by Anonymous Coward · · Score: 1, Informative
  48. Re: Don't backbone routers have backup? by evilviper · · Score: 1
    my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours

    I know we are talking about places like New York, but does nobody have a car??? That beauty is probably 100x the power of your UPS battery, and it comes connected to a mobile, gasoline-powered 12V generator, on wheels.

    If you don't like doing some wiring yourself (to the UPS), 300watt power inverters are under $50, which is more than enough to keep all your lights on, and a TV, radio, possibly computer+monitor, etc.
    --
    Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
  49. Re: Don't backbone routers have backup? by Phroggy · · Score: 2, Insightful

    Of course, I had to wait for MY neighbourhood's power to come back up as my UPS died about 4.5 hours into the blackout; my wife won't let me add the additional 300lbs of batteries required to last a full 24 hours. :(

    You want fuel cells, not batteries.

    --
    $x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
    $x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
  50. oh did this suck by el_guapo · · Score: 1

    i work for a large comm company (i won't mention at&t by name:-), and this was, um, A Bad Thing(tm)

    --
    mas cerveza, por favor politically incorrect stu
  51. KVM Switch Loathing, IIS in Ironic Places by BigBlockMopar · · Score: 4, Interesting

    You've got to be kidding....

    No.

    I hope that it was someone elses negligence & not yours!

    Yes. In fact, I did learn something: there's a very good reason that I hate KVM switches.

    Solution (probably falling on deaf ears, but I shall try anyway):

    • Keyboard and monitor on all servers. Monitors should be LCD to save UPS batteries. This way, if the UPS shutdown sequence fails, it can be done manually. (A distressing number of these machines run Redmond's cancer, so having one central "shutdown console" is less practical.)
    • Serial shutdown outputs from UPS will be "broadcast" to all hosts connected to that UPS. Will require making a custom cable with a few MAX232 line-driver ICs and hang it off a machine's PS/2 port for power.
    • Backup generator is rated for 90kW and showed only about 10kW load running emergency lights around the building. Time to tap out that extra 80kW of capacity. Why this wasn't checked before is an absolute mystery to me.
    • Reorganize server interconnections. The network switch for the users' LAN wasn't on UPS (but neither are the users, so it didn't seem like a big deal). However, over the years, some stuff has come to rely on mapped drives... fortunately in this case, we're not so lucky to have hard-mounted NFS anywhere. :)
    • Have occasional practice power outages before long weekends. Who doesn't test their UPS by unplugging it from time to time? [grin]
    I guess I'll stop complaining about the 50 l-users who ignored my emails on Jul.28 - Aug.1 warning them of the impending RPC vulnerability worm that would destroy their data.... (well, it didn't destory their data, but they did ignore my warnings & they did get the worm!)

    Heheh... Yeah, I know the type.

    Along those lines, how's this for bitter irony? The Canadian government's "Office of Critical Infrastructure Protection and Emergency Preparedness" runs IIS on Windows 2000. (Note also the subjects of a couple of the bulletins on their site...) Somehow, this reminds me of doctors who smoke, or mechanics who don't change their oil.

    My tax dollars at work. [sigh]

    --
    Fire and Meat. Yummy.
    1. Re:KVM Switch Loathing, IIS in Ironic Places by no+soup+for+you · · Score: 1
      very good reason that I hate KVM switches... Keyboard and monitor on all servers. Monitors should be LCD to save UPS batteries. This way, if the UPS shutdown sequence fails, it can be done manually

      I don't get it. Why do you hate KVM switches? Just connect an LCD to the KVM.

      --
      If you blog it...
    2. Re:KVM Switch Loathing, IIS in Ironic Places by alphax45 · · Score: 1

      actually the bulletins appear to be a WARNING to the general public. Just because a site is on IIS doesn't mean it immedatly crap. Unless you know for 100% that the server is NOT patched and resionably secure; none of us can make an option either way.
      From the site:
      Note to Readers OCIPEP's role is to provide national leadership on critical infrastructure protection and effective emergency management. To fulfill this role, OCIPEP collects information related to cyber and physical threats to, and incidents involving, Canadian critical infrastructure (CI). This information allows OCIPEP to monitor and analyse threats to our CI and to issue alerts, advisories and other information products to our partners. To report CI threats or incidents to OCIPEP, please contact our operations coordination centre at (613) 991-7000 or at opscen@ocipep-bpiepc.gc.ca

      --
      K Man
    3. Re:KVM Switch Loathing, IIS in Ironic Places by BigBlockMopar · · Score: 1

      actually the bulletins appear to be a WARNING to the general public. Just because a site is on IIS doesn't mean it immedatly crap. Unless you know for 100% that the server is NOT patched and resionably secure; none of us can make an option either way.

      Having said that, I would imagine that the seismology agencies in Japan are in seismically correct buildings... Isn't that exactly the same? It's not like they control the threats to the infrastructure, all they can do is provide information to civilians and government agencies.

      Isn't using IIS, while there are still more secure and reliable alternatives, exactly the same?

      Back to the Japanese: what with the paper houses with the stone roofs on active fault lines? And you go out and buy a Honda, presumably trusting them to design your brakes? I don't get it.

      --
      Fire and Meat. Yummy.
    4. Re:KVM Switch Loathing, IIS in Ironic Places by Anonymous Coward · · Score: 0

      It reminds _me_ of what I'm more worried about--mechanics who smoke and doctors who improperly use oil....

      Fire...BAD! Burning sensation...BAD!

  52. Snowblower Couch Racing by BigBlockMopar · · Score: 4, Funny

    You wouldn't be that guy from the redgreen show by any chance ?

    Steve Smith? No. But we are on first name basis (no kidding!) and I wear lots of flannel. He once did something on the show which really reminded me of snowblower couch races with friends, and I can't remember if I told him about it or not...

    (Snowblower couch racing? You scoop sofas, mattresses and box-springs from the garbage and store them in your backyard. You buy beer and sharpen the ice-cutters on your snowblower. Then you invite over some friends with their own snowblowers and see who can demolish the upholstered furniture the fastest. Truly a good reason to own a snowblower in a warm climate.)

    --
    Fire and Meat. Yummy.
  53. Re: Don't backbone routers have backup? by mr_walrus · · Score: 1

    > 300watt power inverters are under $50,
    > which is more than enough to keep all your lights on,
    > and a TV, radio, possibly computer+monitor, etc.

    assuming the sum of all the above devices and lights is no more
    than about five 60watt light bulbs...

    i bet my monitor alone sucks 300watts (2.5amps?)

    -

  54. Re: Don't backbone routers have backup? by andrewski · · Score: 1

    I'm sorry, but a generator is the last thing you take camping. RVing, maybe.

  55. Re:redundancy and surge capacity by rediguana · · Score: 2, Informative

    I thought that the internet was supposed to be redundant (like power)... i thought they learned on sept 11, that there wasnt enough capacity on the cell networks

    We had a presentation last night from Tom O'Rourke on the critical infrastructure affected by Sept 11 2001.

    The main point to come out of it is that most critical infrastructure is commercially run. They are designed to run and handle typical peak loading.

    And there is a difference between redundancy and surge capacity. Redundancy allows you to continue operating at a reduced or normal level. Surge capacity allows you to operate at increased levels due to an unexpected event - such as handling increased mobile phone demand by increasing available spectrum bandwidth as was also done after Sept 11 2001.

    The problem that you have in these sort of failures is that not only have you lost capacity, but you see increased demand at the same time. One slide he had detailed cell phones in NYC. Typical block rate (no tone) is about 4% - ie one in 25 calls you can't make. After losing all the capacity after the collapse, combined with everyone wanting to talk on their phones, the blocked call rate jumped to 92%. The expense required handle these sort of extreme events cannot be justified.

    Utilities can handle ordinary spikes in their systems, but it is not economic to design surge capacity into most systems.

    You'll find this problem across a number of sectors... telecommunications, power, and hospital beds tend to provide the best examples. You'd be suprised at how few hospital beds are generally available at any given point in time.

    If we had a mass casualty event in New Zealand, it is quite possible we would be sending victims to hospitals in Australia because we don't have the surge capacity.

    Cheers Gav

  56. Other side of the border? by artrr · · Score: 1

    Yah it was "quiet" all right. If the ISP's were running well then thats nice! We were completely
    in the dark just as much as New York, and continue
    to conserve on a daily basis for fear of rotating
    black outs. Funny i dont hear the conservation message on the "other side".....

    1. Re:Other side of the border? by luciuskwok · · Score: 1

      In the US, our power generation is fully recovered, unlike Canada's. It's also cool enough not to need AC. Hence, no need to tell people to "conserve." Read the news story.

  57. Re: Don't backbone routers have backup? by RollingThunder · · Score: 1

    Depends what you do while camping.

    Me and 1200 of my friends work on a castle for our SCA event in Clinton, BC. Power tools are a must. :)

  58. Atari / Missile Command by AgentPhunk · · Score: 3, Funny
    Who else here (who's old enough) thought of Atari's old Missle Command when you watched that animation?

    anyone? Anyone? Bueller?

    (sigh)

  59. Re: Don't backbone routers have backup? by ZorinLynx · · Score: 2, Funny

    Whereas a 200HP car engine can put out nearly 150KW of power at peak, which is enough to run several suburban homes, the alternator can probably only put out around 1KW max. Be careful not to overload and burn it out, costing you more money than a couple extra batteries would have. }:)

    Cars are designed to haul themselves and your ass around, not keep your little server farm running.

  60. you know... by mschoolbus · · Score: 1

    And I was starting to wonder why I see all those "I love NY" t-shirts...

    Even here in Minnesota...

  61. Re: Don't backbone routers have backup? by arkane1234 · · Score: 1

    I don't know about you, but if the power gets that bad off, I'm not about to rig my car up to my house. I have much better things to do than use electicity in those situations :)

    But, all in all, the alternator would be under some massive load with anything more than a computer setup and a television (*maybe*) connected to the battery. It's just too much juice... especially for some of the not so new cars that have factory alternators that really suck.

    --
    -- This space for lease, low setup fee, inquire within!
  62. Re: Don't backbone routers have backup? by evilviper · · Score: 1
    i bet my monitor alone sucks 300watts (2.5amps?)

    You'd loose that bet. In fact, you aren't even close.
    --
    Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
  63. Re: Don't backbone routers have backup? by evilviper · · Score: 1
    Of course... I'm sure nobody would want to be able to watch TV, listen to the radio, or turn on the lights.

    the alternator would be under some massive load with anything more than a computer setup and a television

    I wasn't suggesting having a computer, radio, and TV on, I was suggesting lights, and one of those. I think that would be obvious by the "300watt inverter" statement.
    --
    Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
  64. Re: Don't backbone routers have backup? by evilviper · · Score: 1
    Be careful not to overload and burn it out, costing you more money than a couple extra batteries would have. }:)

    Everyone seems to read their own meaning into what I say...

    I wasn't even suggesting hooking up your car to a UPS, but hooking up the car's BATTERY, and when that runs low, recharging it with your car... No matter how much load you have comming off the UPS/inverter, it can't possibly damage your alternator.

    Cars are designed to haul themselves and your ass around, not keep your little server farm running.

    And your computer is designed to run Windows, not Linux! What the hell difference does intent make? If it works for the job, use it for the job.
    --
    Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
  65. Re: Don't backbone routers have backup? by evilviper · · Score: 1

    Not only is your monitor more like 50watts, but I never suggested hooking them all up... Imagine those commas as slashes, and you'll get the idea.

    --
    Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
  66. Re: Don't backbone routers have backup? by andrewski · · Score: 1

    That's more of a festival than camping. At least in my book.

    The biggest power tools in my pack are my Dragonfly stove, my Katadyn filter, and my .357 magnum.

  67. W32.Blaster the cause of the blackout? by hyperventilate · · Score: 2, Interesting

    Sender: jm
    To: dave

    Hi Dave -- for IP.

    There's an article from Heise Security in Germany at [1], which raises
    some interesting questions about whether W32.Blaster could be to blame for
    the blackout. Some translated points are at [2] -- quote: ... it becomes a bit more likely if one considers what the authors of
    that article found out:

    - The Niagara Mohawk power grid which seemed to got overloaded first
    is owned by National Grid USA.
    - National Grid is listed as an important customer of Northern
    Dynamic who call themselves the "OPC Experts".
    - OPC is an acronym for OLE for Process Control and is used for
    communications between control systems.
    - OPC is based on DCOM, exactly that Windows technology attacked by
    W32.Blaster.
    - One symptom of a W32.Blaster attack is that a crashing DCOM
    service (not only under Windows), often taking down the whole
    server.

    One usage of OPC is the coupling of so-called SCADA (Supervisory Control
    and Data Acquisition) systems. Among other things is SCADA used in power
    plants and grids to exchange data between some central instance and
    external measuring units. And for some reason did the monitoring system
    which should prevent snowball effects like the one on thursday from
    happening.

    So the questions the authors of the article have are:

    - For which processes does National Grid utilise OPC?
    - Were there any problems regarding OPC when the power went down?
    - If yes, were they related to W32.Blaster?

    1. http://www.heise.de/newsticker/data/ju-15.08.03-00 1/
    2. http://msquadrat.de/archive/03/08/16/02

    --j.

    1. Re:W32.Blaster the cause of the blackout? by MickLinux · · Score: 1

      In line with that, it seems that one of the first sites to go down, is right along the Michigan/Canada border. It's unclear whether it goes down before or after the power outage. Now, I'm willing to bet that 1300MW flows from the Canada side of Niagra falls into Detroit.

      And if that 1300 MW can't get there by the direct route, why then the automatic system is going to reroute it around Lake Erie, resulting in a sudden swap of direction as the current goes the other way. That's what is known to have taken down the whole system.

      So this very well could be a computer problem / virus problem.

      --
      Correct Horse Battery Staple: 72 bits of entropy. Enter "Correct H" into google. When it generates the phrase, that's
  68. Perhaps... by carrier+lost · · Score: 3, Funny

    ...it's time to revisit AC power over TCP/IP.

    MjM

    Groovy. Gear. Mod.

    1. Re:Perhaps... by cranos · · Score: 1

      Yes of course, harness the energy generated by a million AC's all trolling on ./ and we could power the world for a thousand years. Just think what we could do with the Usenet posters!

  69. Map Reuse..... by Anonymous Coward · · Score: 0

    Keep this map around ... in a couple years you can reuse it to show how the US tech jobs have all been taken over by Indians .... dots, not feathers...

  70. Why Boston? by cgleba · · Score: 3, Interesting

    Why was there an outage in Boston, MA and Springfield, MA -- Massachusetts did not loose any power?!

  71. Satellite View of the Outage by Radical+Rad · · Score: 2, Interesting
    It is interesting to see how localized some of the outage was--networks in New York state right up to the Vermont border go dark while everything on the other side of the border is quiet. New York City obviously gets clobbered."

    Here is a nice satellite pic comparison of the Northeast before and during the outage from Natural Hazards

  72. Your network redundancy... by redolsnake · · Score: 0

    doesn't mean a damn thing if your telco provider doesn't have power to keep your frame relay circuits up...

    One of the partners of my company had power shortly after the first blip, but the frame is still bouncing up and down today.

  73. Deja vu all over again or just predicted? by outanowhere · · Score: 1

    The graphic makes it appear as if the network outage propagated outside the blackout area.

    Reminds me of a white paper with just such a scenario published somewhere around the time of the Morris Worm.

    The paper warned that the entire Arpanet would come down if such outages started to propagate outside an affected area.

    Perhaps too much trust is being put into tcp/ip's ability to route around failures?

  74. BS by Anonymous Coward · · Score: 0

    There was never a blackout in the Mpls area. On top of that, I was still able to daytrade during the afterhours session when trading was resumed at about 5:30PM.

  75. Don't you mean... by Captain_Chaos · · Score: 1

    Who else here (who's old enough) thought of Atari's old Missle Command when you watched that animation?

    anyone? Anyone? Bueller?

    Don't you mean: Lightman?

  76. Canada? by lsoth · · Score: 1

    What's with not including any Canadian (Ontario) networks, since we were blacked-out as well! Did everyone figure that without power to keep our igloo's cold, they'd melt and we wouldn't have any networks to monitor??

    --
    ... [Insert decent Sig] ...
  77. Canadian Internet Runs Best by chargen · · Score: 1

    Well, since no Canadian routers show on that map, I have to assume that all Canadian links were unaffected by the blackout!

    Just goes to show the value of buying equipment that runs on snow!

    -Pete

  78. Re: I have to get out more... by Baron_Yam · · Score: 1

    Well, to be fair, government officials were all over the radio telling us NOT to go out at the time, since there were no traffic lights, street lights, gas stations, etc up and running.

    Honestly, traceroutes are not my everyday source of entertainment!

  79. Bad case of chicken pox?? by Grizzlysmit · · Score: 1

    Thats the worst case of chicken pox I've ever seen

    --
    in my life God comes first.... but Linux is pretty high after that :-D
    Francis Smit
  80. Re: Don't backbone routers have backup? by arkane1234 · · Score: 1

    A coleman lantern is a little cheaper in that case....

    would light the room as bright as a normal household light.

    That, along with a crank-powered radio, and your set. All part of the usual household "emergency plan".

    --
    -- This space for lease, low setup fee, inquire within!
  81. Problems with auto electric generation by chainsaw1 · · Score: 2, Informative

    Altenators in new cars aren't really good for this:

    An altenator is different from a generator in that the supplied field is electromagnetic...it depends on another voltage source. This source is supplied (initially) by the battery or residual magnetic field when you start up the car, and is varied by the regulator. When your car needs recharging, the regulator ups the current or voltage in the altenators field wire, and your car cranks out more volts and recharges the battery. When the battery is mostly charged the regulator trims the field down so the alternator produces about 13V.

    This has the advantage of not robbing your engine of power and gas milage when the battery has a full charge. However, to use it as a generator, you're going to have to hard wire the field to a constant (and not to the altenator output-- this creates a infinite voltage loop that will kill the altenator).

    The reason new car altenators are not ideal is because the voltage regulator is internal to the altenator. This was done for simplicity and to make sure the regulator was always grounded, but it really complicates what we're trying to do. Older cars (until late 70's, early 80's) have external regulators and cars older than that (early to mid 60's ?) had generators (field was a permenant magnet).

    Also, you do NOT want to bypass the car battery. It is an excellent power conditioner for the not-so-even generation of the altenator/generator

    --
    - Sig
    1. Re:Problems with auto electric generation by evilviper · · Score: 1

      As I've said multiple times, I was no recomending using your alternator directly, but was recomending using your car to charge a car battery, and then hook that battery up to an inverter. Everyone should have at least 2 car batteries I hope.

      --
      Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
  82. Re: Don't backbone routers have backup? by evilviper · · Score: 1
    A coleman lantern is a little cheaper in that case....

    Not much cheaper, doesn't allow you to run other devices, and is a lot of work for something you'll be using less than once every 5 years.

    would light the room as bright as a normal household light.

    Indeed, mine is very powerful, but it doesn't use normal gasoline, and most stores where you can buy fuel for it are going to be shut-down in a power outage.

    I like my coleman lantern very much, but I only use it for camping, because you have much more flexibility in your home if you use electricity (and gasoline to generate the electricity).

    That, along with a crank-powered radio, and your set.

    An inverter or UPS is a LOT cheaper than a good crank-powered radio.
    --
    Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
  83. Re: Don't backbone routers have backup? by mr_walrus · · Score: 1

    from the datasheet for viewsonic p815
    power consuumption: 180w (typical)

    okay, not 300, but more than 50

  84. Re: Don't backbone routers have backup? by evilviper · · Score: 1

    That is one incredibly power-hungry monitor! The 50watt figure (actually about 54w) is straight from testing my 19" monitor.

    Interesting though... I was thinking about geting a Viewsonic monitor (because of the 3-year warranty). This may be a very good reason not to.

    --
    Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
  85. Re: Don't backbone routers have backup? by arkane1234 · · Score: 1

    I was gonna just skim over this, but I have to point out a few things:

    Not much cheaper, doesn't allow you to run other devices, and is a lot of work for something you'll be using less than once every 5 years.

    eh? I picked mine up for like 30 bucks or so, probably less considering I'm cheap. It's not alot of work to use, just screw on the propane tank (5 seconds of work), light a match, turn the handle and boom.. light. Admittedly you can't plug electrical appliances in, but then again our goal isn't to emulate electricity, it's to exist confortably.

    Indeed, mine is very powerful, but it doesn't use normal gasoline, and most stores where you can buy fuel for it are going to be shut-down in a power outage.

    You're right, it uses propane... and you can pick up containers of this quite easily beforehand and store it, unlike gasoline. I personally have 5 containers ready. Each container lasts a very long time, as you know.


    I like my coleman lantern very much, but I only use it for camping, because you have much more flexibility in your home if you use electricity (and gasoline to generate the electricity).


    If you go the full route and get electicity throughout your house while the grid is down, then yes, by all means it's far more flexible to use electricity. But, we're not talking about having electicity flowing through the house, we're talking about the fundamental difference between being prepared and able to live comfortably autonomously, versus plugging an inverter to your cars electrical system and risk overheating due to sitting and running forever. (and having a very limited amount of items plugged in) It's far better to look at alternative methods than to try to emulate when the outcome isn't the same as desired. In other words: running your car to power a radio, 2 lights, and a television. (let's say you don't have DirecTV, you have cable) The cable company may well be down from the outage, and the local stations would more than likely be effected.... so television is only an option if you have Satellite TV and it's a local blackout.
    Now, your down to radio and lights. Using alternative methods, you can have light and radio all running on their own little self-contained power source without having a carbon monoxide pipe right outside your door.
    Need a more stable lifestyle during an outage, then we're outside of the context of our conversation. You'd definately go with a generator at that point. We've started by talking about cheap alternatives to going gridless and somehow a cars electical system came into it, when alternatives exist that don't destroy the family transportation. Gas can only be pumped when there is electicity, also......
    Please tell me your not one of the people on here that spout off about SUV gas usage? :)

    An inverter or UPS is a LOT cheaper than a good crank-powered radio.

    An inverter or UPS costs less than 10-20 bucks?
    http://cgi.ebay.com/ws/eBayISAPI.dll?ViewI tem&item =3042618198&category=294
    Or 30 in this case:
    http://21st-century-goods.com/Merchant2/mer chant.m v?Screen=PROD&Store_Code=21st&Product_Code=SM889&C ategory_Code=SPTcellular

    Get a cell phone charger while your at it:
    http://21st-century-goods.com/Merchant2/merch ant.m v?Screen=PROD&Store_Code=21st&Product_Code=AHPG&Ca tegory_Code=SPTcellular

    I'm not trying to say your wrong, it's obvious we have 2 different ideas of comfort during an electrical outage. I personally would *prefer* the electricity off. (yes, even the computer, I'd go through withdrawals but it'd be worth it)
    The only bad part about that is our food storage would be a little more primitive :)

    --
    -- This space for lease, low setup fee, inquire within!