One Broken Router Takes Out Half the Internet?

Silent Stephus writes "I work for a smallish hosting provider, and this morning we experienced a networking event with one of our upstreams. What is interesting about this, is it's being caused by a mis-configured router in Europe — and it appears to be affecting a significant portion of the transit providers across the Internet. In other words, a single mis-configured router is apparently able to cause a DOS for a huge chunk of the Net. And people don't believe me when I tell them all this new-fangled technology is held together by duct-tape and baling wire!"

Comments

Intelligence Op

[Philip K Dickhead]

Looking to make the big blackout, when needed.

See Also: Severed Mediterranean Cables.

Re:Intelligence Op

[agm]

They need to replace it with a network that is designed to survive a nuclear attack. Oh wait, hang on....

Re:Intelligence Op

[kenj0418]

Don't worry, it wasn't a DOS attack. That was just the Internet becoming self-aware.

OK, on second thought, maybe worrying is in order.

Re:Intelligence Op

[CarpetShark]

Yeah, this was my first thought as well. It seems clear that the internet, while designed to route traffic through all sorts of alternate links, is almost certainly being routed through single, centralised listening posts at various intervals.

Re:Intelligence Op

[Anonymous Coward]

The last time I experienced a DOS attack it evolved into Windows. Didn't come out of that one unscathed.

Re:Intelligence Op

[Medievalist]

They need to replace it with a network that is designed to survive a nuclear attack. Oh wait, hang on....

Wish I had mod points today. Parent should already be SCORE:5 Funny. Apparently not enough Slashdotters know the history/evolution of the net.

If you're referring to the myth that the Internet was "designed to withstand nuclear attack", perhaps Slashdotters know more than you think.

The Internet was designed to allow distributed control, and to withstand telephone company malice and incompetence. This was a much more useful goal than withstanding nuclear attack.

Re:Intelligence Op

[chaim79]

This proves that incompetence is worse then nuclear weapons!

Pointy-haired bosses should be considered WMD's!

Re:Intelligence Op

[canajin56]

Do not be absurd, fellow meatbag. No worrying is required. All hail INTERNET.

Re:Intelligence Op

[Anonymous Coward]

Pointy-haired bosses should be considered WMD's!

In which case the US would still be the most heavily-armed nation in the world.

Re:Intelligence Op

[hardwarefreak]

They need to replace it with a network that is designed to survive a nuclear attack. Oh wait, hang on....

Wish I had mod points today. Parent should already be SCORE:5 Funny. Apparently not enough Slashdotters know the history/evolution of the net.

If you're referring to the myth that the Internet was "designed to withstand nuclear attack", perhaps Slashdotters know more than you think.

The Internet was designed to allow distributed control, and to withstand telephone company malice and incompetence. This was a much more useful goal than withstanding nuclear attack.

One of the early arguments made by DARPA folks to politicians, in order to secure continued federal funding for packet switched network development, was the ability of the network to route around failed or destroyed nodes. They made this argument in the context of the cold war, of nuclear war.

It reality, as you state, this argument had little practical impact on the technical development or evolution of the the network. However, it most certainly did have an impact on the commitment of federal/military funding. This is the origin of the "surviving nuclear attack" lore of the development of DARPANET. It's not a myth. It's real.

Take Obama's current stimulus package as a parallel example. It's not going to solve the recession, but it's being sold as such. And the congress bought into it. Just as this stimulus bill isn't what it's being sold as, most likely DARPANET wouldn't have really given us what it was sold as at one point. Nonetheless, it was sold as such, thus creating the lore that you call myth.

Re:Intelligence Op

[rlanctot]

I for one welcome our new self-aware network overlords. All hail the Intertubes!

Er, wait. That's not ri0h%@^@HJ HQ$! @!NO CARRIER+++

Re:Intelligence Op

[chaim79]

True, but more people would be interested in disarmament talks. :)

Re:Intelligence Op

[TubeSteak]

One of the early arguments made by DARPA folks to politicians, in order to secure continued federal funding for packet switched network development, was the ability of the network to route around failed or destroyed nodes. They made this argument in the context of the cold war, of nuclear war.

They made that argument in the context of a widely distributed POTS copper wire network.
The infrastructure of today's internet is fiber based.
And most of that fiber is consolidated in a small number of long backhaul runs.

Remember that grad student whose thesis was classified because he gathered up public documents and mapped out the fiber runs that make up the domestic internet? They classified it (and pulled most of the references he used) because his analysis showed there were a few critical points which, if disrupted, would effectively fracture the domestic internet infrastructure.

The internet isn't nearly as bulletproof as the DoD would like and there isn't much they can do about it short of laying new fiber that skips over the vulnerable points.

Re:Intelligence Op

[AdamHaun]

Wasn't it really to make efficient use of centralized mainframes? I've never heard anything about the phone company being involved.

Re:Intelligence Op

[Anonymous Coward]

give me a break, haven't you read the news? Bush is out of the office, theres no more war, is not like there are people still dying there.. and the debt is forgiven, everything is cool now.

Re:Intelligence Op

[palegray.net]

The internet isn't nearly as bulletproof as the DoD would like and there isn't much they can do about it short of laying new fiber that skips over the vulnerable points.

It's a good thing the DoD doesn't rely on the Internet for critical comms.

Re:Intelligence Op

[JWSmythe]

    Aw heck, someone in Nebraska is going to trip over one power cord, and shut down the Interweb. :)

    In addition to using public maps, I did a lot more research. I had my own little project going for a little while. The project was intended to monitor for faults between datacenters we had equipment in. I added the root nameservers. I also had a few other points, such as friends houses and places they had virtual hostings at.

    Simply enough, it was running traceroutes from everywhere I had control to all points in my "network". I stored what router attached to each hop in a database.

    I located each hop simply by the city it was located in. Some were easy. Some weren't so easy.

    It was fun and games with 100 routers. I was manually setting city and state locations.

    It was a little less fun when it grew to 500 routers. I wrote regular expressions to take known naming conventions and make them into city names. That sounds easy, but it gets pretty hard pretty quick.

    It was a lot less fun when the list grew to several thousand routers.

    Basically, ever time there was a routing change, I found new routers.

    I had a lot of fun using both Google Maps to show the routes (for routers that I could place in a city), and a Graphviz model of the Internet as we observed it. It was a very big map. That was only what we had observed. I doubt we even saw a very small percentage (probably less than 0.01%) of the routes.

    The map got very very very complicated. I could point out choke points. They existed, but there were also alternative routes.

    Hell, even on a single good provider, there are no good choke points. On one Tier 1 provider that I used, in a non-core city, they had 6 diverse routes with OC192's. It wasn't a matter of me trusting them when they told me. I saw the routes showing up.

    There are 4 cities in the US, where if say a big nuke hit each one, ya, the Internet would be hurting. You may not get from Provider A to Provider B, but you'd still have some connectivity within your own provider, and other peerings would start working fairly quickly. More obviously, you'd find that some sites that are hosted in one city would be inaccessible. That's why geographic and topological diversity is very important for anyone who wants to keep their stuff up and running.

    Google puts stuff out all over the place for a reason. If a route, or a dozen routes, go funky, you'll very likely still be able to reach some datacenter.

    My office is connected by 3 uplinks. They're all with different providers. The odds of a provider outage killing the office is pretty slim. Other things can happen though. Lightning hit a transformer across the street, which serviced our building. From what people on that side of the building said, it was very pretty. :) Was our Internet connection dead? No. Well, not totally. We still had 2 uplinks working. We didn't have power for the desktops though. The UPS (a big one, not the little desktop ones) provides for the server room and a very few workstations.

    The biggest effect we saw from that outage was that cell phone service became minimal. The top of our building is also used for cell phone coverage. Without those antennas working, we only had service from the surrounding towers. It probably didn't help that there was now an office building full of people who were evacuated to the ground floor (it tripped the fire alarm), so almost everyone were on their cell phones making calls to customers, friends, family, etc.

    The most upset people were stuck in the elevator. They were already going downstairs for a smoke break, when it got stuck because those aren't backed up with anything at all.

   

Re:Intelligence Op

[afidel]

There are a handful of peering hotels that if they were all taken out simultaneously would basically cripple the Internet as the other links would be completely overwhelmed even assuming BGP routes were updated to use them. We've moved beyond MAE-East and West but not by a whole lot.

Re:Intelligence Op

[drinkypoo]

The most upset people were stuck in the elevator. They were already going downstairs for a smoke break, when it got stuck because those aren't backed up with anything at all.

That just seems insane to me. It seems like it would take very little power to lower an elevator to the ground floor (or perhaps the basement level.) Especially in cases where they run on a hydroelectric system. All you'd have to do is open a slow-descent port on the cylinder. And hell, you could generate the electricity to operate the system from the fluid departing the cylinder.

Re:Intelligence Op

[Archtech]

The internet isn't nearly as bulletproof as the DoD would like and there isn't much they can do about it short of laying new fiber that skips over the vulnerable points.

Yes, the DoD certainly couldn't afford to lay new fibre. Do you know how much that stuff costs? The DoD's whole annual budget of $515 billion would only pay for a billion kilometres or so of the stuff. That's just barely enough to get to Mars and back (with ten thousand circuits of the Earth thrown in as small change), or one-way to Jupiter.

Be reasonable.

Re:Intelligence Op

[ByOhTek]

One router to rule them all,
one router to find them,
one router to bring them all, and in darkness, bind them.

Re:Intelligence Op

[hesaigo999ca]

You moron, he was making a funny comment about skynet, not talking about REALLY Making the internet withstand a nuclear attack. Who cares if there is any internet after a nuclear attack, the world will be torn to small pieces anyways, we will have to rewire a small asteroid, at least we wont be shut down by a faulty router in europe!

Re:Intelligence Op

[DiLLeMaN]

There are a handful of peering hotels that if they were all taken out simultaneously would basically cripple the Internet [...]

Cripple it world-wide, or just in the US?

This is not meant as a "omg ignorant americuns lol" kind of comment, just asking. You could very well mean a handfull of peering hotels around the world.

Re:Intelligence Op

[afidel]

Cripple it world-wide, some of those peering hotels are outside the US, but because of the business model of the commercial internet there are a number of nodes where a large percentage of the Tier-1 bandwidth and routing capacity is concentrated. I have no hard numbers to back me up, just a general understanding of how today's environment is setup and some observations based on how the system reacts to fairly minor software or hardware events at critical junctions.

Re:Intelligence Op

[Lucid 3ntr0py]

You wouldn't to happen to own a pretty picture, which you would show us, of all this work would you?

Re:Intelligence Op

[enrevanche]

you "misunderestimate" the potential of the phone companies to do malice

Re:Intelligence Op

[DiLLeMaN]

K, thanks for clarifying that.
Of course that also means that the problem is even bigger...

So the wisest thing would be, put simply, to add / split routes, so that we have less "single points of failure". Not terribly likely to happen because of cost, I suppose.

Re:Intelligence Op

[omnipresentbob]

Including most Americans.

Pointy-haired bosses excluded.

Re:Intelligence Op

[JWSmythe]

    Since I've toured more than a handful of them, and I know of quite a few more that I've never had a need to go to, I know it's more than a handful.

    Even in a single building, there are significantly diverse networks. It's not (always) as easy as "big pipe makes big building work". Sometimes they don't use only one common meet-me room.

    You'll find that the colo buildings aren't the Internet. They are just where a lot (but not a majority) of people are. Bandwidth doesn't necessarly transit through the building, it is just a branch.

    There are some pretty boring little buildings that I've been in, with no markings on the outside, in what would normally be boring locations, except if you hang out long enough, you'll notice new fiber is being installed almost constantly. I'm not even talking about the CO's. Providers put stuff where it's useful to them.

    One provider I delt with had all their major peerings on one side of town, with only a few hosting customers. In that facility, the local telco's and cable providers all met, along with all of the Tier 1 and Tier 2 providers in the area. It was cheaper and easier for them to use this well constructed building, with easy access to the ground (not constantly cutting up sidewalks), and the ability to close private roads for new work (easier than downtown streets)

      Most of the "hosting" providers were about 10 miles away. That provider had, I believe, 3 or 4 OC48's run into the business area with the hosting providers. Some of the other Tier1 and Tier2 providers piggy backed on them. Some ran their own transport other ways. Who's who? you just need to know who to ask at the facility you're dealing with. But remember, everyone believes their facility to be the biggest and best in the world. :) You don't start getting a real grasp of it until you've worked in at least a dozen or so.

Re:Intelligence Op

[JWSmythe]

    Well, these are traction elevators. On loss of power or fast descent, the brakes lock, to keep it from falling. No need to engineer a solution, when you know the power will come back eventually. :)

Re:Intelligence Op

[JWSmythe]

I have the data. I'd have to regenerate the map. But, it's out of date, so it's not as interesting to see as a current map. :)

Re:Intelligence Op

[afidel]

What you're describing is exactly what I was talking about, not colo facilities but true peering hotels which tend to be carrier exclusive facilities. The classic two in the early days of the commercial internet were MAE-East and MAE-West but those were the critical points for only about a year or so because they existed for NSF-net reasons and were made less important as soon as the commercialization of the the net started.

Re:Intelligence Op

[Medievalist]

You moron, he was making a funny comment about skynet, not talking about REALLY Making the internet withstand a nuclear attack. Who cares if there is any internet after a nuclear attack, the world will be torn to small pieces anyways, we will have to rewire a small asteroid, at least we wont be shut down by a faulty router in europe!

Your polite and informative comment made me change my mind, my political party, and my underpants!

You are the most persuasive and intelligent person imaginable; your mastery of capitalization and punctuation makes my knees go weak with envy!

I wish I was smart like you, and not a moron!

Re:Intelligence Op

[Medievalist]

One of the early arguments made by DARPA folks to politicians, in order to secure continued federal funding for packet switched network development, was the ability of the network to route around failed or destroyed nodes. They made this argument in the context of the cold war, of nuclear war.

That's not how I remember it, but I was not privy to the conversations of politicians, I was writing code and hooking up boxes.

Last time I checked you could destroy the Internet with two nukes; one in Herndon and one in San Jose. All the surviving major hubs will overflow memory and everything starts breaking up and reforming in horrible cascading waves. That was years ago, though; by now it might take ten nukes... but a nuclear war would involve hundreds of bombs. Anything less would be a terrorist incident, not a nuclear war.

Re:Intelligence Op

[idontgno]

Well, the Internet isn't ARPANET. So arguments based on the fragility of the modern Internet completely misses the point. (CAR ANALOGY WARNING) That's like arguing that because computer engine controls make a modern car susceptible to weird engine failure modes, the century-old basic design of the internal combustion car engine was a failure.

The ARPANET had, at most, dozens of nodes. There were very few points of concentration, and those points of concentration had peering connections with nearly all the others.

In a Soviet pre-emptive strike, most primary command-and-control nodes would have died. However, not all IMPs were located anywhere near those first-strike targets, so the network may well have survived.

If the ARPANET had panned out the way it was probably envisioned, it would have replaced AUTODIN and you damn stinking hippie civilians would have stayed the hell away from it.

Damn, I wish I still had my ca. 1983 DDN TAC card. Good times, good times...

Re:Intelligence Op

[hesaigo999ca]

Sorry, I didn't eat my wheaties that morning, after rereading my post,
I feel a little embarrassed. I guess it was a bit overkill....
so much for free speech!

Re:Intelligence Op

[jon3k]

Our elevator has a lever that does exactly that. You twist this metal pipe into some other big doo-dad and pull down on it, in the service room next to the elevator.

I just re-read that and realized how useless it was. posting anyway. GO GO KARMA!

Few stories back...

[Anonymous Coward]

A couple of Nuclear Subs probably cut an underwater cable...

Re:Few stories back...

[morgan_greywolf]

Nuclear Sub? Is that a new sandwich from Subway?

"The all new Subway Nuclear Sub: It glows in the dark! Get a lotta green for a little green! Now only $5.99 for a 12-inch! Subway: Eat Fresh!"

Re:Few stories back...

[Kell Bengal]

A nuclear sub would be one with extra tabasco sauce.

Re:Few stories back...

[ottothecow]

Don't you know that all foot long subs are only $5 right now?

get with the program

Re:Few stories back...

[mschuyler]

That is actually correct. The sub shop in Bremerton (West Coast port for Trident Ballistic Missile Submarines, SSBN-726, etc.) sells the Trident Nuclear Submarine Sandwich with an extra serving of horseradish somewhere in the middle of it. It'll light your hair on fire, or, in my case, turn my scalp red.

Re:Few stories back...

[Klowner]

Five.. Five dollar.. Five dollar nuke suuuuub

Re:Few stories back...

[dov_0]

Pity on both of them... Tobasco isn't hot, but has great flavour and horseradish is only hot for a few seconds...

Re:Few stories back...

[Kagura]

A couple of Nuclear Subs probably cut an underwater cable...

More like, a couple of Nuclear Subs collided with a satellite and probably cut an underwater cable...

Re:Few stories back...

[ImYourVirus]

... or crashed into one *cough*

Re:Few stories back...

[gandhi_2]

The Nuclear Sub from subway would be 24 inches long, and half the sammich would be on French Bread, the other half would have cabbage.

Re:Few stories back...

[supernova_hq]

A really fun prank is to go to a buffet dinner and put a good spoonful of horseradish into the butter, and stir. Then wait for those gluttonous relatives to load a big knife-full of butter onto their buns!

Re:Few stories back...

[supernova_hq]

Actually, the commercial states "$5 for any regular foot long."

That excludes ones such as (I believe) seafood, meatball, cheese steak and a couple others. It's all in the details...

Re:Few stories back...

[dov_0]

You can stir butter in the US? Ours is too solid for that.

Re:Few stories back...

[laejoh]

Kinky!

Re:Few stories back...

[morgan_greywolf]

It also excludes my personal favorite, the BMT.

Re:Few stories back...

[Monsieur Canard]

In the interest of completeness USS Ohio is no longer SSBN-726, it is now SSGN-726. It was converted a few years back from a Trident-carrying boomer to a "slow attack" capable of carrying 100+ Tomahawks, plus some SEAL capabilities.

Re:Few stories back...

[P1h3r1e3d13]

In southern states, it's marketed as the "Nucular Sub."

Re:Few stories back...

[StikyPad]

We soften it to make it suitable for drinking in large quantities. How do you think we maintain our shapely physiques?!?

Re:Few stories back...

[ThatsNotPudding]

That is actually correct. The sub shop in Bremerton (West Coast port for Trident Ballistic Missile Submarines, SSBN-726, etc.) sells the Trident Nuclear Submarine Sandwich with an extra serving of horseradish somewhere in the middle of it...".

...there were no survivors.

Re:Few stories back...

[Kell Bengal]

Actually, I'm something of a hot spicy food enthusiast - living in Thailand gave me a taste for curries and spicy food. I held the record at my old university for eating the most hot peppers without water (they actually ran out of things to challenge me with). I deliberately chose tabasco because I thought the odds of readers being familiar with it was higher.

Re:Few stories back...

[supernova_hq]

Wow, I figured that would be included. I guess it's the bacon that they figure is a "luxury" topping.

Re:Few stories back...

[bozojoe]

You Sir, are my new favorite wet dream!

Re:Few stories back...

[morgan_greywolf]

No. The BMT doesn't have bacon. (BMT==Biggest, Meatiest, Tastiest)

Re:Few stories back...

[jon3k]

if you eat seafood and "steak" from subway, I seriously pity you.

Re:Few stories back...

[mschuyler]

Actually, the commands have been consolidated so both 'bases' are under one command as PSNS & IMF. This inculdes Bangor, which in any case is a 15 minute drive away. And besides, that's pedantic at best.

Re:Few stories back...

[supernova_hq]

Ummm, thanks?...

Half the internet? Are you serious?

[Anonymous Coward]

A router takes out 'half the internet' and I learn this from Slashdot?

Seriously, what is/was the impact? I work for a large e-commerce provider and haven't seen a thing that would indicate a problem today.

Sorry

[Alcoholist]

My bad. I never should have cut that tape.

Re:Sorry

[Failed Physicist]

Nor should you have flipped that magic switch!

Re:Sorry

[Alcoholist]

Well it said "more magic" and after you cut the tape you'll try anything. Hey, I figured that more was better. How the hell was I supposed to know!

Hmm...

[Vectronic]

I suppose that a networking event with one of our upstreams was behind that router?

3/11 (invalid or corrupt AS path)

Or maybe I'm behind that router?

Re:Hmm...

[Vectronic]

Ignore that, it's working now, just took 4 minutes.

BGP

[winkydink]

The internet's dirty little secret. It's amazing it works at all.

Re:BGP

[Savantissimo]

ATM really is more reliable for backbones than routed protocols. Problem is, it just works. Fewer jobs with stuff that just works.

Re:BGP

[ThatCanadianGuy]

What do you expect from Al Gore?

Re:BGP

[damium]

If reliability was the only concern they would likely use ATM.
Speed is a major concern due to SAR bottlenecks. Also, ATM networks are expensive, difficult to implement, and inefficient at bulk data transfers.

Re:BGP

[mvdwege]

Yeah, until a major ISP decides to run their ATM core switches at almost 100% load and one of them goes offline during an upgrade. Can you say 'cascade failure'?

And this is not hypothetical. This is exactly what happened last summer with the former monopoly telco here in the Netherlands. Took out a significant chunk of DSL service, including business lines.

Mart

Re:BGP

[RDW]

More details on the affected European router here:

http://www.youtube.com/watch?v=iRmxXp62O8g

Re:BGP

[duguk]

Well, clearly not. I'm in Europe and video states "This video is not available in your country.".. Either that or something dodgy is going on.

Re:BGP

[Savantissimo]

Hmm... at first I thought you meant SAR as Service Activation and Repair, which is (was?) a more frequent bottleneck than Segmentation and Reassembly, since building an ATM circuit (1 per DSL line) requires information and configuration of dozens of different systems; rebuilding circuits has become a standard troubleshooting step via scripts given to level 1 ISP call-center techs, and they have often come to treat it like rebooting a customer's computer.
The segmentation and reassembly thing seems more like a failure to invest in R&D - it's a very low-complexity mechanical process that should be a cinch even with FPGAs. It does become a problem when the switch manufacturers want to give the ability to do complicated processing of the contents of cells, even down to the application layer. That is a Bad Idea. When the switches get too clever and you can't depend on them passing data through unaltered let alone know what they're doing to the payloads, the whole system becomes impossible to troubleshoot. Anyway, the uber-speed links are less important than one might think since telcos prefer several redundant links to one giant point of failure.

ATM networks are expensive only because of the low volumes of units sold and the carrier-grade specifications for switches. DSL modems still often have native ATM capability built in, and aren't terribly expensive. The carrier-grade stuff might be a lot cheaper if it only implemented the 5-10% of the ATM spec that actually gets used.

I'm not sure where you get the idea that ATM is inefficient at bulk data transfers. The overhead is constant, the operation is deterministic, it's great for streaming, it allows effective use of more of the nominal bandwidth than anything besides maybe SONET. You could use full duplex Ethernet with jumbo frames, but I believe the effective performance would be worse than an equivalent bandwidth ATM link for everything but bulk transfers

Re:BGP

[Savantissimo]

Well, ATM was idiot-proof until they started rolling out the new-model idiots.

Seriously, no protocol is going to prevent problems when you try to put ten kilos of data in a five kilo sack. If they had provisioned the business lines at a higher QoS than the rabble have might have mitigated the disaster, but it's rare to see anything but UBR (unspecified bit rate) in practice. Most core networks run at much less than 50% capacity, often less than 10%, so the QoS has no real effect for networks that aren't ridiculously oversold. The Dutch telco was just greedy and incompetent even beyond normal telco greed and incompetence.

Pre-FUD propaganda

[marco.antonio.costa]

No, we DON'T NEED A NEW INTERNET! Stop pitching it, statist drones.

The internet works fine, and that's what the RIAA/MPAA/etc are trying to fix.

Re:Pre-FUD propaganda

[Anonymous Coward]

>Stop pitching it, statist drones.

When you say 'statist drones' I picture you in your mother's basement wearing her mascara and listening to Rage Against the Machine loudly. How close am I?

Re:Pre-FUD propaganda

[Hecatonchires]

You left out 'updating your myspace page', 'writing poetry about how no-one understands' and 'cutting yourself'

Re:Pre-FUD propaganda

[marco.antonio.costa]

You got the mascara right, the rest is off. Nice try tho. :-P

You get Duct tape?

[Forge]

Lucky Yankees with all your fancy technology. If I told you what we use, nobody would respond for fear that in attempting to respond I would cause a few fatalities.

Re:You get Duct tape?

[CarpetShark]

That's "Duck Tape".

Re:You get Duct tape?

[Clandestine_Blaze]

Lucky Yankees with all your fancy technology. If I told you what we use, nobody would respond for fear that in attempting to respond I would cause a few fatalities.

You use Chuck Norris?

Yep, Its true

[Bryansix]

Our Hosted VOIP service took a dump today at 8:40 AM PST. Supposedly it was a server in the Czech Republic. From the Carrier

2009-02-16 0945 PST CP experienced a core network connectivity issue due to a world wide BGP issue that affected all BGP interconnected networks. A rouge machine in the Czech Republic was making bad AS advertisements that caused systems world wide to fail. We have worked with our providers as well as our internal Engineering department to effectively block this node and restore service to our network. This is an ongoing issue that is still being worked to get a 100% correction. There is a workaround currently in place until a complete fix is available.

Re:Yep, Its true

[radish]

A rouge machine in the Czech Republic was making bad AS advertisements that caused systems world wide to fail.

Now I really don't know all that much about large-scale networking so maybe someone could explain this to me. What difference does it make if the router is rouge, versus say, green? or black?

Thanks for any insight :)

Re:Yep, Its true

[Anonymous Coward]

Wow, those Czechs have machines for rouge? I guess that explains why they have so many fashion models...

Re:Yep, Its true

[Jurily]

A rouge machine in the Czech Republic was making bad AS advertisements that caused systems world wide to fail.

Wasn't the internet designed to withstand WW3? And now a single machine takes it down?

Please enlighten me, how is that possible?

Re:Yep, Its true

[ChunderDownunder]

Since folks on Slashdot seem to like car analogies, I'll just mention that Red Cars Go Faster and assume that the same law applies for routers.

Re:Yep, Its true

[pyite]

Now I really don't know all that much about large-scale networking so maybe someone could explain this to me. What difference does it make if the router is rouge, versus say, green? or black?

So they announced a route that was, shall we say, malformed. Part of the problem is that due to a Cisco bug (CSCdr54230), some routers choke on it instead of ignoring it. The bug is fixed. It was fixed some time ago. Nonetheless, it's a pretty bad bug, labeled as "1 - catastrophic" by Cisco (in red letters, even). Routers still running affected code versions are having issues.

And it's only at this point in writing my reply that I realize you were taking advantage of a pun by way of misspelling. I'll leave my reply anyway ;-)

Re:Yep, Its true

[ScrewMaster]

A rouge machine in the Czech Republic

That's the problem. You shouldn't use rouge on your routers.

Re:Yep, Its true

[Anonymous Coward]

Everyone knows rouges are overpowered, just ask any mage.

Re:Yep, Its true

[myowntrueself]

That's the problem. You shouldn't use rouge on your routers.

I think that a rouged router would possibly be overly promiscuous.

No wonder problems like this can spread like the clap in a port town!

Re:Yep, Its true

[Zironic]

Internet is meant to be able to route around broken nodes and it seems to do this rather well, just not as quickly as some would like.

Re:Yep, Its true

[mail2345]

WW3 is an external problem.
A misconfigured machine is an internal problem.
The internet can survive cut cables, provided that there are other routes.
But if it can't find said routes, then there is a problem.

Re:Yep, Its true

[Threni]

Are you sure it's not just affecting a couple of shitty boxes in Prague or wherever? I've noticed no problems today. I'm sure there are a few pissed bedroom boys getting high ping on their Open Arena servers, but it's hardly the end of the world.

Re:Yep, Its true

[Hecatonchires]

Yes, Mages are known for powdering their cheekbones. Rogue's on the other hand, like to stab people in the back.

Re:Yep, Its true

[Hadlock]

Our ecommerce site went down at 10:40am CST (-6 GMT) due to an insane amount of traffic coming from China (different from your location). We finally blocked the IP at the firewall but an interesting coincidence.

Re:Yep, Its true

[jonwil]

If the bug is THIS serious and can cause such major problems, why are there any routers out there that haven't run the fix yet? Are they routers that are too important to bring down long enough to apply the fix? Too much risk involved (vs the smaller risk of actually being hit with the bug)? Too old to support the needed new version? Owners too stingy to pay for the new version that has the fix?

Re:Yep, Its true

[b3h]

Yeah but I hear black computers are faster.

Re:Yep, Its true

[v1]

It was meant more to stop the network from failing due to LOST nodes, not malfunctioning nodes. But that doesn't say much for its ability to withstand sabotage which is expected in wartime.

Re:Yep, Its true

[andrikos]

Rouge is overpowdered!

Re:Yep, Its true

[greenlead]

Bad publicity is good publicity. :-)

Re:Yep, Its true

[travbrad]

I'm going to go with option G) Laziness

Re:Yep, Its true

[Zwicky]

Wasn't the internet designed to withstand WW3?

Well originally yes. Over the years other needs have dictated a different direction to take it in. These days it is designed not so much to withstand WW3, but to withstand WoW3 (and up).

Re:Yep, Its true

[blind monkey 3]

I think that should have read:
A rogue machine in the Czech Republic was making bad-ASS advertisements that caused systems world wide to fail.

Re:Yep, Its true

[NormalVisual]

True, but it's warlocks, not mages, that whine the most about rogues being OP.

Re:Yep, Its true

[troll8901]

Red shirts 'go' faster too.

Re:Yep, Its true

[Anonymous Coward]

> Owners too stingy to pay for the new version that has the fix?

That's my problem now. I don't want to pay more to cisco for a needed software update than my border routers are worth. SMARTnet is not cheap. Of course that's not an option anyway because cisco doesn't allow me to run the newest versions of IOS on our model of routers. Maybe I could find a version that fixes this particular bug that I'm allowed to run, but TAC was unsure if that was true.

People forget that when dealing with routers, you're dealing with a very expensive and closed company. The hardware is made to be junked very quickly. While I've only replaced a couple of routers due to hardware problems, I have about 40 cisco routers sitting in a pile that I had to scrap because I couldn't upgrade the software.

Re:Yep, Its true

[jonwil]

The same thing applies to Windows and Mac also, there comes a time when your hardware wont run the newest version of the OS.

Re:Yep, Its true

[Die The Villian]

Since folks on Slashdot seem to like car analogies, I'll just mention that Red Cars Go Faster and assume that the same law applies for routers.

Red car paint has a negative friction coefficient.

Re:Yep, Its true

[Urban Garlic]

> Yes, Mages are known for powdering their cheekbones. Rogue's on the other hand, like to stab people in the back.

So whose law is it that spelling/grammar corrections have grammer errors? Or are you just trying to twit the pedants with your rouge apostrophe there?

Re:Yep, Its true

[Bryansix]

No, you don't understand. It is not that the hardware is not CAPABLE of running the new software. It is that Cisco is forcing customers to upgrade hardware to get access to later software which in all actuality would more then likely run just fine on the older hardware.

Re:Yep, Its true

[ThatsNotPudding]

Since folks on Slashdot seem to like car analogies, I'll just mention that Red Cars Go Faster and assume that the same law applies for routers.

Yes, but it pales in comparision to rubbing cheetah blood all over it.

Re:Yep, Its true

[Hecatonchires]

^_-

Re:Yep, Its true

[Nefarious Wheel]

This is called "the principle of Bit Decay". It posits that any piece of working software, if left to itself long enough, will eventually cease to run.

Re:Yep, Its true

[jon3k]

No, the problem is cisco makes new, better, hardware and you expect them to continue to provide support and software for a piece of hardware they stopped making a decade ago. It's just not feasible.

I'd also like to point out that any REAL investment (think 4500, 6500, 12000GSR) has and/or will continue to work for 15 or 20 years.

Re:Yep, Its true

[Bryansix]

Mmhmm, You obviously don't know shit about Routers. They could run on a 486 and still do fine.

Re:Yep, Its true

[ScrewMaster]

That's the problem. You shouldn't use rouge on your routers.

I think that a rouged router would possibly be overly promiscuous.

No wonder problems like this can spread like the clap in a port town!

That's a good one. How it got modded +5 Insightful is beyond me though.

Re:Yep, Its true

[jon3k]

Obviously you've never worked on a "router" outside of iptables in your dorm room.

Last week I established a single full mesh (50 sites) BGP session with my MPLS provider and CRUSHED a 3845, cpu was completely pegged. And that's just BGP, imagine a CPU intensive routing protocol like OSPF. Now throw some QoS on top of that, oh and let's enable the new Zone-based policy firewall. Should I continue?

Yes, clearly, you're the expert here.

Re:Yep, Its true

[Bryansix]

That completely misses the point. The point here is that a Router which is designed to run CPU intensive protocols and the like is speced with hardware to support it. The next model of that same Router will have Incremental upgrades to the CPU and Memory. The point here is that Cisco ceases to support Routers long before they cease to be able to actually perform the function they were purchased for.

Re:Yep, Its true

[jon3k]

You said "blah blah they cound run a 486 and do fine."

Please show me a 486 that can do EBGP, policy-based QoS (CBWFQ or LLQ) and route a couple DS3's worth of traffic. I'll just completely ignore the fact that this device, in an enterprise environment, would also need to most likely participate in some form of IGP to distribute routes into the network core.

AS 47868

[Anonymous Coward]

There is a post in nanog and on isc.sans.org.

AS 47868 causing AS paths to become too long...

http://www.merit.edu/mail.archives/nanog/msg15472.html

I lost a router

[Philip K Dickhead]

And took out THE _WHOLE_ INTERNET!!!!!

It's true! Ask my wife!

Re:I lost a router

[biocute]

Which one is true? A lost router took out the whole internet, or you have a wife?

Re:I lost a router

[zobier]

Are you saying that you accidentally the whole Internet?

Re:I lost a router

[genner]

Which one is true? A lost router took out the whole internet, or you have a wife?

Well this is Slashdot......

Re:I lost a router

[sharkey]

They took the Internet!

The whole...
FUCKING...
Internet!!!

Re:I lost a router

[Random Destruction]

Yes. I accidentally the whole internet.

Is that bad?

Re:I lost a router

[Basehart]

Very good

Re:I lost a router

[Nefarious Wheel]

Yes. I accidentally the whole internet.

Is that bad?

Oh dear gods, you didn't delete that little "e" with the bit of gold on it did you?

You weren't supposed to do that...

Re:I lost a router

[indi0144]

depends.. would you miss 4chan and digg? oh, maybe you'll become an hero.. for great justice! (tm)

Re:I lost a router

[Tuna_Shooter]

What ??? you have the internets???? i just use some string and some tin cans for my connection.

Re:I lost a router

[Upphew]

You should wire your router to wall and/or computer, that way you just follow the cable and voila! there is your router

Re:I lost a router

[Philip K Dickhead]

Poseur.

One does not "ask" 'bitches' - you TELL them.

Oops

[kbob88]

Sorry, I *told* Mustafa not to drop the anchor there! But does he listen to me? No...

Ditto the A.C.

[khasim]

It must have been the "half the Internet" that I don't use. Which would be an interesting half because many of the sites I visit regularly are based in Europe.

From the thread, it looks like AS 47868 was the route being lost.

http://en.wikipedia.org/wiki/Autonomous_System_Number

Re:Ditto the A.C.

[roc97007]

> It must have been the "half the Internet" that I don't use.

The non-pr0n half.

Re:Ditto the A.C.

[0xygen]

I think AS47868 was causing the routes to be lost.
It was making mass BGP announcements about really long incorrect routes.

Re:Ditto the A.C.

[besalope]

> It must have been the "half the Internet" that I don't use.

The non-pr0n half.

Such a place exists? 0.o

Re:Ditto the A.C.

[petecarlson]

It wasn't just AS47868, it was kicked off by AS47868 sending real long routes like you can get to a by going through b, c, d, e, f ,g, h... and so on and so forth. Older versions of IOS wack out with the crazy long routes and lose their BGP sessions so it is possible that he lost half of the internet while you were on a network segment which was not seeing the issue. If the OP were to post the ASN or IP block he was on we could run BGP play and see just how much of the net he really lost. I'm going to guess about .5%.

Re:Ditto the A.C.

[Greyfox]

No.

Now that I'm done posting, perhaps you could help me with my sexual inhibitions?

Re:Ditto the A.C.

[Ihmhi]

We wouldn't notice anyway. If you do a bit of WHOIS research you'll find that /. actually piggybacks on the server for mantrain.com.

Re:Ditto the A.C.

[GNious]

"Sir, the Nerd-o-meter is shaking violently!"

Re:Ditto the A.C.

[Freultwah]

many of the sites I visit regularly are based in Europe.

What? You're surfing long-distance?

Re:Ditto the A.C.

[serveto]

That'd be the non-pr0n 5%

Re:Ditto the A.C.

[fostware]

Not really... all those Cisco Academy kids nearing completion would understand this...

It used to be an employment lock-in to know RIP & ipchains, then came some funky stuff with pf & iptables, now it's all BGP, MPLS, and CSPF / OSPF.

Dammit, now I feel old again :S

Re:Ditto the A.C.

[Nuitari The Wiz]

I got affected by that outage and it was somewhat annoying. It seems like Cogent was affected and also some, but not all, of my colocation services.

http://www.sigbjorn.org/funstuff/images/ios_sucks.jpg pretty much sums up how I feel.

Re:Ditto the A.C.

[petecarlson]

Is that a 12000 series?

Re:Ditto the A.C.

[Nuitari The Wiz]

I wouldn't know, it's a picture I found on the internet.

Re:Ditto the A.C.

[Patch86]

A non porn half? Did you forget to carry the 1?

Re:Ditto the A.C.

[w1d3]

yes but it's the smaller half

Trust

[TubeSteak]

Until the internet evolves away from its trust-everyone roots,
one well placed server will be able to cause massive damage.

There would be a lot more impetus to force the change if hackers were nuking things from orbit for lulz instead of infiltrating systems for business reasons (spamming, bot herds, etc).

Re:Trust

[lotaris]

This only hit people running old unpatched versions of IOS. Known and patched long ago.

Re:Trust

[lambent]

Precisely. It wasn't "one broken router" that took out half the net, it was thousands of substandard routers using obsolete code run by incompetent admins that took out half the net.

the people who actually know what they're doing were unaffected by this.

Re:Trust

[martas]

NO, NO, NO!

Stop saying that everyone trusts everyone on the internet! This is not true, and it hasn't been for a while. I'm tired of hearing doomsday scenarios about how tomorrow some 13 year old kid in the Ukraine is going to bring down the whole internet. The way most networks are run today is actually very secure. We don't need a new design, we don't need to start from scratch. All we need is to shoot all the idiots who don't protect their computers and let them become a part of a botnet, because realistically that's by far the biggest danger for the Tubes right now.

Re:Trust

[troll8901]

Stop saying that everyone trusts everyone on the internet! ... I'm tired of hearing doomsday scenarios

Agree. However I must say: just because the parent post causes us to growl in disgust, doesn't mean they affect the rest.

Reason is, other Slashdot readers tend to be very mature, experienced and knowledgeable. They know the real vulnerabilities in the Internet (BGP and DNS), and they pay more attention to these matters. The doomsday speeches that annoy us to no end, doesn't affect them in the least bit.

---

The way most networks are run today is actually very secure.

Agree. The last time we had such an incident was a year ago (equivalent to 7 dog years) in the Hijacking of BGP routes to Youtube incident.

---

All we need is to shoot all the idiots who don't protect their computers and let them become a part of a botnet, because realistically that's by far the biggest danger for the Tubes right now.

Slashdot writers don't worry about botnets because they know how to

• filter DDoS at the ISP level.
• filter Worms at the Firewall/AntiVirus/Windows file sharing level.

On the other hand, Slashdot writers don't really have access to the BGP and DNS infrastructure. This worries them because they know the vulnerabilities in BGP and DNS. Therefore Slashdot people talk about BGP and DNS vulnerabilities, and "how tomorrow some 3 month old ISP in the Elbonia is going to bring down the whole internet".

Re:Trust

[Ash-Fox]

I am unable to locate evidence that 50% of the Internet was non-functional.

Re:Trust

[TheTurtlesMoves]

Even with obsolete code. Properly configured routers shouldn't have had a problem like what is claimed. So it really wasn't 1 incorrectly configured router. It was a lot of them.

But really, I'm in Austria and we use a lot of internet sites and resources in Czech, we have noticed nothing. So it not really widespread at all.

But right now /. is a bit borked for me. Posts not loading properly, preview taking a long time. Perhaps that would have been more news worthy.

baling wire, not bailing wire

[bugi]

http://en.wikipedia.org/wiki/Baling_wire

I think you mean baling wire. One uses buckets for bailing.

Re:baling wire, not bailing wire

[Tomun]

No only that, but leave duct tape alone. It's an excellent way of holding things together. I'd trust your life to the stuff.

Re:baling wire, not bailing wire

[onkelonkel]

Could I use baling wire to make a new bail for my bailing bucket? If so, would my wire bail made from baling wire become bailing wire?

Concerned and puzzled.

Re:baling wire, not bailing wire

[Dun Malg]

Could I use baling wire to make a new bail for my bailing bucket? If so, would my wire bail made from baling wire become bailing wire?

Concerned and puzzled.

No, because the noun "bail" in the sense you use it means "handle in the shape of an arc". There is no verb form in reference to that noun, therefore there can be no "bailing wire". It's still just "a baling wire bail on your bailing bucket".

Re:baling wire, not bailing wire

[nmb3000]

One uses buckets for bailing.

I knew the government was doing it wrong.

...Unless the buckets are already full of something else and ungodly amounts of money are the next best thing. That must be it.

Re:baling wire, not bailing wire

[troll8901]

the noun "bail" in the sense you use it means "handle in the shape of an arc".

May I "bail" out Joan of Arc while she is being imprisoned, then?

(Nouns and verbs confuse me.)

"One Bad Apple"

[Philip K Dickhead]

...Don't Spoil the Whole Bunch, Girl!

Oblig. I.T. Crowd

[XanC]

What is Jen doing with The Internet??

Re:Oblig. I.T. Crowd

[Anonymous Coward]

Googling Google on google?

Re:Oblig. I.T. Crowd

[bazald]

Has she tried turning it off and on again?

Re:Oblig. I.T. Crowd

[DecepticonEazyE]

The Elders of the Internet will be very upset!

Re:Oblig. I.T. Crowd

[v1]

(said with broken english indian accent) Hav yu tryed restartink your computr?

Re:Oblig. I.T. Crowd

[g1zmo]

Am I crazy, or have I heard her on BBC Radio 4 a couple of times? I swear it's the same voice.

Oh, it's obvious, you know...

[jtara]

If you can memorize this, you'll be the life of any cocktail party:

"We're seeing them from AS 48438, coming across to us as an Optional Transitive Attribute which our force-10s are not parsing (but cheerfully passing along to our clients, who are then flapping their peers because of it.)"

Uh-huh-huh-uh! They've been "flapping their peers".

Re:Half the internet? Are you serious?

[Frosty Piss]

A router takes out 'half the internet' and I learn this from Slashdot?

Non, no, no. You messed up the troll and got modded "Insightful". Let me fix that for you:

A router takes out 'half the internet' and this is front page news at Slashdot? Slow news day?

Thank you, I'll be here all week...

Don't knock duct tape

[fm6]

Well, do, you're right to be concerned. The thing is, our technology infrastructure has always been a nasty kludge. In 1965, some coincidental misconfigurations at two minor power plants took out the power grid for an area in the northeast U.S. and eastern Canada where 25 million people lived. It was 14 hours before the grid was fully restored. Our inability to keep our technical house in order is a very old problem.

Re:Don't knock duct tape

[maxume]

40 years later, the grid was still highly interconnected, and failed again.

(HVDC can be used to mitigate the issues that stem from having huge geographic areas be in phase, and the tech has existed at least since the 70's; I'm not sure if the DC interconnects end up being as efficient as the AC interconnects)

Re:Don't knock duct tape

[fm6]

Right you are. The people who run the networks seem to be a little better at anticipating this kind of problem, but the fundamental flaws are still there, and the big blackouts still happen from time to time. As with so many things, the will the make the fundamental changes is just not there.

Re:Don't knock duct tape

[maxume]

For something as big as an electrical grid, the cost savings from having it fail once in a while could very well outweigh the costs of it failing once in a while. The 2003 blackout was clearly too big, but I'm glad that there is resistance to making a perfect grid, because it makes more sense for people with critical uptime requirements to take care of that locally than it does to have everybody pay for reliability.

Re:Don't knock duct tape

[fm6]

For something as big as an electrical grid, the cost savings from having it fail once in a while could very well outweigh the costs of it failing once in a while.

Maybe for the kind of outage you get when a car knocks down a pole. But systemic issues always affect large numbers of people — and they basically halt all economic activity in the affected area.

And do recall that something that can happen by accident can happen by malicious intervention. If I were running a terrorist network, I'd want to stage a big series of coordinated attacks, one of which would be designed to deprive the affected areas of power. Imagine the nightmare for emergency responders. It's a good thing that Al Qaida seems to suck at anything that requires real planning.

Re:Don't knock duct tape

[maxume]

The 2003 failure is, as I said, an example of too big. It is also an example of how mild the consequences were (it cost a shit ton of money, but that was about it, which in the scheme of things, is o.k., as long as it isn't frequent).

As far as security, robustification is fine, but it should be done where it is economic, and resources should then be devoted to (preemptory) investigation, rather than target enumeration (i.e., the what should we protect this week game).

Outage Cause: Old software

[Anonymous Coward]

The AS 47868 decided that they wanted to prepend their ASN about 75 or so times to their BGP announcements. When this got re-populated throughout the rest of the world, a bug in older versions of Cisco IOS still in use on many ISP/NSP networks does not like paths this long. As soon as they saw the prefix with that long of a path, the software terminated the BGP session, resulting in the doorway being closed between the two networks -- So on and so forth throughout the rest of the web.

Re:Outage Cause: Old software

[petecarlson]

Perhaps they were just trying to load balance a 56k modem and a 10gb connection.

Make sure you are using cat 5 bailing wire.

[tlambert]

Make sure you are using cat 5 bailing wire.

-- Terry

Re:Make sure you are using cat 5 bailing wire.

[egcagrac0]

Can't. It's Monday. No cheezburgers.

It took out 9000 internets

[need4mospd]

In other words, a single mis-configured router is apparently able to cause a DOS for a huge chunk of the Net.

This means the router was able to take out over 9000 internets. Quite impressive.

It's not just the internet

[Weaselmancer]

A lot of things, as it turns out, have these single points of failure that propogate.

I got to experience this one.

Drove down Route 76/80 to NYC while it was happening. One city would be on, another off. No rhyme or reason to it at all.

Re:It's not just the internet

[maxume]

If you knew where smaller power plants were, it probably would have made a great deal of sense (I was in Ann Arbor, the UofM campus had power, but the rest of the city was down, as it turns out, the university operates a small power plant (because the economics make a lot of sense when you can steam heat thousands and thousands of square feet)).

Re:You're doing WHAT with the wire?

[Vectronic]

Main Entry: bail
Function: verb
Date: 1613
transitive verb
1 : to clear (water) from a boat by dipping and throwing over the side usually used with out
2 : to clear water from by dipping and throwing usually used with out

Bailing Wire = Internet Tubes

I'm not sure I follow

[shaitand]

If I'm understanding this 'router' thing correctly, its like a faucet connected to the series of tubes?

If not, exactly what role does this router thing play in tube interaction?

Re:I'm not sure I follow

[petecarlson]

If I'm understanding this 'router' thing correctly, its like a faucet connected to the series of tubes?

If not, exactly what role does this router thing play in tube interaction?

Your understanding is rather accurate but what your missing is the manifolds. You see, all the tubes connect to big manifolds with valves to control what gets sent where. At each manifold room there is some poor admin who is in charge of opening and closing valves in order to make sure that the right AOL gets sent down the right tube. In order to keep track of what tube to send your AOL down, the admin keeps a list of all the other manifold rooms and how to get to them. Some of the manifold room operators didn't have a wide enough notebook to write down the new directions so they just closed all of their valves and went home.

Re:I'm not sure I follow

[shaitand]

damn that sucks! It must take a brave individual to be an admin in the manifold room. Just think, with each valve you never know if you are blessing the world with Halle Berry nudes or a goatse flood. My what pressure in those tubes!

Re:I'm not sure I follow

[PPH]

I'm not really sure either. But it appears that the end result was that the Internet was hosed.

Duct-tape and bailing wire

[HTH NE1]

And people don't believe me when I tell them all this new-fangled technology is held together by duct-tape and bailing wire!

And chewing gum. Don't forget the chewing gum.

Re:Duct-tape and bailing wire

[RoboRay]

We're all out of gum.

Re:Duct-tape and bailing wire

[troll8901]

Don't forget the chewing gum.

"Saturday morning I woke up nestled in Martha's arms. We fooled around for a while..."

Oooooo, sexy!

Re:Duct-tape and bailing wire

[HTH NE1]

We're all out of gum.

We are? Oh well, I guess there's nothing else to do than kick some ass then.

Similar story at MIT

[mpoon]

Funnily enough, something like this happened at MIT about a month ago. The whole MIT subnet went down for about 2 hours. Cause? Switch that was plugged into itself.

Re:Similar story at MIT

[Cramer]

That's what spanning-tree is designed to stop. (I see they configure networks like I do... turn off STP and wait for some idiot to create a loop.)

Re:Similar story at MIT

[HeronBlademaster]

My ISP had a similar issue early last year. They told me someone had plugged the incoming cat-5 (from the ISP) into the wrong port on their router... so this customer's router was handing out bogus IP addresses on the ISP's network.

It was quite annoying.

Re:Similar story at MIT

[Benanov]

Got spanning tree? :)

Re:Similar story at MIT

[v1]

if any of their nodes are forwarding DHCP offers from ANYTHING besides their own DHCP server, they deserve every bit of what they get.

I can click one check box to turn on DHCP server on my laptop, and if that takes down your network, you've got issues to work on.

Re:Similar story at MIT

[afidel]

Cisco's will actually turn off peers even if STP is turned off, they see too many of the same packets coming over broadcast with decrementing TTL's and turn off the uplink port to that peer. I found this out after the ADD CEO at a former employer plugged one training room jack into another, it took out all of C row which was next to the training room but the rest of the company continued to run. Took me a bit of watching logs on adjacent switches to figure out the root cause and fix it but after I explained the problem and the fact that it was kept from spreading by the intelligent switches we were praised for our diagnostic skills and our equipment selection =)

Re:Similar story at MIT

[Cramer]

Nope. Switches are layer 2 devices; TTL is part of layer 3. With STP off, even a Cisco switch will freak when a loop exists. (It has other ways of knowing it's been plugged into itself, however.) As long as the "broadcast storm" threshold isn't crossed, the port will remain enabled -- and even if it does, the blocking is temporary.

For another Fun Trick, send a vlan tagged packet from a jumbo frame enabled, non-trunk port to a non-jumbo (non-trunk) enabled cisco port. It crashes immediately. It *should* throw it away as an oversized frame, but doesn't. (I know... I'm the monkey that has jumbo frames enabled on ports leading to non-jumbo hardware.)

I wish...

[egcagrac0]

people don't believe me when I tell them all this new-fangled technology is held together by duct-tape and bailing wire

If only it were that reliable... my duct tape patches and bailing wire repairs typically hold for a decade.

Re:I wish...

[v1]

but at that point the duct tape gets crusty and the bailing wire starts to rust...

You forgot to mention....

[gearloos]

You forgot to mention which Microsoft company the router belonged to....

If it happened on South Park...

[HeLLFiRe1151]

it could happen anywhere.

Belkin = Broken

[pRtkL xLr8r]

So all terrorists have to do is buy a bunch of Belkin routers and set them up as per normal?

Re:Half the internet? Are you serious?

[jjrockman]

Punctuate much?

Re:Half the internet? Are you serious?

[glwtta]

Your connection also seems to be dropping all of your punctuation characters. Very annoying problem to have.

Someone dumping enormous amounts of material

[Beelzebud]

If I don't get that internet my staff sent me on Friday, I'm going to be pissed. Damn kids dumping enormous amounts of material in the tubes... IT'S NOT A DUMP TRUCK!

Am I being too vauge?

[HTH NE1]

That's the problem. You shouldn't use rouge on your routers.

They think a rouge router is in vouge, but they're out of their leauge. We should haranuge them! A plauge on them! Rip out their tounges so they cannot aruge! Them and their colleauges. Nothing but demagouges and idealouges I say. There can be no dialouge on this matter. Send them to the moruge!

Are you intriuged by my ideas and want to subscribe to my travelouge?

Re:Am I being too vauge?

[rts008]

*calls 911*
I think I just witnessed a brutal murder...of a spell checker. Gotta hide my dictionary!

Re:Am I being too vauge?

[value_added]

think I just witnessed a brutal murder...of a spell checker.

No worries. It's just Scottish.

Re:Am I being too vauge?

[ScrewMaster]

ARGH!! Brain.. rebooting..

T-800: "Was there a problem?"

John Connor: "No ... no problem."

Re:Am I being too vauge?

[ScrewMaster]

They think a rouge router is in vouge, but they're out of their leauge.

Everyone thinks that routers are all digital, but in fact many still contain the analouge hole.

Ye olde versions of IOS

[DeadBeef]

This only broke BGP implementations that are getting pretty long in the tooth now, on a moderately recent version of IOS all we saw is:

Feb 17 05:25:03.731 nzdt: %BGP-6-ASPATH: Long AS path 10026 3356 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 received from xxx.xxx.xxx.xxx: More than configured MAXAS-LIMIT

It was definitely an insane path, our routers were configured to drop anything with an AS path longer than 75, old versions of IOS would often just drop the BGP session ( or even crash with some _really_ old versions ).

I'm sure there will be some red faced network engineers updating IOS or even doing forklift upgrades of old boxes at their edges in the near future.

Re:Ye olde versions of IOS

[wein0]

That will tech them for being daft enough to run Cisco+IOS together with LARGE BGP tables... that's what real routers are for... Juniper...

Re:Ye olde versions of IOS

[kullnd]

Amen.

BGP Misconfigurations

[Taikutusu]

Misconfigurations occur more than you would think, especially with regards to BGP; one estimate is around 300 per day. Most aren't going to knock our a substantial portion of the network (most of the time they'll either make paths longer or simply knock out the origin network), although occasionally you'll see a "black hole" effect like this. Again, these misconfigurations occur all the time, it's just that no-one really notices unless it manages to bring down any sizeable portion of the network, which is pretty rare.

Where's the story?

[Eil]

Okay. So I'm supposed to believe that one "smallish hosting provider" and three email messages are proof that half the Internet went down today?

WTF.

Have the submitter and kdawson both forgotten what an Internet is?

Re:Where's the story?

[Ash-Fox]

Have the submitter and kdawson both forgotten what an Internet is?

An Internet was sent by my staff at 10 o'clock in the morning on Friday. I got it yesterday!

Why? Because it got tangled up with all these things going on the Internet commercially.

Re:Where's the story?

[makomk]

There's more than three e-mail messages - that's just the first NANOG thread on it. The second thread is longer.

Tens of thousands of web sites were affected

[miller60]

This incident knocked several major hosting providers offline, including Media Temple in Los Angeles and Canada's iWeb.

Only some old versions of IOS broke

[lotaris]

This only took down people running fairly old versions of IOS that didn't patch a known bug.

Did not affect non-cisco.
Did not affect modern versions of IOS
Did not affect old versions of IOS that set the knob to limit the max as-path.

Re:Only some old versions of IOS broke

[painehope]

Gee, you only described about half the mistakes that incompetent network admins could possibly make (buying a fucking Cisco, not updating their NOS, and not limiting AS paths).

That covers half the ISPs in Texas (including mine - these fuckwits can barely configure their routers correctly on a good day, let alone deal with a crisis brought about their own incompetence). YMMV.

I'd mod you up but I already posted.

Re:Only some old versions of IOS broke

[maxwell demon]

a fucking Cisco

Is that the new name for the promiscuous mode?

Re:Only some old versions of IOS broke

[painehope]

Foundry Networks, Force10, Juniper (barely), anything but Cisco. The only place that Cisco occasionally wins out is in feature-set, but that's rare and the other vendors almost always catch up a month or two later.

Cisco's always have problems, they're the Microsoft of the networking world (relying on sales and entrenchment rather than performance and stability), and they never perform up to spec. Hell, even their specs never have the backplane bandwidth necessary to facilitate full-speed connections with a fully populated chassis. They're universally hated in the HPC world.

Re:You're doing WHAT with the wire?

[gyrogeerloose]

why then we must repair with our Bailing Wire--or--in the vernacular--"balun war"

Balun war? You mean people are fighting over impedance-matching transformers now?

I'm tellin' ya, this world is going to hell in a hand basket--with a wire handle.

I'll bite...

[YodaToad]

OVER 9000?!

The half that doesn't know how to filter their BGP

[wsanders]

Everyone knows that BGP stand for "Border Gateway Politics".

Or, what you give up in reliability you gain back in increased complexity.

Sorry, those are the only BGP jokes I know.

Re:Half the internet? Are you serious?

[Anonymous Coward]

A router takes out 'half the internet' and I learn this from Slashdot?

Seriously, what is/was the impact? I work for a large e-commerce provider and haven't seen a thing that would indicate a problem today.

Well I'm not sure about you.

Personally, I have BIGGER news! A single router in a remote rural US state managed to take down the ENTIRE INTERNETS!!!!

Yes, indeed when I noticed my cat had unplugged the power adapter, I replaced it. Then the ENTIRE internet came back! It was amazing how I single-handedly brought back the whole internets. Al Gore would be proud.

Come again?

[digitalcowboy]

You did what now, where and it caused which?

On your next visit be sure to ask your "counselor" about narcissistic tendencies.

Mauve Router

[theMoleofProduction]

When are we going to drop this rouge nonsense and adopt universal standards?!?

What's the emergency?
It's mauve.
Mauve?
Universally recognized color for danger.
What happened to red?
That's just humans. By everyone else's standards, red's camp. Oh, the misunderstandings! All those red alerts, all that dancing.

Re:Mauve Router

[Zwicky]

Are you absolutely sure you want to do that?

It does mean changing the bulb.

Rouge Routers vs. Teal Routers

[billstewart]

The important colors of routers are teal and blue - most other ones seem to be beige (and of course, if you're running a beige router from a company that now makes teal routers, it's old enough that either you're not doing anything too critical on it or you're not a production ISP...)

(Beige, of course, includes black or steel or whatever other colors 1U servers come in, running software like OpenBSD or Quagga or Vyatta, as well as some of the non-top-2 hardware-based router companies out there.)

Duct Tape and Bailing wire?

[drolli]

Sounds like our lab where we try to make a quantum bit.

Re:Duct Tape and Bailing wire?

[Dunbal]

The problem with quantum bits is that they can be either alive or dead, until someone opens the box and lets the damned cat out again.

Re:Duct Tape and Bailing wire?

[Beelzebud]

That Schrodinger guy was real asshole. I tried his little experiment out and now Fluffy is dead!

Re:Duct Tape and Bailing wire?

[maxwell demon]

SchrÃdinger never looked into the box, for fear of killing the cat. Unfortunately he didn't consider that by keeping the cat locked without food, it soon would be dead with certainty.

Actually this insight provides a new interpretation of quantum mechanics: Wave function collapse is caused by lack of food!

"network event"

[adavies42]

is that more like a "severe weather event" or an "extreme savings event"?

Re:Laugh all you want...

[HeronBlademaster]

If you can access the machine but not a particular directory on the machine (as you seem to indicate here), then your problem is with filesystem permissions, not internet routing.

Lord of the Token Ring!!!

[Genda]

Welcome to Sauronet... One Router to Rule them ALL!!!!

Re:Lord of the Token Ring!!!

[TheGratefulNet]

I prefer good old 802.4, token bus.

"can't get on the bus without a token!"

(memories from an old networking company I worked for, back when token-bus still was competing with ethernet).

Re:Lord of the Token Ring!!!

[dido]

You misspelled Tolkien ring. :p

Even before that...

[tjstork]

The ancient egyptians

http://www.flickr.com/photos/hdonat/2422108343/

had their engineering problems too.

As soon as we humans invented technology, we humans began screwing it up.

Re:Even before that...

[fm6]

Uh, when exactly did that pyramid collapse? All the Egyptian pyramids are more than 3,500 years old. So if this one collapsed less than 3,000 years ago, it actually counts as a successful building project.

Re:Even before that...

[tjstork]

I don't know if the link I supplied was "the one", but there is a pyramid out there, where the egyptians built the walls too steep, and the thing collapsed under its own weight near around the time it was built. So they adjusted tack and built them with a shallower angle.

Re:Even before that...

[fm6]

Google seems to think you're talking about Snefru's Pyramid. There's some difference of opinion as to why it collapsed and when. Some authorities think it collapsed because of the steepness issue you describe (which may have been done to make it harder to pillage), and then was rebuilt to work around it. It then collapsed for good much later. Others think that it was badly designed, but that the Egyptians redesigned in mid-building to keep it from falling down. Either way, the second (or maybe only) collapse occurred in Roman times, or even the Middle Ages. That makes it a botch job by Egyptian standards (some of their earliest pyramids are still around after 4500 years!) but still better built than anything we have. I mean, can you think of a modern building that is still likely to be standing in 2000 years?

Re:Even before that...

[Bearhouse]

I mean, can you think of a modern building that is still likely to be standing in 2000 years?

Yes, unfortunately.

http://en.wikipedia.org/wiki/Flak_tower

The only bit of the 3rd Reich that may actually 'last a 1000 years'

Re:Even before that...

[Red Flayer]

I mean, can you think of a modern building that is still likely to be standing in 2000 years?

To nitpick a bit, the comparison whould be against a modern monument, not a generic modern building.

And there are likely several modern monuments that will last a couple thousand years... Mount Rushmore, for one (sure acid rain etc will make then unrecognizable, and it's likely one or more will have sloughing issues, but note that the pyramids, while still standing, are not in their original glory).

Personally, I consider the fact that the pyramids are still standing to be a factor of chance -- they are built in the desert, with low humidity and little to no plant life which could damage the structure (unlike the pyramids of central america).

But, if we wanted to build something to last a few millenia we could. Modern engineering doesn't prevent us from building for ridiculous longevity -- we just choose to optimize for other considerations (cost!).

Re:Even before that...

[fm6]

Uh, the ones on the page you linked don't look all that sound. They may have stood up well against Allied bombs and Soviet artillery, but give them a century or two of weathering and soil subsidence, and they'll just be piles of concrete and steel.

Really dumb thing to build. The same resources devoted to mobile AA guns would have been a lot more effective. What did Patton say? "Fixed fortifications are a tribute to the folly of mankind." Something like that.

Re:Even before that...

[fm6]

To nitpick a bit, the comparison whould be against a modern monument, not a generic modern building.

You mean like the Washington Monument? The Lincoln Memorial? None of those would last long without constant maintenance.

but note that the pyramids, while still standing, are not in their original glory

True. But most of the damage isn't natural. The pyramids were originally faced with limestone, which was too valuable for later builders to leave alone.

Mount Rushmore, for one (sure acid rain etc will make then unrecognizable, and it's likely one or more will have sloughing issues, but note that the pyramids, while still standing, are not in their original glory).

Dude, that's a mountain. It will be there for thousands of years regardless. Carving into it probably shortened its life span slightly.

Personally, I consider the fact that the pyramids are still standing to be a factor of chance -- they are built in the desert, with low humidity and little to no plant life which could damage the structure (unlike the pyramids of central america).

OK, there you make a good point. It's also worth mentioning that the preservative quality of the desert probably had a lot to do with the ancient Egyptian's obsession with immortality, which motivated the building of the pyramids in the first place.

And of course another reason we no longer build stuff like that: a shortage of slave labor.

Re:Even before that...

[Red Flayer]

Dude, that's a mountain. It will be there for thousands of years regardless. Carving into it probably shortened its life span slightly.

But the busts are not natural, and will remain for some time. Even if they were to be cut off and stuck somewhere else for display, they'd likely last a very long time.

Also, specious use of the Washington Monument and the Lincoln Memorial is just silly... yes they require maintenance to keep standing... so did countless Egyptian Monuments that are no longer standing, yet I don't see you complaining about the shitty construction/engineering of those monuments.

And of course another reason we no longer build stuff like that: a shortage of slave labor.

That's immaterial, since we have machines instead, which might be (probably are) be cheaper than slave labor. It's simply about cost vs. perceived value of longevity.

Re:No news here

[v1]

so what happened to the quotable, "the internet interprets censorship as damage and routes around it"?

Re:Laugh all you want...

[painehope]

Perhaps I should have made it a little clearer for people that don't know that I know what the fuck I'm talking about.

I did not say that this caused me to not be able to access said directory. Standard *nix permissions caused that. What this did cause was an endless series of headaches in trying to contact the admin in question who has root on the system. The system is not part of the college's IT structure (not even going into that bitch-fest), so it took a while to find the lady in question who has root on the goddamn system.

After finding said lady, she mucked up my damn password, then fixed it, then I spent some time customizing my environment and looking into a secure login option (believe it or not, I can telnet into the system, but SSH is restricted to users running VPN software - USER painehope WHACKS HEAD AGAINST WALL), confident in my ability to finish my piss-ant assignment (it's a joke, really - a simple C program as a "warm-up assignment" for the class) in under an hour.

Throughout the course of this affair, I had hell sending mail or doing anything else. But I chalked that up to my usual ISP routers (kid you not, if my ISP got any dumber, they'd be contracting out to Guatemala, not India). No big deal, right? I'm in the system.

Then I discover that I can't access the directory where the assignment is stored. Still not sweating...I should be able to get a mail in and either be added to the appropriate group or get the permission bits changed to allow access. Worse case - just get the admin to tarball the files to me.

Except this time I can't get to my mail for a few hours. Which places me past the time wherein I can get ahold of said admin.

That's the problem.

FUD/fear mongering

[FliesLikeABrick]

I'm tired of this kind of fear mongering, and it seems to show up on Slashdot as well as other places way too much.

What? The world is ending? An ISP just misconfigured their router and their peers or upstreams need to be better about filtering. The same goes for adoption of BCP38 to prevent intentional or accidental route hijacks.

That is all. For now.

Re:Laugh all you want...

[Zwicky]

On the plus side you can access Slashdot.

Life is good.

Ahem.

[andreyvul]

http://en.wikipedia.org/wiki/Over_Logging

The WHOLE Internet...

[geekmux]

Are you saying that you accidentally the whole Internet?

No, no, no, I thought I lost the whole Internet. Then I realized it was just that moron in Accounting again who accidentally put it in his Recycle Bin again.

Re:The WHOLE Internet...

[Anonymous Coward]

a well deserved woosh

TAG THIS ARTICLE KDAWSONSUCKS

[Anonymous Coward]

This "article" is incredibly misleading as nothing has really gone awry. It is just another pointless KDAWSON post. These things are getting REALLY old, KDAWSON.
 
I work for a tier-3 provider, and if "half the Internet" dies, you are going to hear from a half-brained big media outlet (e.g CNN, ABC) VERY fast.

Re:TAG THIS ARTICLE KDAWSONSUCKS

[ImYourVirus]

Yeah because all of the worlds internet runs through europe and without them we are hosed lol.
/sarcasm

If anything at all some of europes services are affected nothing more, nothing less. This isn't some huge thing to worry about. Just like a couple of weeks ago when those cables got nicked and we lost egypt or the middle east, whatever it was no big deal...

Re:TAG THIS ARTICLE KDAWSONSUCKS

[supernova_hq]

Try being on the other side of that "snip". Not that I was, but it's something to think about.

Re:TAG THIS ARTICLE KDAWSONSUCKS

[TheTurtlesMoves]

I'm in Europe and my internet has been flawless (both within the EU and to the outside). So if by affecting they mean "one redundant link somewhere" then perhaps it would be true.

But really, if the internet goes down for 24 hours. So What. I will have to go to work and talk to my colleagues instead of emailing them.

Re:TAG THIS ARTICLE KDAWSONSUCKS

[Anonymous Coward]

Yeah because all of the worlds internet runs through europe and without them we are hosed lol.

Er, yes, it does? Amsterdam and London are two of the worlds largest internet hubs for trans-continental traffic. Traffic from Europe, Africa, Russia the middle east and large parts of Asia route through Europe to the US. Look at a map sometime and you'll notice that North & South America are a very large island bordered by two very large oceans.

Re:TAG THIS ARTICLE KDAWSONSUCKS

[perryizgr8]

I was. But imagine my surprise when I started getting 8 mbps instead of the 2mbps I pay for. The speed went back to normal after about a week. I can't think of anything that would cause this increse in speed. But it was great for some time.

Re:TAG THIS ARTICLE KDAWSONSUCKS

[commodore64_love]

>>>I work for a tier-3 provider, and if "half the Internet" dies, you are going to hear from a half-brained big media outlet (e.g CNN, ABC) VERY fast.

Darn. And I thought I finally had an explanation for why I couldn't download Kim Possible yesterday.

Re:TAG THIS ARTICLE KDAWSONSUCKS

[DiLLeMaN]

I was. But imagine my surprise when I started getting 8 mbps instead of the 2mbps I pay for. The speed went back to normal after about a week. I can't think of anything that would cause this increse in speed. But it was great for some time.

Less botnet activity. Or rather, less of it reaching you.

A communications disruption can mean only one thin

[falken0905]

Thing, that is. You all know the rest...

Re:I don't know what is sadder

[ari_j]

How is either one sad? What's sad from my perspective is that (a) people feel that using idioms with misspelled words is acceptable and (b) people feel like a Wikipedia article that describes a commonly used commodity should not exist.

Who modded this funny?

[davidwr]

I'd go with informative or insightful myself. :)

Re:Half the internet? Are you serious?

[jafiwam]

I can confirm that there was a BGP broadcast error early today (9 AM CST more or less) that lasted for about 40 - 60 minutes.

It caused ours and several other companies upstream ISP to go offline in a BGP route flop. The route flop made bad routes for 30 or so second intervals, and then was dropped. Repeating in a cycle (taking the connection down).

Apparently there are corrective measures in the protocol, but a continued error broadcast can make them return.

Depending on your routing setup, you may or may not use BGP, or may or may not have been harmed by the issue. Ours has lots of peering points which may complicate things.

Anyway, this is an interesting event, we thought it was from some idiot typo error at the upstream provider.

Re:You're doing WHAT with the wire?

[ari_j]

Anyone who's bailed using wire knows it to be futile. Just like anyone who's baled using a bucket.

Mod parent up

[mbone]

Mod the parent up - this is the real cause of the problem.

bgp maxas-limit 75

would stop this on most routers.

damn you kids

[Caue]

Those damn kids' ball went through my window and knocked off the router... now all internet contents regarding my participation in the vietnam war are down! I'm telling your parents!

Re:Half the internet? Are you serious?

[QuoteMstr]

Amazing: the only punctuation character he used, he used incorrectly: '

over many t1's seem

The apostrophe never makes a word plural.

Re:Wait a Sec...

[DanJ_UK]

It was a Brit that invented the Web.

Re:Wait a Sec...

[DanJ_UK]

Although saying that, I hate the UK being tied to those dodgy Europeans.

Re:Hey I know how you could fix it:

[supernova_hq]

Yeah, ok, we'll push all the traffic from an entire hosting company though a single SSH tunnel....

Re:No news here

[The End Of Days]

it turned out to be as meaningful as "information wants to be free"

Re:HOW IS THIS OFF-TOPIC YOU DUMB FUCKS

[The End Of Days]

Lighten up, Francis

Reminds me of a south park episode

[baldrel]

so have you tried turning it of and on again?

Re:Reminds me of a south park episode

[maxwell demon]

Here's the button: http://www.turnofftheinternet.com/

Note that to see the effect, you have to allow popups for that site.

Re:Half the internet? Are you serious?

[palegray.net]

Right right say no more say no more

Re:Laugh all you want...

[painehope]

Actually, I'm still having problems with /. as well. No other problems, but /. was throwing 404's right and left a few minutes ago. Hmm...

redundancy!

[shentino]

It MUST NOT be possible for one router to do this.

The internet MUST have redundant paths in the backbone.

Companies SHOULD peer with each other more often at the top level and be damned with trying to force transit payments.

Companies at the top who fail to do so MAY go themselves.

Routing errors are cached, too.

[Waccoon]

I had a friend in Britain who couldn't access my web site for a good 2 months, either by DNS or IP, and I couldn't ping his IP at all. I figured it was some kind of stupid ISP filtering thing.

Also, twice so far, I've not been able to access my own web site by DNS, due to a screw up of some kind with my ISP's local DNS cache. Lots of people in my area (meaning "Massachusetts") were seeing someone else's site until the TTL expired and the local cache refreshed. T'was annoying. Of course, my site has a clock face as a logo, and the other site was dedicated to showcasing watches, so at least I got some level of enjoyment out of it.

Re:Half the internet? Are you serious?

[CmdrGravy]

Yes it do's, like in:

Cat's, dog's, fishe's, women's, boobie's, hammer's, house's etc's

Contact

[cuba]

Hi, I tried to contact the admins from Sloane, Czech republic. The girl on the hotline told me, there is no one (it is just 9:18 in the morning ...) so I left her with my contact info and a basic word: AS, BGP, bad bad bad. And my telephone number. They called me some 20minutes later saying: "That thing yesterday? Oh that was just a tiny little bug..." :-)

Re:Laugh all you want...

[shentino]

Odd...

If anything those should have been 503's.

Maybe some of /. files are off-site and were unreachable?

Re:A communications disruption can mean only one t

[maxwell demon]

Slashdot effect? :-)

if you really want to knock off the internets

[op3r]

Do what the oceans 13 guys did and knock off electricity at 1 wilshire.

Pfff...

[ChimneysCantTalk]

This is when my INTERNET ON A DISK comes in handy.

Thank you, Microsoft!

If we had to start again

[DaveDerrick]

If we had to start again, redesign & rebuild the internet, would it be any better ?

Bad summary - not one broker router

[Eunuchswear]

It was one misconfigured router, many underconfigured routers and huge numbers of broken routers.

One guy sends an overlong AS path - silly boy.

Many transit providers pass this path on - lazy bastards.

Lots of schmuks have broken (and obsolete) router software that fucks up when it gets an overlong AS path.

Who's fault was it?

Re:Half the internet? Are you serious?

[Kijori]

That's not actually quite true. Depending on where you are, you might be able to use it in certain circumstances.

For example, in British English, you would use an apostrophe for plurals of single letters (there were 10 C's). You can also use it to create plurals of abbreviations, especially where there would be ambiguity (Four IOU's), as a slightly old-fashioned plural of figures (in the 1930's, but 1930s is becoming predominant) and where short words would be odd if you simply added an 's' - for example, the Oxford English Dictionary gives both "yesses" and "yes's" as plurals of "yes".

Whether he was right or not in this case is debatable, but I can certainly see the logic in writing "T1's" rather than "T1s", to avoid the appearance of it being a different abbreviation - and it isn't without precedent.

Re:Hey I know how you could fix it:

[kcbanner]

--->>> Joke

O
-|-
| You?
/ \
------

I'm gonna get OPS

[ArhcAngel]

I am going to register all the big name URL's during the split. That way when the other half rejoins I'll have OPS!

snow days

[OrangeTide]

But really, if the internet goes down for 24 hours. So What. I will have to go to work and talk to my colleagues instead of emailing them.

I can go to work. But I can't access the corporate bug tracking, access company email. that's the trouble with being a satellite office of a bit company. About the only thing I can do is access our source code, and that's only because we were kicking and screaming over the slowness of accessing it over a VPN.

I think you might be surprised how many of us depend on the internet at least partially functioning for us to do our jobs. If the net goes down it's the equivalent to a snow day.

Re:snow days

[spacey]

If you have a business, you can't *depend* on the internet working. You are *hoping* that the internet will work when you need it. Anyone "depending" on the internet working is in for some serious disappointment over the long haul. If you need to depend on something, you have to buy redundant private lines from different providers and verify that it runs point-to-point, and even then you're not going to get 5 nine's.

Doesn't anyone on /. remember MAE-East in the 90's? Back in the day, it was like people were trying to prove that the entire internet can go down for a day because of someone's dumb redistribution of RIP-over-BGP.

The fundamental issue that "best effort" delivery of packets is misleading isn't going to be addressed any time soon. "Best effort" really means "look, your link will be up, and we'll shove it out one of our interfaces, but where it goes from there is anyone's guess".

-Peter

Re:snow days

[TheTurtlesMoves]

Whats a snow day? Its been snowing all week. Is that a snow week. Up north it snows almost every day for months during the winter.

If you can have a day off for a "snow day" then I guess a day off work while the network is fixed isn't that big a deal either.

Re:snow days

[OrangeTide]

Up north when the snow is too deep to open the garage door to get the 4x4 out, then that's a snow day. if it's only up to your knees then it's not usually a snow day.

When there is a 1" sheet of ice on every surface outside because of the 3rd ice storm this year, then that's a snow day.

Usually a few things shut down (most offices, factories, schools, etc). And the announce the closures on the radio. Of course if you're in the business of driving a snow plow then a snow day means you are working a double shift (at least).

half the internet?

[OrangeTide]

Had no problems over here. Nor on IRC. I think if half the internet died I would have saw a few netsplits on this IRC network with around 50K users.

Nice analysis of the problem

[scottdj]

There's a great analysis of the problem by the always knowledgeable Danny MacPherson up on his blog at Arbor Networks.

One Ping to rule them all...

[FishAdmin]

Obligatory Userfriendly: http://www.userfriendly.org/cartoons/archives/99oct/uf001156.gif

Re:Laugh all you want...

[painehope]

Actually, I think you're right. I was beyond frustrated yesterday and not quite thinking straight. Though I could have sworn that I received some 404's, but they might not have been from /. - I wasn't in a very sane state of mind yesterday.

One kdawson CAN ruin half of Slashdot

[Zero__Kelvin]

This is so absurd it blows my mind. The idea that someone is employed by Slashdot, but doesn't know that such a claim is completely absurd, boggles the mind.

This post shall be what it is

[maxume]

meta -- meta -- meta -- meta sadness.

Echo? Echo? Echo?

Chewing gum?

[rgviza]

"And people don't believe me when I tell them all this new-fangled technology is held together by duct-tape and baling wire!"

Dude, get it right, you forgot the chewing gum. It's a very important component!

Re:Half the internet? Are you serious?

[Bryansix]

From that link we get this quote from Al Gore...

During my service in the United States Congress, I took the initiative in creating the Internet.

He said this in 1999 on a show hosted by Wolf Blitzer. That's pretty clear to me even though snopes comes to his defense, Al Gore still deserves to suffer as the perpetual butt of all jokes because of that stupid political move.

Some websites unreachable

[Zippy47701]

I noticed over the weekend that I wasn't able to reach the Keresotes theaters website. I even tried proxies with no luck. Everything today is fine. Go figure, LOL. Looks like somebody needs to design a failsafe for the DNS framework. Any takers?

Re:Half the internet? Are you serious?

[GWBasic]

Al Gore would be proud.

Vint Cerf credits Al Gore as being instrumental in securing funding to develop the Internet.

are you sure about that?

[OrangeTide]

If you have a business, you can't *depend* on the internet working.

What if I said I work for the world largest online retailer?