Tier One ISPs Dying

xbmodder writes "Two tier one ISPs are down today. At about 23:30PST both Verio and Level 3 starting having problems with routes. According to Level 3 this is a software upgrade gone awry. Is this the end for Level 3?" Many, many reports about this are coming in, and if you're wondering why the stories were rather sparse overnight, it's because it's difficult to post them without internet access. Hope everyone else is back online too.

Comments

Showing solid green now.

[micronicos]

See http://scoreboard.keynote.com/scoreboard/Main.aspx ?xAxis=Destination&yAxis=Origin&zAxis=Metric&nAxis =Period/

Noticed it this morning

[discord5]

Noticed this this morning when a customer called upset about his hosting services being unreachable. A quick traceroute showed one of level3's ip to be down. A few minutes later more customers had problems with different routers from level3. As soon as I saw level3 I knew enough, shrugged it off and told the customer that it was routing problem we couldn't fix but those responsible were most likely already trying to fix it.

It seems fixed now though, so no, this isn't the death of the Internet just yet.

What kind of Timeframe

[PhraudulentOne]

Is this even an issue? I mean, this was probably scheduled maitenance that went a little longer than expected. I have been through this before. It just sounds like Level 3 dropped some core routers for a few minutes to do a code upgrade - it didn't work so hot, so they were down for a few more mintes, OSPF/BGP decided to tell all the clients that they have no routes, Level 3 gets the routers back up, OSPF/BGP tells everyone that their fine again. Was this like 6 hours, or 45 min?

Re:Call me silly?

[Anonymous Coward]

In 1994, The National Science Foundation (NSF) awarded contracts to replace the National Science Foundation Net (NSFNet) Internet backbone. These contracts were for backbone transport, routing arbiter and traffic exchange points (NAPs).
          These contracts were awarded for the original 15 NSF sponsored NAPs, and to become a Tier 1 ISP, you had to have atleast DS3 connectivty to all 15 NAPs.
          It's a very old and crappy definition, and I wish people would stop using it, because it is very easy to meet now adays, and most of those original NAPs are now insignificant, compared to the power of the force.

flapping

[SpectralDesign]

Way back in the day when I was a Network Controller at BBN Planet, if we began to have cascading routing outages we'd call it "Flapping"... Visualize a wounded bird squirming around on the ground flapping...

Takes me back... My first night on the job a rat in Berkeley chewed through the wrong cable and got himself fried -- he also happened to take the entire west-coast off the internet for the better part of a day.

Then there was the time an electrical worker got vaporized in a hole near MIT which caused quite a problem too as it overloaded the MIT power station, but the fallout wasn't nearly as bad as the day of the rat...

Re:flapping

[rah1420]

halon defeat

OT, but it brings back memories of working at Purolator Courier in the machine room. IBM mainframe shop.

We had had trouble with the damn fire suppression all day. On third shift, around 3 AM, the trouble alarm went off (again) for the umpteenth time. One of the operators, a nervous fellow who was a little bit green, went over to the annunciator panel and opened it to see what the Trouble Might Be.

A fire technician he was not, and he apparently didn't know the difference between the trouble bell and the klaxon that would sound when a halon dump was about to occur; so he reached around the open panel door and hit the halon defeat.

Or so he thought.

It was actually the Big Red Switch.

The whole room (full of 3420 and 3480 tape drives, the 3745s, the 3800 laser printers; and the floor above, containing trivial bits like the DASD and the CPU all plunged into a deafening silence.

We all stared at each other and at the newbie BOFHeck.

A few minutes later, the phone rang. It was the Indianapolis air hub for Purolator, wondering why (when they were about to receive about 150 planes from all over the country) they didn't have anything useful displayed on their green screens.

That was a fun morning.

Ah, those were the days indeed.

The main problem areas...

[dantheman82]

happened in Detroit in the last 24 hours. Apparently all ingoing/outgoing traffic to other Tier One ISPs had problems in that city. Also, Philadelphia had really slow traffic within Level3 (and slower to all the others), and had major problems connecting to Verio. San Diego also had some problems, especially within the Level3 network. St. Louis was the only area without major problems...

For a breakdown, check out this view of the data.

Re:Call me silly?

[w4pso]

My understanding is that a Tier 1 ISP is now defined as one that has established free peering points with all other Tier 1 ISPs.

Considering that free peering is likely only established between 2 networks that have close to a 1:1 bit exchange, this is a very high bar to meet.

Re:Guess not

[Dun Malg]

The scary thing is it makes you wonder is some terrorist who has intimate knowledge of how Tier 1 ISP's work doing a trial run in the middle of the night by knocking out Level 3 and Verio backbones so later they could try to knock out ALL the backbones in a co-ordinated terrorist attack. (eek!)

Oh please. You know, it's pretty easy to figure out if it's something likely to be attempted by terrorists or not. The simple test is does it cause mass "terror". As annoying as it might be, lack of internet access is an annoyance. Perhaps a very expensive and exasperating annoyance, but it won't cause mass terror. Terrorists prefer things like bombs, or poison gas, or disease. Some other things people get worked up about but terrorists are unlikely to attempt: sabotaging bridges and tunnels to cause traffic jams; sabotaging electricity distribution to cause blackouts; sabotaging railroad tracks, making commuters late for work!. Think DEATH, not irritation. Quit with the automatic "terrorist hysteria" already, people!

Re:Nitpick - gang of thugs

[Kenshin]

It doesn't speak much for the Slashdot community when Wikipedia has to put this warning at the top of a Slashdot-linked page:

This article has recently been linked from Slashdot.
Please keep an eye on the page history for errors or vandalism.