Tier One ISPs Dying

xbmodder writes "Two tier one ISPs are down today. At about 23:30PST both Verio and Level 3 starting having problems with routes. According to Level 3 this is a software upgrade gone awry. Is this the end for Level 3?" Many, many reports about this are coming in, and if you're wondering why the stories were rather sparse overnight, it's because it's difficult to post them without internet access. Hope everyone else is back online too.

Comments

Could be good

[sdo1]

Maybe I'll get some work done today for a change.

-S

Re:Could be good

[malhombre]

And that, sir, is an excellent sig.

Flicker

[Oculus Habent]

Is there a term for this kind of intermittant site inaccessability due to Internet outage -- not the user or the server being offline, but the Internet failing?

Re:Flicker

[Taladar]

You mean something like "connection loss"?

Re:Flicker

[Cerberus7]

Yeah, it's called an "Oops!" Now somebody needs to make an acronym for this type of event using those letters. The two "O"s next to each other are giving me a hard time.

Re:Flicker

[Kalear]

Internetwork outtage?

In other news, Internet Health Report looks ok to me at the moment & I didn't see "intermittent" anywhere in the article.

Did somebody say the sky was falling? o_O

Re:Flicker

[EvanED]

Outage of plentiful service?

Re:Flicker

[Vo0k]

Luckily, seems no. At least this happens way more rarely than a Broken Arrow to get a name.

Re:Flicker

[aamcf]

Online to Offline Provisioning Situation?

Re:Flicker

[Wizzo1138]

I think that's an Obligatory Offensive Partitioning Scheme

Re:Flicker

[aamcf]

Actually, I think calling it an Online to Offline Provisioning Suituation in the Internet Environment would be better - "Call the engineers, there's been an OOPSIE!".

Re:Flicker

[Undergrid]

Try the report for the last 24 hours, not the big yellow blocks.

http://scoreboard.keynote.com/scoreboard/Main.aspx ?Period=RH24

Re:Flicker

[Syberghost]

It's a bit like a dead dog; it doesn't matter what you call it, it ain't coming.

Re:Flicker

[Ando[evilmedic]]

Kill yourself.

Re:Flicker

[Mr Guy]

Close, Outage Of Persistant Services

Re:Flicker

[sadler121]

*breaks aamcf's lower virtbrea, and pulls out katana, holding it out in front of aamcf*

"It is a good way to die"

Re:Flicker

[aamcf]

Heh, I had that nerve cluster removed because of a war wound.

Re:Flicker

[StikyPad]

I dunno, but for the first time ever, thousands of users were correct when they called tech support to claim that "the internet is broken."

Outtage Explained

[MMyers5]

It's nice to see something explaining why I was paged at 2:30am. And now, to whom from Level3 do I send my bill?

Re:Outtage Explained

[SRoberts7758]

HERE HERE!!!

Re:Outtage Explained

[mikeswi]

LOL...

I have to admit, my site went away for a few minutes last night. And I panicked and put in an emergency ticket. And so did a few hundred other people apparently.

I don't imagine anyone at my web host slept much last night.

Re:Outtage Explained

[Anonymous Coward]

It's spelled "Hear! Hear!"

Call me silly?

[Killjoy_NL]

But what is a tier 1 ISP?

Is that like a bandwidth wholesaler or something?

Re:Call me silly?

[rlanctot]

First Po..12qwr13652y235y1@$%154[NO CARRIER]+++

Re:Call me silly?

[rylin]

One that doesn't lease their infrastructure.
Eg. you have your own large backbone, you own all your equipment.

In effect, a small and wholly owned internet that peers with other internets.

Re:Call me silly?

[CvD]

Tier 1 are the huge ISPs, which peer with eachother (and don't pay eachother transit fees) and sell transit services to smaller ISPs (which do pay fees to send traffic through the Tier 1 ISPs). So yeah, bandwidth wholesalers is pretty accurate. See this wikipedia article.

Re:Call me silly?

[Killjoy_NL]

Thanks :)

(hehe he said internets, George is that you?) ;)

Re:Call me silly?

[Killjoy_NL]

Thank you too :) I understand now :)

(How could I forget to check Wiki)

Re:Call me silly?

[Anonymous Coward]

1. (Lower case "i"nternet) A large network made up of a number of smaller networks.

2. (Upper case "I"nternet) The largest network in the world. It is made up of more than 100 million computers in more than 100 countries covering commercial, academic and government endeavors. Originally developed for the U.S. military, the Internet became widely used for academic and commercial research. Users had access to unpublished data and journals on a variety of subjects. Today, the "Net" has become commercialized into a worldwide information highway, providing data and commentary on every subject and product on earth.

Re:Call me silly?

[nharmon]

Yes, internets for the 11th time.

Perhaps you've never heard of Internet2.

Re:Call me silly?

[Anonymous Coward]

In 1994, The National Science Foundation (NSF) awarded contracts to replace the National Science Foundation Net (NSFNet) Internet backbone. These contracts were for backbone transport, routing arbiter and traffic exchange points (NAPs).
          These contracts were awarded for the original 15 NSF sponsored NAPs, and to become a Tier 1 ISP, you had to have atleast DS3 connectivty to all 15 NAPs.
          It's a very old and crappy definition, and I wish people would stop using it, because it is very easy to meet now adays, and most of those original NAPs are now insignificant, compared to the power of the force.

Re:Call me silly?

[jasongetsdown]

Thanks for posting a perfect illustration of the quality issues at Wikipedia.
Don't you think that article is sort of a cursory treatment given the importance of tier ones? Also clearly written by someone with some kind of grudge about big telcos.

Re:Call me silly?

[CvD]

Yeah, the article isn't very good unfortunately. But I guess no one with the adequate knowledge has taken on the article to make it better. It does sort of summarize what the Tier 1 ISPs do, though.

Re:Call me silly?

[w4pso]

My understanding is that a Tier 1 ISP is now defined as one that has established free peering points with all other Tier 1 ISPs.

Considering that free peering is likely only established between 2 networks that have close to a 1:1 bit exchange, this is a very high bar to meet.

Re:Call me silly?

[petermgreen]

couple of things:

1: many tier 1 isps are involved at the retail end as well to some degree
2: many isps that could be considered bandwidth wholesalers are not tier 1.

Re:Call me silly?

[saintlupus]

useless sig advice - Read Nabokov.

I want to be Pnin when I grow up.

--saint

blah, blah, blah, I type too fast, blah blah

Re:Call me silly?

[jusdisgi]

That's not accurate. Lots of tier 2 and lower providers own their infrastructure. The important qualification of being a tier 1 ISP is that they don't pay anyone else to exchange traffic with them. The tier 1 guys are all predicated on the idea that they are huge enough that none of the others of them can afford to not have good and direct peering with them. Level3 can't afford to not be peered with MCI, and MCI can't afford to not be peered with AT&T, etc. So they all peer for free with each other. The tier 2 providers pay somebody to exchange traffic.

Also, the description of this story is probably also wrong; cogent isn't a tier 1 provider. Most sources seem to think (although the contract negotiations are confidential) that cogent was already paying Level3 for their peering, but that Level3 decided they wanted more money, based on the amount of traffic they were moving and which direction it was going.

But anyhow, your description doesn't work...the ISP that I used to work for that had 2000 customers would have qualified as tier 1 by your definition.

Re:Call me silly?

[lgw]

Also, the DoD has a seperate Internet, and I'm sure the NSA does too, given the size of their academic community. That makes 4, at least, worthy of the capital "I". Not that Bush knew any of this, I'm sure, but truth is a defense against pendantry!

Re:Call me silly?

[jasongetsdown]

I just checked and it has improved to what I would call a satisfactory degree. I'm sure the Slashdot exposure played a role there.

Re:Call me silly?

[porky_pig_jr]

This is not correct. Tier 1 ISP has nothing to do with leasing or own telco. I have worked for Tier 1 ISP which did not own any of its telco lines. Everything was leased from different companies: MCI, AT&T, GTE. (hint: that ISP had AS 1).

The way Tier 1 ISP is defined is mostly by its magnitude. At the time I've worked for that ISP, the rough rule of tumb was that Tier 1 ISP must have a few large capacity pipes from coast to coast at least. Must carry enough traffic so other Tier 1 ISPs can exchange the traffic (peer-to-peer) with this entity. Not strict rules as you can see, but in reality it works well.

Re:Call me silly?

[Igneous]

Bush probably didn't know, but I'll bet Gore did, being the inventor 'n all...

Re:Call me silly?

[psiphre]

the DoD has at least three separate internets, one of which "hangs off of" the Internet, but can function wholly without it: NIPRNET (unclass), SIPRNET (classified, secret), JWICS (classified, top secret).

Re:Call me silly?

[lgw]

Is the network informally referred to as "MilNet" part of NIPRNET? With the recent outage in the commercial internet, it's comforting to recall that some of "the internets" really are still designed to survive nuclear attack.

What is this about?

[PlatinumX]

An ISP's server being down 1 day is unacceptable of course, but to say it is dying already? or is there more to these ISP's? (haven't heard of them before)

Re:What is this about?

[neosake]

This is not _just_ their server being down, this is the entire network of two tier 1 carriers.

The (basic) implications of this is that a good chunk of the internet as a whole is inaccessible to the rest of the internet.

Re:What is this about?

[PlatinumX]

Doh, my bad. That explains of course then why my connection was dreadfully slow today between 9 am and noon. I am in the netherlands so the effect of this is grand yeah.

Re:What is this about?

[raddan]

We're not talking about just a server. We're talking about the entire ISP's networking capability. Tier 1 ISPs own huge swaths of networks-- literally miles and miles of cable, and sometimes radio and other links. They route the traffic across these lines.

When a Tier 1 provider goes down, their customers go down too. That picture on the Boing Boing page shows a list of the Tier 1 providers. Every ISP that is NOT a Tier 1, gets their access from a Tier 1.

People speculate that Level 3 is dying because they've been making some really bad decisions lately, resulting in a lot of outages. A couple of weeks ago, they actively filtered out traffic from their competetor, Cogent, over a dispute from how much to charge at the point their networks exchanged traffic (called a 'peering point'). Now this. The rumor is that the company is in financial trouble.

Re:What is this about?

[Alioth]

It's not a rumour that Level 3 is in financial trouble - it's clear for all to see. They have crushing debt repayments right now.

The Cogent spat isn't over yet either - Level 3 are going to de-peer Cogent again on November 9th. They are trying to force Cogent to pay for transit, but right now it looks like Cogent holds the strongest hand and Level 3 will be once again forced to back down.

Re:What is this about?

[c0bw3b]

wow that article is really bad. it says a lot about what a Tier 1 carrier is not, but not a whole lot about what one is. Hopefully someone with more knowledge will re-write that.

Re:What is this about?

[Jugalator]

Tier 1 ISPs own huge swaths of networks-- literally miles and miles of cable, and sometimes radio and other links. They route the traffic across these lines.

More precisely, Level3 seem to own 23,000 miles of optic fiber. :-)

The rumor is that the company is in financial trouble.

Yeah, not so much of a rumor anymore either -- Level 3 loss widens.

Re:What is this about?

[senor_jt]

Finally -- somebody that gets it. No offense to others who didn't feel like posting... As dissapointed as I am by the level of discussion about this topic in Slashdot, I'm thankful that it's here! I tried a search for DNS and DNS problems on Google news this morning and didn't come up with any stories, then tried Slashdot. And viola!! I wasn't going crazy at 11:30p PDT last night, the Net routing was... having problems. DNS service to wide swaths of the net was down/unreliable. I had to try a mix of different nameservers to get to sites(work, personal mail, etc...). Thankfully, I was too tired to worry about my clients and hoped it would all be solved by morning. Yay. But this event does underline a topic seen on Slashdot(and other esteemed zines) many times before -- the fragility of the Internet. Not even close to bulletproof.

Re:What is this about?

[shokk]

Yeah, that's it. They're dead. You pinned it right on the nose.
They won't be back tomorrow. All gone.

Reality check: An internet outage, no matter how big, is no different than a power outage. Yeah, here in the US we would be talking about loss of power to both coasts with only the middle left running. But after the outage life goes on.

Re:What is this about?

[Viper Daimao]

So the EU wasnt kidding!

Re:What is this about?

[petermgreen]

Every ISP that is NOT a Tier 1, gets their access from a Tier 1.

true in a sense but highly misleading. A large tier 2 isp is likely to have uplinks to MULTIPLE tier 1 providers as well as many peering links of thier own.

Re:What is this about?

[HavocBMX]

The reason that Level 3 isn't happy with the peering arrangement currently is that it's not even remotely even. Level 3 sends almost nothing over Cogent's network and Cogent sends over a vast majority of their traffic through Level 3. A peering agreement is based on the premise that the companies will be sending almost equal amounts of traffic through each network. Level 3 has been analyzing that for a time now but the last straw was when Cogent had a sales blitz targeting Level 3 customers saying that they would dramatically drop their prices to almost nothing to get them to switch away from Level 3. They are now also using the downtime that was experienced due to the peering problem in their advantage even though Cogent is in the wrong. Cogent knew about the depeering and did nothing to resolve it.

Re:What is this about?

[Alioth]

Oh yes, I'm aware of all of that - but (generally speaking) Cogent has the content, and Level 3 has the users. Guess who catches the most heat from the de-peering from its customers - Level 3 - as their customers will tend to see the problem first.

I predict that Cogent will do the same again as well - not lift a finger to fix the problem when they are de-peered on November 9th, and Level 3 will probably end up being forced to re-peer as customers whine that they are not getting the whole Internet and threaten to take up Cogent's free 1 year offer.

Re:What is this about?

[mr. methane]

Not really, but close. A peering agreement is based upon the premise that there's some business advantage to both parties. I've negotiated agreements with companies - where it was very clear ahead of time that the traffic flow would be hugely lopsided, like 10:1 easily.

In these cases, the companies wanted to get traffic, usually web content, to end users. They could either pay a company like level3 to carry it, or I could get it to a big pool of users. Long story short, both companies save money. The content provider is happy, the customer is happy, and I get a paycheck so I'm happy too.

As with any other business, the market leader will seek to use their scale to make it difficult for other companies to do business. UUnet did the same thing way back when, and the panic and imminent death of the internet was predicted in exactly the same way.

Instead, the cost of bandwidth continued to drop; while Bernie was cooking the books, Level3 and others were busy picking up cheap rights-of-way, and signing up big customers who were willing to give up the UUnet brand for a much better price from what was then a no-name called Level3. But now Level3's business isn't scaling.

This isn't the first time Level3 has pulled a class-A faux pas causing huge outages, and it probably won't be the last. In the meantime, the discomfort that both media companies and end-users are feeling is just the sound of opportunitiy knocking for providers who have better reliability or more competitive pricing. How long would Sprint have lasted if AT&T decided to compete instead of just jacking up long-distance prices on a shrinking customer base of people who didn't know better?

Showing solid green now.

[micronicos]

See http://scoreboard.keynote.com/scoreboard/Main.aspx ?xAxis=Destination&yAxis=Origin&zAxis=Metric&nAxis =Period/

Re:Showing solid green now.

[Professor_UNIX]

I guess they decided not to go out of business today afterall. Imagine that.

Guess not

[springbox]

Take a look at the scoreboard now. The mentioned problems are gone and Level 3 is no longer in the red.

Re:Guess not

[Kalear]

Hey, that's exactly what I just said! Anyway, it's prolly just a photoshop. :P

Re:Guess not

[ticklejw]

No, as of right now anyway if you look at the 24-hour average instead of the 1-hour average, you can still see some yellow left over from whatever happened. (I was a little skeptical myself... but the pattern is too unmistakable to debate.)

Re:Guess not

[680x0]

Select the 4-hour or 24-hour time periods, and you'll see the average latency is still in the yellow, so clearly there was a problem, but it's gotten better.

Re:Guess not

[Jugalator]

Umm, Wikipedia was down due to this earlier today, and some even had trouble reaching Google.
Level3 simply disconnected from Cogent causing parts of Internet to partition.

Re:Guess not

[LurkerXXX]

Uh no. Level3 disconnected from ALL the other Tier1 ISPs. So did Verio. This had nothing to do with last weeks Level3/Cogent spat.

Re:Guess not

[Jugalator]

The history showing a great disturbance in the force still remain at Internet Traffic Report though.

Re:Guess not

[Jugalator]

Ah, sorry, managed to confuse myself with the news and you're of course right. ;-)

Re:Guess not

[tomjen]

Well then i hope he strikes soon, so i can kill this slashdot habit of mine.

Re:Guess not

[MtViewGuy]

I honestly think it's just Teir 1's growing too big for their briches LOL.

I wished it was that funny but alas, it tells us that if a terrorist is really determined they could cause unprecedented havoc if they start to interfere with the Tier 1 ISP backbones.

I remember reading about some hackers at the DefCon convention a few years ago who claim they could come close to completely shutting down the Internet; given what happened last night showed it may not be as far-fetched as people think. :-(

Re:Guess not

[mexter2005]

This is highly unlikely. Most modern day terrorists rely on the Internet, so knocking it out just doesn't seem in their best interest. - ME -

Re:Guess not

[EpsCylonB]

Think of it, the horror of not be able to access the internet for an hour or two.

You realise you are more likely to get run over today than you are to have any kind of contact with a terrorist ?. Where is the mass hysteria over cars ?.

Re:Guess not

[Dun Malg]

The scary thing is it makes you wonder is some terrorist who has intimate knowledge of how Tier 1 ISP's work doing a trial run in the middle of the night by knocking out Level 3 and Verio backbones so later they could try to knock out ALL the backbones in a co-ordinated terrorist attack. (eek!)

Oh please. You know, it's pretty easy to figure out if it's something likely to be attempted by terrorists or not. The simple test is does it cause mass "terror". As annoying as it might be, lack of internet access is an annoyance. Perhaps a very expensive and exasperating annoyance, but it won't cause mass terror. Terrorists prefer things like bombs, or poison gas, or disease. Some other things people get worked up about but terrorists are unlikely to attempt: sabotaging bridges and tunnels to cause traffic jams; sabotaging electricity distribution to cause blackouts; sabotaging railroad tracks, making commuters late for work!. Think DEATH, not irritation. Quit with the automatic "terrorist hysteria" already, people!

Re:Guess not

[fimbulvetr]

Were they discussing the flaws in using AS numbers? You got me curios now;)

Re:Guess not

[Thangodin]

This is a little too high tech for terrorists, too many moving parts and unknowns. 9/11 was actually quite low tech, using box cutters and people with western passports. It worked because it was simple and unexpected.

Besides, shutting down the internet would shut down all the terrorist networks as well--there's not much more to them then a few isolated cells connected through the web. You could probably fit the leadership of all these groups onto a single bus. Shut down the internet, and they lose most of their means of recruitment, and these guys are nothing without the young cannon fodder they sucker into being the next walking bombs.

Re:Guess not

[jemecki]

I'm actually taking a class right now that addresses questions such as whether there is a terrorism threat on the internet infrastructure, and generally addresses the relationship between terrorism and technology. The lectures are posted online if anyone is interested. It is a four-campus class between UW, UCSD, Cal, and Microsoft Research. link

As an interesting aside, they might get Butler Lampson to give a lecture on computer forensics...

Re:Guess not

[sfurious]

"Terrorist" is a name assigned by people. It isn't necessarily a helpful one to try and split into component parts.

"Terrorists", for the most part, don't directly care about causing terror. But they'll use it in an attempt to further their aims. If they feel that destroying a country's infrastructure will further their aims, do you *really* think they'll stop and say "wait, this won't induce terror, can't do that"?

(as an example, IRA (generally considered to be a terrorist group) bomb hoaxes, whilst having some element of terror involved, were much more about mass disruption than anything else)

Re:Guess not

[ink]

Microsoft Research comprises a campus now?

Maybe they should include the FBI Campus and the RIAA Campus too.

Re:Guess not

[Dun Malg]

Make the people in charge look like fools that cannot keep something as simple as say an internet

I have highlighted the major flaw in your reasoning above for your convenience.

You make those people wonder about stuff like that, they'll get scared.

Scared? Unlikely. Try "angry". Fear of death is the top fear. One random bomb on a SUBWAY will strike fear in the hearts of most subway riders for months after. An internet attack would have people pissed off at the tier 1 carriers and the government. What is there to fear? Most people remember life before the internet, and they remember that they didn't DIE from the lack of it.

huh??

[Anonymous Coward]

Why would a software upgrae going wrong be the end of a gigantic Tier-1 ISP?

Re:huh??

[PhraudulentOne]

Because maybe they were upgrading their routers? And the "hitless failover" on their routers may not have worked so hot - or there was something else wrong with the code (new command syntax, so their configs didn't work when rebooted, etc, etc). Since about 98% of the equipment is routers for a Tier 1 ISP, a code upgrade gone awry could cause some crazy problems.

Re:huh??

[John Nowak]

Don't you get it? They deleted the internet!!

Re:huh??

[psyon1]

Good thing I've got my copy right here on the internet. I should mail it to them, then they could restore their systems from it.

Re:huh??

[LurkerXXX]

Maybe because for one of the gigantic Tier-1 ISPs, this comes right on the heels of another deliberate outage where they disconnected from another Tier1 provider with no warning to customers of either. On top of that, in a couple weeks they intend to do the exact same thing again. I think many Level3 (and Cogent) customers might be much more likely go go with one of the other 10 major providers rather than one involved in constant spats with others, or ones who disconnect entirely from the rest of the Teir1's from bad software upgrades. I know several companies who are planning to change from those two once their current contracts run out. I'd be shocked if there weren't lots more considering it or doing it.

Re:huh??

[Anonymous Coward]

At least be informed when you spew crap like this. L3 *did* notify it's customers prior to the Cogent disconnect. Several weeks before actually. Some of those customers requested extra time to work out their routing beforehand, and L3 delayed the disconnect at their request for more than 3 weeks. Cogent, which has had similiar peering issues with AOL and France Telecom in the recent past, chose not to pass that information on to its customers. Level 3 issued press releases to that effect, which you failed to notice because you'd rather continue the silly rant that makes for better discussion on Slashdot. http://money.cnn.com/services/tickerheadlines/prn/ 200510071811PR_NEWS_USPR_____LAF057.htm

Re:huh??

[Soldrinero]

No no no! It wasn't deleted at all! It's my fault, actually. You see, I just had to press this shiny red button that I found. My mother always told me not to push buttons I didn't know anything about, but did I listen? Sigh...

Re:huh??

[petermgreen]

btw what happened about the cognet/L3 issue in the end?

did they re-peer (possiblly under different terms to before)?
did cognet start buying transit to L3?
are they still disconnected?

Re:huh??

[LurkerXXX]

Level3 started to re-peer again. Temporarily. They will de-peer again on Nov 9th. Users of Level3 and Cogent have until that time to make plans for alternate hosting/routing. Basically a bunch of people spent a bunch of money to get multi-homed since Level3 and Cogent want to play games. Others who didn't spend money are going to be very unhappy again after the 9th unless Level3 and Cogent playing nice.

Re:huh??

[Gondola]

It's been a while, but I would estimate that there are a few more switches than just 2% of the network.

A switch problem, however, would just take stuff down completely, so you are very likely correct in saying that it was a router problem.

There have been times where a single customer announcing the entire internet through their connection, and those routes being re-announced (bad BGP filters) can cause this kind of havoc (not naming names to protect the guilty, *cough* AS4200 *cough*), but any tier 1 ISP worth its salt would be filtering those kinds of third-party route announcements.

First Post And...

[ploss]

Only a handful of posts and everything's back online again. See for yourself here (link from TFA)...

http://scoreboard.keynote.com/scoreboard/Main.aspx ?Login=Y&Username=public&Password=public[coralized ]

Seems like a non-event.

Re:First Post And...

[Anonymous Coward]

"Level(3) had an OSPF failure in Chicago, that is resulting in nationwide failures across the US and parts of Europe. They are currently working on restoring service in all markets."

Good posting
http://www.dummocrats.com/archives/001174.php

Rumors are it started out with L3 but Verio couldn't handle the load at first.

About a quarter of the people i know lost internet

[packeteer]

Today i was playing world of warcraft and on our raid about 25% of my guild mates lost their internet on and off. Other than that the lag was higher than normal but i wondered what the hell was going on. Anyway we still pwn some dragons in BWL :)

No wonder...

[arof]

No wonder I couldn't get through to /. through the rss.

I actually had to go directly to the front page for the second time in my life.

Looks fine now

[mojo17]

The Internet Health Report cited in TFA shows all green now. It looks like whatever problem they had is solved.

Re:Looks fine now

[smokeslikeapoet]

You can always get the daily traffic report here:
http://www.internettrafficreport.com/main.htm

From the packet loss graphs you can see when the outage occured. Yes, everything looks back to normal.

Re:it's difficult to post them without internet ac

[EvanED]

I prefer carrier pigeons.

Noticed it this morning

[discord5]

Noticed this this morning when a customer called upset about his hosting services being unreachable. A quick traceroute showed one of level3's ip to be down. A few minutes later more customers had problems with different routers from level3. As soon as I saw level3 I knew enough, shrugged it off and told the customer that it was routing problem we couldn't fix but those responsible were most likely already trying to fix it.

It seems fixed now though, so no, this isn't the death of the Internet just yet.

Re:Noticed it this morning

[Winkhorst]

This is precisely why my webhost is multi-homed with over 150 carriers.

Re:Noticed it this morning

[Qzukk]

But if your customer is hosted by an ISP who only uses L3, it's still Your Fault.

Tier 1's dying? OMG!

[Junior J. Junior III]

Has Netcraft confirmed this?

This just in...

[mtec]

The reports of my death have been greatly exaggerated.
Tier One

Re:About a quarter of the people i know lost inter

[ShadowFlyP]

I'm not sure which amazes me more: that the only people you know are the ones you play WoW with or that "pwn" has become some kind of short-hand for "0wn3d".

point of the internet?

[elementik]

wasnt the internet envisaged so that this specifically could NOT happen?

Re:point of the internet?

[Detritus]

Nope. Redundancy and reliability cost money. Fast, cheap, reliable, pick two. Take a look at a typical network and count the single points of failure. Then there are common mode failures, like bugs in router software, that can take down entire networks.

Re:point of the internet?

[DrSkwid]

The internet was never "envisaged".

That it can't die is 90s hype

Re:point of the internet?

[Seraphnote]

Yes and no.

Yes, the Internet enables/permits/allows redundant routes, but...

No, it doesn't require/demand/"enforce with any government or legal authority" redundancy at all levels.

So any smaller ISPs connected to Level3, and all their customers would have had problems reaching the rest and being reached by the rest.

(sarcasm mode)Obviously this wouldn't have happened if the EU had been in control!(/sarcasm mode)

Actually, how many of these corporations are US companies, and how many are NOT?

Re:point of the internet?

[IANAAC]

Do you know what the word means?

Re:point of the internet?

[DrSkwid]

hehe I guess

the one in the vision isn't the one before us was more my point

True vision would have not used IPv4 =)

Re:point of the internet?

[hey!]

I suppose it depends on the kind of geography you're talking about. The kind of failure here isn't like a bomb blowing up a single NOC, it's organizational in scope and therefore potentially a distributed failure.

I suppose the Internet isn't that immune anymore to a misplaced backhoe digging up a single fiber link either. If we all used it mainly for telnet and email like it we did the ARPANet, it'd probably be fine, but the level of traffic (net bandiwth) and kinds of traffic (e.g. VOIP) it's expected to handle are totally different.

Re:point of the internet?

[MajorDick]

No, not really, it was inteded so chunks could go down (zapped with a 100 megaton nuke) and the rest would remain up.

In this sense it functioned perfectly.

Re:point of the internet?

[mrchaotica]

I thought the entire point of the Internet is that they picked reliability. You know, "national security" and all that...

Re:point of the internet?

[Lehk228]

the internet can route around damaged areas but if every connection is severed there is no architecture that can save you

Re:point of the internet?

[Geoff-with-a-G]

I thought the entire point of the Internet is that they picked reliability.

Then your first problem was assuming that "the Internet" could be represented as "they". The Internet is an abstract concept, it's the interconnection of a bunch of different networks. Each of those networks is run by different people/companies/goverment-agencies, each of which has their own policies. Obviously they have to agree on a bunch of things in order to interconnect (ie, "let's speak BGP for routing, rather than this nifty routing protocol I wrote last night") but that doesn't mean they all share the same values and priorities (fast, cheap, reliable).

In addition to that, sometimes even when you prioritize reliability, it still fails you. You can dual-home your routers, but if they're both running the same code, they're both vulnerable to the same bugs. Now you decide to quad-home them, so you can run a set with different code, but they're all running the same hardware platform. Maybe the WAN modules included the same bad batch of ASICs. The "cost" vs. "reliability" graph grows exponentially, so it doesn't take long to stray into absurdity. It sounds like this particular outage was a case of "we designed it to be highly-redundant, but something happened which we hadn't anticipated."

There's really no way around that, you just try to minimize it.

Re:point of the internet?

[MKalus]

That would be Milnet these days.

Yes, the original idea was that, but when it became commerical the "nuclear strike" option wasn't really high on their thread assessment list.

IP does allow you to route around damaged areas still, but if the "Gateways" of one ISP all go down at the same time, their network IS down and there really is nothing you can do.

I have to wonder though what really went wrong there, did someone force an update of all the routing tables of all their core routers at the same time? Update the software on all the systems at the same time?

Seems a bit odd to me, I would have thought that they do a staged roll out of any of those changes, but apparantly that wasn't done.

TEOTWAWKI

[D4C5CE]

Is there a term for this kind of intermittant site inaccessability due to Internet outage -- not the user or the server being offline, but the Internet failing?

Yes. Domesday as predicted in the ancient scrolls. In this day and age it is commonly called The End of the World as We Know It.

Re:TEOTWAWKI

[maxwell demon]

Yes. Domesday as predicted in the ancient scrolls.

In which way are the domes involved in the internet failing?

Re:TEOTWAWKI

[Ripp]

The End of the World as We Know It.

Sure, but I feel fine.

*shrug*

Re:TEOTWAWKI

[Lucractius]

*calls the RIAA for stealing a line from a song*
SICK HIM!!!

Re:TEOTWAWKI

[erlenic]

They're part of a vast international conspiracy. The Karl Rove and the Catholic Church are plotting to weaken the defenses of every nation in the world so the aliens can take over without a fight. It's called the domesday conspiracy because the conspirators are going to hide under the dome of the Sistine Chapel when the invasion happens.

And no, your tin-foil hats won't save you. Better stock up on lead.

Re:TEOTWAWKI

[bodger_uk]

Domesday, http://en.wikipedia.org/wiki/Domesday is the middle English spelling of Doomsday.

Re:TEOTWAWKI

[Golias]

The End of the World as We Know It.

Sure, but I feel fine.

Me too. I have some time alone.

Re:TEOTWAWKI

[i wanted another nam]

Yeah, but they also spell "color" with a "u" so why should we trust them?

Re:TEOTWAWKI

[quarkscat]

I could accept "The End Of The World As We Know It", except that I couldn't get on-line to get confirmation from Netcraft.

What, exactly, does Netcraft have to say on the subject? (They are, after all, the premier internet authority...)

Re:TEOTWAWKI

[Clockwurk]

LEONARD BERNSTEIN

Re:TEOTWAWKI

[TheoMurpse]

calls the RIAA

You, sir, are a traitor. Hand in your geek card!

Re:TEOTWAWKI

[Lucractius]

*submits a pirate copy of his geek card for confiscation*

/.'ed

[connah0047]

Hey look, we slashdotted Level 3!

I noticed it.

[HoneyBunchesOfGoats]

I was up late studying for a German exam, and I was having problems connecting to websites hosted in Germany that I was using to help myself review (dict.leo.org and canoo.net, if you're curious). US websites worked no problem.

Off to the test!

Re:I noticed it.

[dr_dank]

This is like the 21st century equivalent to pulling the fire alarm in school in order to get out of taking a test.

I can't take the test, professor. The sites with the study guides were Slashdotted this morning!

You deserve an A for shrewdness.

Re:I noticed it.

[gentlemen_loser]

That's fine. But do you understand that I was booted from WoW???

Which is more important, really? ;-)

Good luck today.

Re:I noticed it.

[shish]

Exam? Bah humbug. I couldn't get online to submit my university application, with today being the last day to get it done :/ They were saying "get it done early in case something bad happens", and I thought "Nothing can happen... unless the internet goes down. Heh, like that'll ever happen!"

Re:About a quarter of the people i know lost inter

[ckaminski]

It had to. So many game servers were being immature, and letting people say "Die Fucker", and "Eat Shit & Die", but if you say "owned", "0wned", you got booted...

Stupid, really.

Non event... for now

[elfguygmail.com]

While this only lasted a few hours, it still caused a mess across the North American Internet during those hours. The point is a small amount of big networks are responsible for over 90% of the traffic on the Internet. If alter.net went down it would be total chaos. If just one of the major peering points went down, sure the traffic would be rerouted, but overloading the other points at such high latency that it would be almost unusuable. You better hope no one destroys MAE-EAST or we'll have a live example of what ife without the Internet is like.

Re:Non event... for now

[Quixadhal]

You better hope no one destroys MAE-EAST or we'll have a live example of what ife without the Internet is like.

Yeah, I remember life before the internet. I used to read books, watch TV, and even occasionally go outside under that big yellow face.

*shiver*

Re:Non event... for now

[Lucractius]

id like to say why dont you just paint a bullseye on it while your at it. *Points to climate of fear and loathing against terroists*

Re:Non event... for now

[scribblej]

or we'll have a live example of what ife without the Internet is like.

Way to remind the rest of us we're old, young whipper-snapper!

Why in my day, we didn't have the Internet! You'd have to carry your packets to your neighbor's house so he could copy them to his computer, and that was networking. Of course without the Internet, we didn't have IP, we only had P. Thus the phrase, "I'm going to go take a P" -- as in "I'm going to take this data to my friend's house."

See, I bet they didn't teach you that in school. We old people know things!

Re:Non event... for now

[Anonymous Coward]

dude - have you been behind a rock for a decade? Mae-East is hardly the only peering point carrying US international traffic any more, and alter.net/uu.net/worldcom is not carrying nearly all of the net's traffic anymore.

Re:Non event... for now

[MegaFur]

Are there any good web sites out there that have current info on what networks are carrying what traffic? TIA

Re:Non event... for now

[Anonymous Coward]

even occasionally go outside under that big yellow face.

No! Not... The Big Blue Room!

Re:Non event... for now

[LilGuy]

You speak in blasphemous terms. Life without internet indeed! I tried it once... lasted 2 months. I traded my sanity in for a so-called "social" life. It wasn't worth it. Nobody, but NOBODY better mess with my interbits.

Re:Non event... for now

[Scott7477]

Some people call it the "scareball"....

Re:Non event... for now

[bitflip]

Fool! Its not a big yellow face!

THE MOON IS ON FIRE!!!!

Re:Non event... for now

[jistanidiot]

http://www.internettrafficreport.com/main.htm http://isc.sans.org/

Re:Non event... for now

[orkysoft]

That yellow thing is called the DayStar...

Re:Non event... for now

[Hell O'World]

That's no moon...

Re:Non event... for now

[EvilTwinSkippy]

Ach the Internet is down. I guess I'll play some Magic the Gathering... Nah. I haven't played a D&D campaign in years.

(Oh yes, there was a time when Geeks inhabited other subterannian habits. Now that I think about it, we always did play D&D in somebody's basement...)

Re:Non event... for now

[Arandir]

Maybe if the UN was in charge of the internet, we could all sleep easier at night. After all, the UN is legendary for its efficiency, expertise and fairness.

Re:Non event... for now

[C0rinthian]

Your star, it burns!

Re:Non event... for now

[droptone]

I guess they didn't have closing tags for HTML back in the day either ... ;)

Re:Non event... for now

[ankhank]

> before the Internet

What, you weren't using FidoNet?

Hmmm, maybe we _should_ all practice, one day a year, the old system of store-and-forward. Pull out the modems, pull out the dialup numbers, pull out the text-only interfaces.

I wonder if anyone ever wrote an ASCIIsemaphore interpreter.

As cheap as hardware is these days, we could probably put a mechanical semaphore network up -- line of sight, with telescopes and image recognition software -- in a few weeks.

Plan ahead for outages ...

Re:About a quarter of the people i know lost inter

[DrSkwid]

pwn is what the kids say these days,

To them own3d is something your dad would say.

Get with the times, daddio =)

Link in the article

[Alphabet Pal]

I notice the article links back to Slashdot... I wonder is Slashdot is going to get BoingBoing'ed?

Re:PST?

[dwayner79]

GMT -8:00

Re^2:TEOTWAWKI

[Anonymous Coward]

http://dictionary.reference.com/search?q=Domesday

Are ATMs using Tier1?

[Ruvim]

I wonder, can this outage affect Banks ATMs and if it would, who would be held responsible for people not being able to get their money and all the problems originating from it?

Re:Are ATMs using Tier1?

[mattnuzum]

Uh, everyone uses a Tier 1. The only way you can not use a Tier 1 is if you access one of your ISPs web servers. Otherwise you get routed through a Tier 1.

Are you asking if ATMs use the internet, then, yes, some do. Slammer a few years ago took out Bank of America and others. That was the day the Internet felt like it stood still.

The biggies learned their lesson and now should survive. I suspect the small guys, your average mall/convienence store ATM, would go black and start spewing money across the floor.

Or maybe they'd just go black.

Re:Are ATMs using Tier1?

[jtev]

Yeah, but most ATMs use dialup or a dedicated connection to the bank that owns them as their connection. They don't communicate with their home bank using a Teir 1. They MIGHT use a Teir 1 for comunicating with other banks, or several Teir ones, or they might simply have a banking only internet that connects all the banks, and is connected to the public internet. Or isn't realy connected to the public internet. anyway I'm starting to get less coherent, so I'll go back to my corner and shut up now.

What kind of Timeframe

[PhraudulentOne]

Is this even an issue? I mean, this was probably scheduled maitenance that went a little longer than expected. I have been through this before. It just sounds like Level 3 dropped some core routers for a few minutes to do a code upgrade - it didn't work so hot, so they were down for a few more mintes, OSPF/BGP decided to tell all the clients that they have no routes, Level 3 gets the routers back up, OSPF/BGP tells everyone that their fine again. Was this like 6 hours, or 45 min?

Re:What kind of Timeframe

[Jugalator]

It was maybe 2 hours or so before new routing tables started spreading to bypass Level3's and Verio's networks, and afterwards it started stabilizing again, then it seems Level3 has since then woke up again. The XO network also had routing troubles from this btw, maybe more too. Sites and services such as AOL, SpeakEasy (when asked, they were stumped and could only say it affected all their customers, hehe), Google, and Wikipedia had access problems depending on where you lived during this timeframe.

Re:What kind of Timeframe

[JNighthawk]

My internet was down for 2-3 hours. It went down at around 2:30 AM EST. I checked again at 5:00 AM EST and it was still down, but when I checked again at 6:00 AM EST, it was up.

Re:What kind of Timeframe

[Anonymous Coward]

I dropped my BGP session to Level3 but they did not retract the routes, so not only could they not route my packets but they claimed (via the routing table) that they still could. From my vantage point (Chicago) the problem was resolved in about an hour,

Re:What kind of Timeframe

[Scuff]

I had the (mis)fortune to be working in a NOC for a web hosting company last night, and it turned out to be a period of 4 hours where some of our monitored systems would have flaky connections, they'd be down for a few minutes, then come back up, but probably go back down again a half hour later. Frustrating, yes, but it didn't take very long to determine that Level 3 was the issue. Trying to get a timeframe out of them as to when it would be fixed was much more frustrating, but was pretty much what I expected from them.

Re:What kind of Timeframe

[Pii]

This is an interesting side-effect of promoting BGP "stability."

Many times, ISPs will nail-up routes (pointing a prefix to the null interface with an undesirable administrative distance, and routing traffic based on the dynamically learned, and more desirable, route) to prevent flapping.

This keeps the entry in the router's table in the event that they temporarily lose the connection to actual origin of the prefix, and prevents the router from having to issue a withdrawl to it's remaining neighbors.

The behavior you experienced would be the downside of this practice.

Re:What kind of Timeframe

[ArtStone]

When I lost connectivity to my web server (from home on Adelphia cable - which routed through Level(3)), I attempted to connect via dialup to see if the problem was the network or my server crashing.

These days, a very very large part of dialup internet access is on Level(3) soft switches - AOL, Earthlink, Juno/Netzero and MSN all use Level(3) as their primary network - so do the wholesalers like GlobalPops, Ikano and DialupUSA (recently acquired by Ikano) that provide "nationwide" internet access to your hometown mom and pop ISPs.

Dialing into any Level(3) POP number got a fast busy - so probably most dialup access in the US was also down during this time (except California - which is "owned" by O1 and Pacwest). I eventually dialed into a ChoiceOne POP (a smaller CLEC/LD provider) and was able to connect and verify that the problem was just a connectivity issue (albeit a huge one)

History of this?

[jasongetsdown]

Is this the first time this has happened? Is it too early to start talking about re-thinking the way this is put together?

So much for the internet surviving a nuclear war!

[Viol8]

Now it can't even survive a software upgrade on some of the routers!

Is the Internet Down?

[PortWineBoy]

Why couldn't this have happened during my business day? For just once when a user calls and asks "is the internet down?" I'd like to be able to say "actually, yes, it is."

Re:Is the Internet Down?

[RPoet]

As in "Yes, sir, the Internet IS down."

call for distributed infrastructure

[Anonymous Coward]

This should be a wakeup call to keep moving ahead with creating a more distributed and resilient Internet infrastructure. The first step in that direction are wireless neighborhood mesh networks.

Re:Isn't God trying to tell us something?

[m50d]

People have been able to say something like that at every point in history. And I'd hardly call this nastier than hurricanes, and the Tsunami was worse than either them or this. The sky is not falling.

Mod parent funny

[Zwets]

Insightful? I certainly hope comparing a short Internet outage to large disasters is a joke..

Looks good now...

[Tuna_Shooter]

Seems to have cleared up. ahref=http://scoreboard.keynote.com/scoreboard/Mai n.aspx?Destination=Veriorel=url2html-7588http://sc oreboard.keynote.com/scoreboard/Main.aspx?Destinat ion=Verio>

Re:Looks good now...

[afd8856]

First you fail to create a good link, and then that link goes to a login screen? Your link posting rights have been removed.

Re:Looks good now...

[EnronHaliburton2004]

Don't blame him! This was a software upgrade gone awry.

Re:Looks good now...

[kisielk]

You can login with public/public to see the scoreboard.

Re:bad for US

[Punkrokkr]

I was thinking this was the "or else" that the EU issued the US.

Re:About a quarter of the people i know lost inter

[germ!nation]

"lets go and kill some mobs" doesn't sound right... but I couldn't bring myself to say pwn, so I decided to just say prawn. much nicer.

Nitpick

[s20451]

Domesday is something like a census of Britain circa 1085. It has nothing to do with internet outages, which is more akin to doomsday.

Overlay Routing

[omnirealm]

This sort of event provides motivation for overlay routing schemes, which can compensate for major outages along various routes of the backbone:

http://www.usenix.org/events/nsdi04/tech/full_pape rs/subramanianOver/subramanianOver.pdf
http://www.eecs.umich.edu/~farnam/pubs/2005-hwj-in focom.pdf

Time in UTC

[GerritHoll]

23:30 PST = 07:30 UTC = 09:30 CEST

But perhaps what's really meant is:

23:30 PDT = 06:30 UTC = 08:30 CEST ?

slashdot slashdotted :P

[adinu79]

Hmmm ... was wondering WTF Slashdot wasn't working today. My route to slashdot goes through Level3.

shea right more like the CIA is "upgrading"

[Anonymous Coward]

More like the CIA is upgrading their equipment.

Over 100 comments so far....

[mblase]

...and not one Netcraft joke?

flapping

[SpectralDesign]

Way back in the day when I was a Network Controller at BBN Planet, if we began to have cascading routing outages we'd call it "Flapping"... Visualize a wounded bird squirming around on the ground flapping...

Takes me back... My first night on the job a rat in Berkeley chewed through the wrong cable and got himself fried -- he also happened to take the entire west-coast off the internet for the better part of a day.

Then there was the time an electrical worker got vaporized in a hole near MIT which caused quite a problem too as it overloaded the MIT power station, but the fallout wasn't nearly as bad as the day of the rat...

Re:flapping

[ScottKin]

I had a similar situation back when I worked at the Computer Center at LBL - mice chewing through the high-voltage supply for our CDC 7600Z back in '80 set-off the fire alarms...I was the lucky stiff who had the joy of holding-down the "HALON DEFEAT" deadman switch until they could find where the carbonized rat was.

Ah, those were the days.

--ScottKin

Re:flapping

[rah1420]

halon defeat

OT, but it brings back memories of working at Purolator Courier in the machine room. IBM mainframe shop.

We had had trouble with the damn fire suppression all day. On third shift, around 3 AM, the trouble alarm went off (again) for the umpteenth time. One of the operators, a nervous fellow who was a little bit green, went over to the annunciator panel and opened it to see what the Trouble Might Be.

A fire technician he was not, and he apparently didn't know the difference between the trouble bell and the klaxon that would sound when a halon dump was about to occur; so he reached around the open panel door and hit the halon defeat.

Or so he thought.

It was actually the Big Red Switch.

The whole room (full of 3420 and 3480 tape drives, the 3745s, the 3800 laser printers; and the floor above, containing trivial bits like the DASD and the CPU all plunged into a deafening silence.

We all stared at each other and at the newbie BOFHeck.

A few minutes later, the phone rang. It was the Indianapolis air hub for Purolator, wondering why (when they were about to receive about 150 planes from all over the country) they didn't have anything useful displayed on their green screens.

That was a fun morning.

Ah, those were the days indeed.

Re:flapping

[Anonymous Coward]

I was watching the looking glass's last night and just about every router and core router within Level3 was seriously Flapping, though I was under the impression that flapping was the equivelant of: Ok peers im Up, no down, oops I mean up, erm down, I mean .......

But If you float through the IRC channels on Linux.org there may still be ##level3 and #level3 with some people in there, you might be able to snag a copy of the log from last night from an op.. Best bet, ##level3 on irc.linux.org... Otherwise you might find some remnants form the night before in #fedora

Re:flapping

[llefler]

I used to work in a large datacenter for a mutual funds company. At a guess, the computer room was 200k sq/ft with about half of that 3480 drives and the tape library.

Every Sunday night they would switch from mains to batteries to exercise the system. So at around 1am the air conditioners and lights would go out and the silence would be deafening. It always made your heart skip a beat while you checked to make sure the lights on the drives were still on. 30 seconds later the lights and air conditioning would come back, but I never got used to it.

Oh, and I also worked in a Gov't datacenter for a while. So of course, Halon wasn't allowed. The VAXen were 'protected' by a sprinkler system. The disaster plan was for one operator to hold the sprinkler abort button while another pulled the t-bars and covered the machines with plastic. Then of course, I worked Sundays as the sole operator. Hmm, do you burn holding the button or get electrocuted pulling t-bars. Good thing we never had a fire, because I would have been listening for the explosion from my car.

Re:flapping

[sjcrash]

It wasn't in Berkeley. It was the Palo Alto/Stanford Forsyth Datacenter. So you worked at the NOC? I was at the WR-colo, always enjoyed the late nigth calls from you guys with the thick Bostonian accent.

Re:flapping

[belmolis]

How come Halon wasn't allowed in government installations?

Re:flapping

[llefler]

How come Halon wasn't allowed in government installations?

Halon contains bromine and contributes to ozone depletion, just like CFCs. When I worked there, Halon was being phased out, so no new installations where allowed to use it. The facility I worked at was a temporary data center for the 1990 Census.

Re:flapping

[bhiestand]

The decomposition from halon can also be rather harmful. It's still used for many government/military aircraft fire extinguishers, and I'm sure it's still being used elsewhere in the government.

Netcraft conferms it

[Anonymous Coward]

This joke is dead.

Re:About a quarter of the people i know lost inter

[hobbit]

It's amazing what these crypto experts can come up with to stay ahead of the game. What's next -- qwned?

Verio?

[VikingThunder]

Well we seem to know why Level3 went down, but why did Verio go down at the same time?

Re:Verio?

[jaredmauch]

Oh, and incase anyone is wondering, one of the customer related things was in Houston, where Keynote indicates their one 'Verio' node is. Note how the Level(3) graphic is both horiz and vert, and the Verio graphic is only vertical. (Meaning it's only one of the sites, ie: the one in Houston).

At least in this case on the 'verio' side i know what it is, as far as Level3 goes, that's obviously something different.

The main problem areas...

[dantheman82]

happened in Detroit in the last 24 hours. Apparently all ingoing/outgoing traffic to other Tier One ISPs had problems in that city. Also, Philadelphia had really slow traffic within Level3 (and slower to all the others), and had major problems connecting to Verio. San Diego also had some problems, especially within the Level3 network. St. Louis was the only area without major problems...

For a breakdown, check out this view of the data.

Re:The main problem areas...

[OverlordQ]

Got a login for that?

Re:The main problem areas...

[mattnuzum]

I was up running traceroute at 3:30 am (CDT) this morning. At first they always stopped in ATT's network, presumably the pop between them and Level 3. When level 3's website came up (about 3:45) my pager stopped beeping but I still couldn't get a route to my servers. At that point the traceroute stopped at a POP for Level 3 in New York. The pings just kept getting bounced back and forth between two servers. All said and told, I was down for about 45 minutes. I use a service called "Web Site Pulse" that monitors my network/servers and their connection to my servers came up about 15 min before I could see them. (Normally that would seem like a bad thing, but in the case of a major tier 1 backbone being down, the normal scenarios all go out the window) It's kinda funny, cause it's always the new guys who get stuck doing night-time duty at the NOC where my datacenter is. I always feel bad because it seems like the worst problems happen at night-time. When I called there (it would have been about 4:15 am where he was) the guy was obviously freaked out.

Re:The main problem areas...

[SPY_jmr1]

While I do not have technical details to offer, I am in the STL net area, and at about 1:00-1:30 AM local time everything just went *KERBOOM*.

Random sites not working, huge times to get traffic back from the ones that did.

Yes, it was working, but very intermitantly.

This was connecting via charter cable, which routes all over the place, usually. Most things get sent to chicago, and I imagine they were having troubles as well.

In conclusion, St louis was having problems as well, ;-)

All IMHO, of course.

Re:The main problem areas...

[DFossmeister]

None needed

Re:The main problem areas...

[TheoMurpse]

I'd check out the data, but I don't have a login/pass.

Re:Isn't God trying to tell us something?

[erlenic]

Yes, that's exactly what this is. You better curl up into the fetal position in the corner and start crying.

Could really be good

[TheSpoom]

Maybe I won't be able to get some work done today ;^)

(/me has an entirely Internet-based job)

Microsoft?

[Frankie70]

Is there any way we can blame Microsoft for this?
Were they upgrading to one of the Beta builds of Windows Vista Home Edition?

You insensitive clod!

[Anonymous Coward]

I'm not back online

Re:my net was down earlier

[Jekler]

I was in the same boat. It was like 07:30 UTC or so, I was trying to play a game, but my ping was high and I'd intermittently get disconnected from it every few minutes. It was the weirdest outage I'd seen because I could get to some web sites, google, yahoo, excite (at first I thought it might just be cache so I went to excite which I don't think I've ever visited with this computer). But I couldn't get to slashdot, guildwars page, or update my antivirus. It was like my connection was down for anything except the major search engines. Even as of about 12:00 UTC my connection was still slow.

Verio?

[jaredmauch]

Can someone who has data on what the percieved "Verio" outage is let me know? Obviously Level(3) customers would have issues reaching Verio and Verio would have trouble reaching them but reviewing one of our external monitoring systems I only see 3 events and only one that is not customer related. So unless you're in that isolated corner of the network in Europe...

Re:Tier 1's dying? OMG!

[coolfrood]

They would... except that they're not accessible..

Level 3 started earlier than Verio

[DFossmeister]

Level 3 went down at 22:42 pst and was available around 23:50 pst. Verio started having problems right around the same time that Level 3 was coming back up. The Internet Health Report from Keynote showed me what was going on, scary that it was.

Re:Level 3 started earlier than Verio

[DFossmeister]

Bloody articles. Its actually Internet Health Report.

Fey.

Level 3?

[Greyfox]

Haven't they been hanging on by their fingernails since the dot-com bust? I've know a few guys who got burned working for them just before the bust, and I've seen several recruiters post stuff like "A local communications company (NOT Level 3!)" in their job reqs.

I don't know that they've replaced Sprint yet on my list of most sucktastic internet companies. Time was you lost connectivity to an important piece of the Internet (Like your favorite Quake TeamFortress server) and a traceroute would show the failure somewhere in the Sprint backbone. So far they've been more reliable than Sprint at their worst, at least for me.

If they go under, well Tier 1's don't ever really die. Chances are one of the other Tier 1's will buy their assets and it'll be business as usual. Usually the buyer is MCI.

Of course the true test is pretty easy -- has anyone who works at Level 3 had their paycheck bounce yet? Surely there are a few readers among their employees...

Re:Level 3?

[RPoet]

If they go under, well Tier 1's don't ever really die. Chances are one of the other Tier 1's will buy their assets and it'll be business as usual. Usually the buyer is MCI.

And with fewer and fewer tier 1's, is it any wonder things like this happen?

Drat!

[rleesBSD]

I can't tell you how many times I cursed my Windose box yesterday trying to figure this one out .... Now I have to take it all back ...

  - Turn to the West
  - Bow humbly
  - Beg forgiveness

FFS, what a fucking dreadful summary

[Cally]

Is this the end for Level 3?

No, of course not, you blithering imbecile. L3 had a 2 hour global routing meltdown. Now, it's fixed. Whilst their routes were flapping, other carriers saw transient increases in latency and some problems with reachability, to some sites. However, everything continued to work properly for non-L3 customers. Two hours later L3's routes are back and working properly. End of story, nothing to see here, move along please.

Slashdot editors, do you really expect us to believe that no-one had submitted a more coherent or accurate story than this one? Come on, for heaven's sake.

Anyway, a network engineer's view can be seen in the overnight traffic on NANOG: http://www.merit.edu/mail.archives/nanog/2005-10/ "Tier One ISPs dying" indeed. Worst. Story. EVER.

Re:FFS, what a fucking dreadful summary

[gimple]

Worst.Story.EVER.

"Like anyone can ever know that Napoleon."

Re:FFS, what a fucking dreadful summary

[Guppy06]

"No, of course not, you blithering imbecile. L3 had a 2 hour global routing meltdown. Now, it's fixed."

However, L3 has been having "issues" this month that have left a lot of lower-tier ISPs in the uncomfortable position of explaining to their customers "We know the internet is down but there's nothing we can do about it." This outage really can't be good for their reputation, and I can see more potential customers taking their money elsewhere because of this.

Just because the technical issues have been fixed doesn't mean their finances have been fixed as well.

Re:FFS, what a fucking dreadful summary

[darrylo]

"Tier One ISPs dying" indeed. Worst. Story. EVER.

Come on, this is SlashDot.

It should be:

"Tier One ISPs dying" indeed. \/\/0rst. 570ry. EVAR.

Re:FFS, what a fucking dreadful summary

[Profane Motherfucker]

I can't wait until the day when people stop quoting that worthless fucking shit of a movie under the assumption that it's funny. I've been waiting a very long time. And perhaps I spend far too much time with dumbshits.

Re:FFS, what a fucking dreadful summary

[pruneau]

Well, If you cared to read your own linked archives, you could read this:

* From: Alex Rubenstein * Date: Fri Oct 21 13:45:24 2005 Gary, I understand your statement, but I am sure the gentleman below does not. If you want a story to be done, so that the world can see how something like this can impact thousands of businesses, the best bet would be to help educate this guy so that he has something to write. Are, were you trying to scare him off from doing a story? Personally, I am quote fed up with the issues that the huge providers have and cause, yet never have anyone document it, find out about it, or do anything about it. I laud this guys effort for actually trying to do his job and expose something that needs to be exposed. I am now putting on my level-3 bullet proof jacket, and will be looking over my shoulder for the next 3 NANOGs.

Seems like someone want the media attention, /. is not the worst choice here. Of course, the sensationalism could have been avoided, I can grant you this. Does this warrant cursing, hmmm....

Re:FFS, what a fucking dreadful summary

[Cally]

However, L3 has been having "issues" this month that have left a lot of lower-tier ISPs in the uncomfortable position of explaining to their customers "We know the internet is down but there's nothing we can do about it." This outage really can't be good for their reputation, and I can see more potential customers taking their money elsewhere because of this.

Just because the technical issues have been fixed doesn't mean their finances have been fixed as well.

See now that's a story, if you add a couple of links to back it up. Not saying there's any problem with L3's finances, though of course we're all still waiting to hear the story behind the Cogent issue. Incidentally (as has also been discussed on NANOG recently) there's increasing pressure on not only ISPs but even corporate networks whose parent orgs are large enough to merit audits and certifications (think NIST, SOx, ISO17799,..) to start thinking that being multi-homed is a necessary precondition to really `being on the Internet`. (And who's to say they're not right?) One thing's for sure - demand for BGP-clued bodies with experience with 'enable is on an upwards curve. (Interestingly, routing is one of those IT areas that can't easily be distilled into a "...in 28 days" type crash course, ie,. commodified - along with systems programming, solid C++ coding, DBA-dom, and lots of things under the umberella of 'security'.)

Re:FFS, what a fucking dreadful summary

[Ryan Amos]

Yep, if you're multihomed, you can confidently have your marketing guys say "100% uptime." And stuff like this L3 "qwhoops" don't affect you.

Re:FFS, what a fucking dreadful summary

[Cally]

NSP sued... for... having... outage.

Riiiiiight.

Re:FFS, what a fucking dreadful summary

[sfurious]

Sued, maybe not. As long as they honour whatever compensation clauses exist in their Service Level Agreements, which will hurt.

X is Dying

[Christianfreak]

Glad to see that Tier 1 ISPs are joing the ranks of BSD and Apple.

Re:Tier 1's dying? OMG!

[hoborocks]

yeah

In other news...

[hoborocks]

....or is it related?

http://www.internettrafficreport.com/asia.htm is another place to check internet traffic - it's a different metric, but it's still useful. Taiwan and India seem to be the only places doing okay.

Re:In other news...

[after fallout]

I was wondering if anyone else noticed this

As a sophisticated veteran Windows user...

[mindaktiviti]

I could easily fix this problem. I would just restore it from the Recycle Bin.

Re:As a sophisticated veteran Windows user...

[NatasRevol]

Maybe Dvorak can consult for you.

Just don't ask any Mac users!

Next up...

[mindaktiviti]

Slashdotting the entire internet!

Re:Isn't God trying to tell us something?

[FidelCatsro]

So what , was this some sort of Divine Slashdot effect ...

Clarification

[Just Some Guy]

I'm sure you know this, but for the rest: "flapping" is the common term for when a router's routing tables rapidly cycle between two invalid states. The dead bird analogy is pretty descriptive, but the term "flapping" has technical and not allegorical origins.

Re:Clarification

[jafiwam]

?

In WI we call it "flopping". As in the routes are flopping between a valid and invalid state.

"Flapping" is too close to "Fapping", which one only does when the porn sites are still available.

Re:Clarification

[EvilTwinSkippy]

"Flapping" is too close to "Fapping", which one only does when the porn sites are still available.

Well, I guess some folks have different work environments...

In any case, deep down in the router programming references the process is referred to as Flapping. Or more accurately, anti-flapping (what routers do to detect when a chunk of the net is flapping, and to ignore it.)

Re:Clarification

[StillNeedMoreCoffee]

Of course this leads to Frapping which is the process of getting an iced drink at the Starbucks down the street, waiting for your packets to arrive.

Re:Clarification

[ipjohnson]

I was going to mod you up but you didn't provide any examples.

Re:Clarification

[NelsonM]

Remember kids, if you're going to Google "flapping," please don't forget the letter L like I just did.

Although my brain was rapidly cycling between two invalid states after looking that up.

Re:Clarification

[TheoMurpse]

Did somebody say "fapping"? Because that's something that becomes more difficult when the internet isn't working.

*BURP!* My bad.

[OgTheBarbarian]

(finishes stuffing last 'Pizza Pop' in gob) (unplugs microwave and removes from telco rack) Thorry.(spitting crumbs)

Nitpick to the max: Ye Olde Spell(ing)s

[Anonymous Coward]

nothing to do with

Your own first link explains:

"Domesday" (Middle English spelling of "Doomsday")

We regret to inform you that have failed the nitpicker's exam. Better luck next tyme.

EU

[NitsujTPU]

These are the first salvos of the EU's attack on the Internet, since the US wouldn't give in to their demands.

Good job guys. Now that incredibly fragile IP protocol is completely screwed, along with any chances of my getting onto match.com tonight.

Comment removed

[account_deleted]

Comment removed based on user account deletion

WoW interuption

[Profcrab]

I was playing online last night and it was like the internet had a stroke. WoW went down but I was still on voice chat with my friends. My trace routes were timing out at at about 5 hops down the line.

The good news is, I got more sleep last night.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Comment removed

[account_deleted]

Comment removed based on user account deletion

Not me, man!

[zrk]

I got the DTs(delerium telnets) from not getting my fix last night...

YES, TERRORISTS WANT TO TAKE OUR PORN!!!11!

[Wakko Warner]

The scary thing is it makes you wonder is some terrorist who has intimate knowledge of how Tier 1 ISP's work doing a trial run in the middle of the night by knocking out Level 3 and Verio backbones so later they could try to knock out ALL the backbones in a co-ordinated terrorist attack.

It doesn't make me wonder that. Terrorists do not give a shit about this kind of thing. To even invoke the word "terror" in this discussion is ludicrous.

- A.P.

Re:my net was down earlier

[petermgreen]

not surprising, the major search engines are afaict EXTREMELY well connected.

Has to be said...

[KC7GR]

Oh, so THAT's why my daily spam load suddenly dropped by about 35% or so...

Re:Has to be said...

[mattnuzum]

Mine went up. I got all the bounces because it couldn't be delivered or forwarded to the intended recipient.

Every little hickup causes me to get somewhere around 700 e-mails. This was more than a little hickup, but I actually didn't get as many as usual.

I will say that there is one thing worse than SPAM. That is BOUNCED SPAM. With SPAM you can look at the subject or the recipient and delete it without hesitation. With BOUNCED SPAM you get a delivery notification from postmaster. Hmm... is it important or is it just bounced spam. Better open it to see for sure. Yikes!

clearly supernatural

[Lord Ender]

It isn't just a coincidence that this outage happened on the same day as a gay pride parade somewhere. Clearly, our Lord Jesus hates these sinners and is exacting his revenge.

[a modest proposal: google for "satire" before modding down]

Re:Isn't God trying to tell us something?

[Lehk228]

more like a grand version of the time the boss "organized" the cables on the routers so they line up nicely

Hardly "Hanging On"

[Anonymous Coward]

They are doing pretty well, not amazingly so (what telco is?) but they have a lot of cash and a stable recurring revenue base. They also have a pretty good outlook because they are one of the few companies not caught with thier pants down when the FCC mandated E911 support - which a lot of people are coming to Level 3 for. If you think VOIP has a future then so does Level 3. The market thinks so; regardless of your outlook the stock has been up quite a bit recently.

To call them a "dot bomb" is really unfair since they were far more financially prudent during the timeframe, which is why they are still around at all in the dark forest of discarded Telco husks.

Disclaimer, I work for Level 3. But on the other hand doesn't that mean that I know more than most people about the real situation here?

I have had my paycheck bounce at companies I've worked for in the past and been told I'd have to wait an extra month or two for pay at said companies (you know the kind, six employees and the owners mom uses the company AMEX for trips to DisneyWorld while you wait weeks more to get paid). Level 3 is a few billion dollars away from that sad state.

And don't accuse me of drinking Kool-aid either - after going through a lot of layoffs over the years you have a VERY realistic outlook on what the company does well and what it does not.

Re:About a quarter of the people i know lost inter

[DrSkwid]

yeah, i never liked mobs

I bet most players don't even know it means "Monster OBjectS" !

That yellow thing is a *face*?

[raygundan]

It's a face? I never looked at it that close, because my parents told me it would hurt my eyes. Next time I'm outside, I'll make sure to stare directly at it until I can make out the face!

Re:That yellow thing is a *face*?

[Dachannien]

It's a face? I never looked at it that close, because my parents told me it would hurt my eyes.

Yep, sure is.

I R smrt

[Just Some Guy]

Could I possibly have sounded more pompous when I wrote that?

Let me try again:

Your dying bird analogy is OK, because that's pretty much how it acts. But they say a router is "flapping" because its routing tables are flailing about aimlessly like maps to the Superdome.

So, it describes what's actually happening, not how it compares to something your cat's trying to eat.

Re:I R smrt

[birge]

Dude, you're my idol. You just took a single post and got effectively modded +9 (Insightfully arrogant then obsequiously self-effacing). Brilliant. That's better than the guy last month who posted an incorrect statement, got modded +5 interesting, then posted a correction and got THAT modded +5, too. I'm waiting with bated breath for the first guy to hit the trifecta: post, recant, rerecant and get all modded up.

Re:I R smrt

[Just Some Guy]

Dude, you're my idol.

Thanks, man. But you still have LaTeX on your resume (I also have a long memory). :-)

Actually, the second post didn't work so well. I was hoping for a Funny, but got the dreaded Not Interested instead.

Completely down, not a routing problem

[Animats]

From NANOG, it looks like Level 3 went down completely, including their Border Gateway Protocol (BGP) sessions. That's a consistent failure, unlike that spat with Cogent last month where they denied transport yet sent out bogus routing info indicating they'd take traffic. With BGP down, there's no question that their customers and peers knew they were down.

So everybody with an alternate path around Level 3 should have routed around them properly. And yet, they weren't routed around. That's a concern.

If you went down because of this outage, your provider is totally dependent on Level 3, which is not good. This is a useful warning - if you went down, and your operation is important enough that it needs to stay up, you need to look very hard at your provider's upstream connectivity. Better hosting services have connections to several Tier I providers, just in case something like this happens.

Flap around the world...Began in Germany ?

[l.johnson2nd]

Seems a Fiber ISP in Palo Alto says it began in Germany. Thay have Deutsch Telecom with problems before Level 3. Long link to the NOC updates is on the top of their homepage today.

Lucky...

[infinite9]

Don't you get it? They deleted the internet!!

Good thing I downloaded it last night.

Unixshell graphs caught my eye

[tezza]

I don't work there, or own any shares in unixshell.

I was looking for a Linux Virtual Host, blah, blah.

Stumbed apon these pretty pictures (near bottom of page) .

Curious, I thought, what happened to Level(3) ? I though for a second because perhaps unixshell had a peering with those people that Level(3) were in dispute with.

Nope, just one of those regular outages that make the 99.999% promises sound a little over done.

2:30am??? You're lucky

[ooglek]

My pager went off at 1:48am EDT. Was able to get to my boxes from my Verizon connection, but couldn't get to other hosts via L3. Put in a ticket at 2:15am that L3 was having problems. Stupid L3 sucking woke me up. Grrrr.

Re:2:30am??? You're lucky

[toddbu]

My pager went off at 1:48am EDT

You should move to the West Coast, where it was only 10:48 PDT and you wouldn't have had to get woken up. I was watching the Simpsons when you logged your ticket.

Re:Yep My ISP went wrong

[gooberguy]

No you idiot, it's because we block you on sight.

Send your bill to SRoberts7758

[EnronHaliburton2004]

It's spelled "Hear! Hear!"

Nonononono... he was clearly replying to the parent's question. Please everyone, send your bill to SRoberts7758.

"And now, to whom from Level3 do I send my bill?"
HERE HERE!!!

False

[Andy Dodd]

Mobile OBjects, not Monster OBjects

Re:False

[DrSkwid]

hehe more fool me, funny how I ended up "knowing" that. Wonder where I picked that falsehood from? Thanks for the correction.

Re:About a quarter of the people i know lost inter

[jtev]

Let's go kill some MOBs sounds great to me. Unfortunatly the people I MMO with often don't understand what a MOB is. Damn you Graphical games for changing terms I've been using for years!

FIDOnet still up?

[davidnicol]

Or the global mesh of cooperating, overlapping wireless footprints?

Things are better in both the past and the future.

ouch

[fr3nch_com]

cd /Internet; rm -rf Level3

Re:So much for the internet surviving a nuclear wa

[Carnildo]

No one said it would react *quickly* to a nuclear war. It took about two hours for routing to go around Level3 rather than through it.

A streaming video clip of people during it!

[antdude]

Click here to watch it. A group of people going crazy during an Internet outage. Perfect timing. :)

Spammers

[Halvard]

Maybe their bandwidth is really just saturated by the spammers they harbor.

Re:Nitpick - gang of thugs

[Kenshin]

It doesn't speak much for the Slashdot community when Wikipedia has to put this warning at the top of a Slashdot-linked page:

This article has recently been linked from Slashdot.
Please keep an eye on the page history for errors or vandalism.

Now Cogent

[after fallout]

Now Cogent appears down too.

Cogent is down

[rush22]

Now Cogent is having a completely random and unrelated failure. What a total coincidence.

Re:It is Faked.

[after fallout]

you moron, go to the site and see for yourself.
http://scoreboard.keynote.com/scoreboard/Main.aspx ?Period=RH24

right now cogent is having a problem.
http://scoreboard.keynote.com/scoreboard/Main.aspx ?Period=RH1

Terror

[BadDream]

TAGGE: "And what of the Rebellion? If the Rebels have obtained a complete technical readout of this station, it is possible, however unlikely, that they might find a weakness and exploit it."

You know what happened less than two hours later :-)

Loss of the internet would have a very large financial effect, more then enough to be a juicy target for those that feed on hate.

Re:So much for the internet surviving a nuclear wa

[Curly]

I'd rather get an X-ray than have brain surgery.

Cyber attacks unlikly from terrorists, but...

[einhverfr]

Some other things people get worked up about but terrorists are unlikely to attempt: sabotaging bridges and tunnels to cause traffic jams; sabotaging electricity distribution to cause blackouts; sabotaging railroad tracks, making commuters late for work!

You have to assume though that any major landmark could be a target. For example, I could see the Golden Gate being a potential target, along with the Space Needle in Seattle, a few other bridges, etc. Also I can imagine that tunnels could be attacked with poison gas, anthrax. Even though the likelihood of damage from such an attack would be low, the fact that it happened would be terrifying to a lot of people. I.e. it is not enough to think death, and attacks on one's security need not take that form. Instead attacks on familiar landmarks and attacks against a general sense of safety.

Now, a large-scale cyber attack seems to me to be economically damaging enough that someone who wished our country harm might try it but it lacks the propaganda capabilities that traditional terrorist attacks have. For example, if Bin Laden destroyed the Golden Gate Bridge, he might be able to drum up some additional support, but if he claims to have disrupted the internet, most of his supporters will probably respond with blank stares. North Korea, OTOH, would have a much more likely motive....

Someone had to say it....

[AnotherScratchMonkey]

The dog ate my router!

Re:Isn't God trying to tell us something?

[cli_rules!]

So God knows IOS?! I'm starting to believe in Him again.

Halon stories

[Ungrounded Lightning]

Used to do contract work at an auto company's plant. The main data center's primary job was to feed test programs to an distributor testing line and collect the stats. It was located in the middle of the plant on the second floor, next to the row of test stands.

Some time after my contract had ended I visited the place and it was a total disaster.

During the model change shutdown (when most of the plant maintainence and rearrangement was done) the millwrights were welding on some cableways on the ceiling of the plant floor below. The fumes from the welding, of course, rose to the ceiling and escaped through the first hole they could find - around the big fire sprinkler pipe that went up through the floor of the computer room and into the space beneath the raised floor.

It tripped one ionization smoke alarm and sounded the warning - but nobody was around during the shutdown to hear it. Shortly thereafter it tripped a second one and the halon system went off. The computer power shut down and $10,000 worth of halon blasted into the computer room. Half of it came out through vents under the floor, throwing the raised floor panels and a decade's accumulation of fine dust (much of it byproducts of metal cutting and anealing) all over the room. And finally sounding an alarm at the guard shack.

The guards came over and found the room in disarray but no slightest sign of a fire. A couple million bucks worth of computer equipment, slated for replacement in another few months but still critical to the plant's operation, was standing there, covered with dust (likely to cause trouble for the disk drives later) but otherwise intact. So they followed procedure and reset the halon system, switching to the backup cylinder, to protect the computer in case an actual fire made it to the comp room. (Normally that's a good idea, since smouldering that sets off smoke detectors is often followed some time later by an actual fire.)

Of course the welding was still going on - just not at the moment the guard sniffed the comp room. (Welders out to lunch, pulled out due to the alarm, or having decided to come down off the ceiling for a bit after the blast of gas from above.) And they still had work to do. So of course they went back to it.

In less than an hour the situation repeated, dumping the SECOND $10,000 worth of halon on the non-fire. B-(

Flapping and barriers to entry.

[Ungrounded Lightning]

"flapping" is the common term for when a router's routing tables rapidly cycle between two invalid states.

Also known as "route flapping". Also applied when the routes are valid, but a router still alternately advertising a pair of routes because it keeps changing its mind about which is better.

This, by the way, is partly the result of an internet standard that isn't sufficiently prescriptive. The BGP protocol itself is well defined, but its implementation is left open. Unfortunately, it prescribes a system that has gain, delay, and negative feedback. So a naive implimentaion leads to oscilation when deployed, and something must be hacked to stop it.

Getting it to stop is a black art of coding workarounds - both to keep yourself from oscilating and to react quickly and appropriately (rather than blindly doing a full recomputation of your routing tables with each received flap) when somebody else starts flapping, so your packets keep going through despite the bad net weather.

If you get something stable among your own machines, you still have the issue of whether it stays stable when they're talking to somebody else's, or to an earlier version of your own. Avoiding breaking your own earlier stuff in a new release is difficult. Especially so since you don't necessarily know WHY your implementation is working (if nothing else, because you don't know what some of the other implementations it's successfully conversing with are up to). And testing before release is very hard, because you can easily get something that works just fine in all your lab test cases but breaks when deployed on the real net.

Since deploying a broken BGP implementation breaks, not just the routers it's deployed on, but large sections of the rest of the net, backbone providers and ISPs are very leery about buying routers from somebody without a proven BGP implementation. This creates a chicken-and-egg problem for companies trying to break into the router business: You can't get customers without a proven implementation, and you can't prove an implementation without customers.

Last I heard (by word of mouth, a couple years ago) there were only two independently-developed implementaions of BGP that had achieved this level of confidence.

I was thinking

[einhverfr]

Netcraft confirms it. Tier 1 ISPs are dying.

* The IDC confirms once again that market share among Tier 1 ISPs has fallen again.
etc....

I thought about writing a long wandering spoof but don't have the time...

MPEGs at 11....

[billstewart]

Used to be that people would predict "The End of Usenet As We Know It, GIFs at 11", but technology has progressed a lot since then...

Connectivity has evolved a lot, MAEs irrelevant

[billstewart]

The US-located parts of the Internet have evolved a lot since your ideas of design were formed. It's really a big change from the days when a pickup truck driving the wrong direction in the parking garage where MAE-East lived could have taken down half the Internet, and don't even *ask* which Tier 1 ISP was once renting POP space in the floor below a restaurant and had their servers taken out by a spilled pot of broccoli soup...

The MAEs don't really matter much any more, at least for Tier 1 peering, which has almost all moved off of public peering and into private peering. A lot of that private peering takes place in carrier hotels or telco POPs - Equinix has 7-8 big locations, Seattle's Westin building, a bunch of different buildings within a block or two of the main Los Angeles telco POP, and a few others. Some private peering also happens on fibers run between carrier offices.

Most of the Tier 1 providers have lots of excess bandwidth - if the DC area peering were to fail, most of them would have enough spare peering capacity in New York or Atlanta or Chicago to recover without major capacity losses, and BGP would reroute most of the rest reasonably well. The West Coast is in a bit worse shape, just because the distances are longer - SF-LA is only ~350 miles (~3.5ms one-way), but SF-Seattle is a lot farther, and isn't quite as overbuilt, and with many carriers, if you lose the direct route, you take a 2-3000 mile loop through Salt Lake City to recover (unless you've got two central California routes, on I-5 and 101 or railroads.)

European peering architectures have much differently - the geography's different, and the carrier relationships were different, so huge fractions of that traffic go through LINX and a few other points like AMSIX, and losing LINX would be seriously bad.

Publish Longer-Match Prefixes to Avoid That

[billstewart]

If you've got a given prefix, and people are exporting bogus reachability information about it, start advertising two prefixes of half the size. I know one large ISP with a /8 who had to start publishing two /9s because some bozo outfit was doing incorrect route summarization and claiming that their little circuit in South America had a really great route to that /8. It's a cheap trick, and you shouldn't leave it up too long if you can avoid it, but works really well when you need it.

Software Problems Worse than Physical Damage

[billstewart]

Yeah, the Internet can find any available routes around physical damage and repair itself. But software problems are often much harder to work around, and can be much more insidious. Most ISPs run most of their services on big routers from companies starting with C or J (or sometimes L or N or A), and if an ISP installs a software version or a set of router configurations on most of their machines without testing it carefully enough, they can take down their whole network. (I'm not just blaming either the ISP or the router vendor there - many of the Tier 1 ISPs are carrying traffic loads that are too big and weird to duplicate in a lab that's smaller than the real network, and sometimes there's a bug that only shows up if somebody configures two or three different things in subtly-inadequate ways on different models of hardware and then overloads the hardware.) And you'll notice they did their install at US midnight, when it's about as quiet as it's going to get and they've got time to restore things before morning.

Most of the time, ISPs can protect themselves against their neighbors failing, and most of the time they do. A few years back, some random company advertised that their T1 was the best way for half the world to reach MAE-EAST (or some target of that size), and suddenly half the world's internet traffic was trying to get down that wire before it melted, making it difficult for any equipment nearby it to even scream for help. Lots of ISPs started doing a lot more BGP filtering after that, and developing methods to monitor the advertisements the outside world was seeing about them, and things got safer, but it's still possible to screw up.

5-20% of US bandwidth at Midnight == No Problem

[billstewart]

The US has about 20-30 Tier 1 ISP carriers, depending on who you count, and most of them have significantly more bandwidth than their average busy-hour traffic levels, so on the average, each one's under 5%, not even counting the bandwidth of well-connected Tier-2 carriers, international carriers, etc. Level 3's one of the bigger ones, but if that meant they were 10% or even 20% of the total bandwidth, losing that much in the middle of the night wouldn't bother the Internet as a whole.

Their direct customers would obviously be affected, so if you've only got one ISP connection and they're down, you're out of service - if losing connectivity in the middle of the night is a problem, you need to arrange for diversity, but during the daytime you're more likely to have a backhoe take out the wires on your street than have your ISP down for more than an hour or two.

L3's a big wholesale provider, so if they're down, it can affect people who didn't know they were using their services; maybe they're using a small ISP that buys half its bandwidth as transit from L3, or maybe their ISP is using L3 to reach specific areas where they don't have geographical coverage or provide specific types of service. So the outage may feel a bit more widespread than it really was, but it's still the middle of the night. The recent L3-Cogent fracas was a bit more visible because they handle different kinds of customers - L3 provides bandwidth to lots of small ISPs with consumer end-users, while Cogent provides big cheap pipes to lots of hosting business, so the interference was fairly synergistic, and it lasted a lot longer because it was a Layer 8 / Layer 9 business disagreement, not a Layer 3 technical problem that can be fixed by engineers.

here's what went wrong in Level3

[AviN456]

what went wrong in Level3: http://www.merit.edu/mail.archives/nanog/msg13166. html