Duke Research Experiment Disrupts Internet Traffic

alphadogg writes with this excerpt from Network World about an experiment gone wrong which affected a big chunk of internet traffic yesterday morning: "It was kicked off when RIPE NCC (Reseaux IP Europeens Network Coordination Centre) and Duke ran an experiment that involved the Border Gateway Protocol (BGP) — used by routers to know where to send their traffic on the Internet. RIPE started announcing BGP routes that were configured a little differently from normal because they used an experimental data format. RIPE's data was soon passed from router to router on the Internet, and within minutes it became clear that this was causing problems. ... [f]or a brief period Friday morning, about 1 percent of all the Internet's traffic was affected by the snafu, as routers could not properly process the BGP routes they were being sent."

Comments

It was you

[Anonymous Coward]

So it was you who blocked Ted Steven's tubes.

Re:It was you

[Conchobair]

We should name the biggest tube of the internet after Ted Stevens as a memorial. The Ted Stevens Memorial Internet Tube. *Stevens's

Re:It was you

[Anonymous Coward]

So it was you who blocked Ted Steven's tubes.

Why don't you bend over and let me unblock your tubes?

Wow

[Voltageaav]

So you really can crash the internet?

Re:Wow

[click2005]

Only if you're a really bad driver.

Re:Wow

[Anonymous Coward]

Yes, where have you been?

The internet has been nulled before, Youtube has been blocked, countless other huge breakages before.
The internet is a very frail entity, mainly kept together by trust in the organizations who run the main backbones of each country / continent.
And DoSing, more commonly botnet controlled DDoSing tools, are becoming a very popular for blackmarkets of the net these days.

Hell, there was that time where someone, somehow, managed to run a rogue DNS root server for a while and got away with god knows what kinds of information.
This was all down to simple human error in configuring IPs and forgetting that one was moved to an entirely different IP.
(Anyone know if there has there been any follow-up to that?)

Re:Wow

[timeOday]

So you really can crash the internet?

Maybe, yes. BGP has been identified as vulnerable for a long time, and this is further proof. On the other hand, this research is probably motivated by fixing the problem. But the Internet is no longer something you can just shut down or reboot to upgrade; you must operate on a live patient. It does make you wonder, though, if well-intentioned people can do this trying to help, what somebody malicious could do. Hopefully governments will decline to use this as a weapon - like poisoning the ocean.

Re:Wow

[Anonymous Coward]

In my experience incompetent people trying to help can do more damage than someone whe actually tries to do bad stuff. This often because the nice people are trusted with stuff they shouldn't have access to just because they have no reason tinkering with it.

Re:Wow

[g0bshiTe]

The difference is that even a malicious person most likely would not kill the internet. It would be far more beneficial to exploit it.

Re:Wow

[suso]

I wouldn't say countless. There have probably only been less than 10 blackhole type events with BGP/routing that affected a significant amount of Internet traffic in the past 15 years. The big one being back in 1997. There is a website somewhere that keeps track of them and explains what happened.

Re:Wow

[IhateMonkeys]

I don't think I would consider Youtube being blocked a crash.

I see it as more of a blessing.

Re:Wow

[bill_mcgonigle]

But the Internet is no longer something you can just shut down or reboot to upgrade; you must operate on a live patient.

That's a really important point that often goes undiscussed - it's been suggested that if the Internet did go down (major solar storm, EMP, etc.) that it's not likely that it, or the interconnected systems (electrical grid, etc.) could come back up. Too many race conditions, mostly unknown/undocumented. Sure, eventually it would all get back on track, but it could be weeks-to-months. I'm planning to hike the Appalachian Trail while it gets straightened out. ;)

Hopefully governments will decline to use this as a weapon - like poisoning the ocean.

That sounds like a major societal vulnerability that needs to be patched. Nuclear weapons marked an important turning point in history where governments became too dangerous to keep around.

Re:Wow

[Anonymous Coward]

But where will all the Youtube commenters go? The seething tides of illiterate morons could wash up anywhere on the internet. I would be wary.

Re:Wow

[BillGod]

Yes if you search for google in google that can crash the internet.

Re:Wow

[Rolaulten]

They'd end up at 4chan or ebalmsworld, which in turn buffers the rest of the Internet...turns out those websites do have a reason for existence.

Re:Wow

[insertwackynamehere]

Troll? Really? This isn't even a polarizing opinion that a moderator could abuse the system for. This is just like avante garde moderation.

Re:Wow

[Anonymous Coward]

The first rule of network testing is to not test on a live network. What where they thinking? With today's network test tools you can easily emulate hundred of routers (thousands if you have a lot of money to spend on test equipment or if you build a big virtualized test bed).

Re:Wow

[bill_mcgonigle]

Hey, thanks for the support. Probably just a government worker reading Slashdot while on the clock. ;)

Re:Wow

[Anonymous Coward]

The part about the electrical grid (which is the only other example given), yeah, I'd consider that a troll. At best, it's just silly. Ignoring the scaremongering about the electrical grid going down with the Internet, look up black start.

Re:Wow

[Cramer]

Heh. Well, 1% of it at least. (reportedly) I didn't notice anything at all. (there were rumblings on NANOG since Thursday/Friday.)

Re:Wow

[tehcyder]

Nuclear weapons marked an important turning point in history where governments became too dangerous to keep around.

Let me guess, those nice ethical corporations should be in control instead and the free market will solve everything?

Re:Wow

[bill_mcgonigle]

Let me guess, those nice ethical corporations should be in control instead and the free market will solve everything?

Of course not, corporations don't exist without government strongmen backing them.

A big chunk?

[nurb432]

1% isn't big in my book.

I would have liked to see what would happen if they kept going with this.

Re:A big chunk?

[Anonymous Coward]

Well, 1% is all the traffic that is not "Linux ISO" downloads.

Or it could be the 1% of the internet that is goat porn?

These statistics brought to you by the department of made-up statistics.

Re:A big chunk?

[odies]

You do understand how much traffic Internet has? 1% is big.

Re:A big chunk?

[Anonymous Coward]

That's not what your mom said...

Re:A big chunk?

[dreamchaser]

Ok, so what if 1% of all people on the planet just dropped dead? That would be over 60 million people.

Now apply that ratio to the thousands of nodes and terabytes of data that flow over the 'net. One percent is quite a bit more than you seem to think it is.

No I'm not comparing it to people dying. I am just illustrating the point.

Re:A big chunk?

[Voltageaav]

If they had, you probably wouldn't even know about it because you wouldn't be reading slashdot. You'd probably just be calling your ISP asking them why your internet connection isn't working.

Re:A big chunk?

[Anonymous Coward]

And 99% of that 1% were offline users.

Re:A big chunk?

[Shanoyu]

I think it's actually more significant that it was not 'large'; it was almost an accidental targeted attack.

I mean, sure, 1% is a lot depending on how you want to look at it.

Re:A big chunk?

[whoisisis]

I seem to recall that CERN produces about 1% of all data that goes through the internet every day. Hmm...

Re:A big chunk?

[Anonymous Coward]

Just like you're a moron no matter how I want to look at it.

Re:A big chunk?

[Anonymous Coward]

Now if that 1% could have been 100% of spam the intertubes would have been virtually empty!

Re:A big chunk?

[nurb432]

In relative terms 1% is not a "big chunk of" anything. It may be a lot in total amount, but is still a small percentage of the item in question.

Re:A big chunk?

[Kilrah_il]

Well, judging from the comments on /. yesterday, I couldn't tell the difference. Duke, next time try harder.

Re:A big chunk?

[linebackn]

1% can be either large or small depending on what is being measured. For example, a Pointy Haired Boss may think the following:

* 1% of your web site user base using a different web browser is insignificant and can easily be ignored.
* 1% of your annual profits is HUGE and losing or failing to obtain those means heads must roll.

(of course, a true PHB will never see any potential relationship between the two)

Re:A big chunk?

[thePowerOfGrayskull]

That's kind of the point isn't it? 1% isn't; a few hundred million is. That's the risk of using percentages: they tend to minimize the significance of the real numbers -- or alternatively, overstate their significance.

Re:A big chunk?

[Anonymous Coward]

How much of that 1% is useful data and not spam, pirate torrents, porn or other entertainment?

Re:A big chunk?

[Delarth799]

1% is a HUGE number, I mean its almost 2%!!!

Re:A big chunk?

[straponego]

Poppycock! It's barely half of 2%!

Re:A big chunk?

[mcgrew]

I wish I had 1% of Bill Gates' money. Not big, my ass.

Duke Research Experiment

[Major Downtime]

What's there to research? 3D Realms announced publicly in 2001 that Duke Nukem Forever would be released simply "when it's done"

Hmm...

[fuzzyfuzzyfungus]

The description of this incident makes BPG sound as brittle as it is trusting...

Re:Hmm...

[pandaman9000]

BGP is, like all routing protocols, very secure in and of itself. The difficulty is that a router peering with all routers on the internet can "inject" bad routes, and the "mail" gets reliably delivered to a wrong address. This is ONLY a difficulty if you can somehow gain access to a router that is directly connected to a backbone, and has peering status. You will have to have your own Autonomous System number also, although I am sure you could fake that.

The only time that I have seen even isolated internet routing issues is due to mis-configuration of the router by the owner. Well, that and the extremely rare (yes, really, it is rare) OWNing of an edge router.

I am not all- knowing on this subject; far from it. If someone has something to add/update/correct, please do.

Re:Hmm...

[Kilrah_il]

What's the problem? Telnet to one of the routers. When it asks for user name write "admin". Password "1234" and there you go! Oh, and when it prompt you with "Your password hasn't been changed in a long time, do you want to change it?" click on "Never ask me that question again".
Easy as cake.

Re:Hmm...

[Cylix]

Any ISP network engineer has some good BGP stories.

For me I was I fighting for over a year to get some of MY blocks back from another provider. They simply continued to announce the routes for them and made it uttererly worthless. It was also fairly horrible to get any upstream traction against the offender.

Eventually, we simply started announcing the routes for those blocks and caused turmoil for those who were using them. It didn't take long to get that issue cleaned up afterwards. Though it was funny because they had asked my guys to stop announcing.

BGP is a bit of a trust relationship, but it isn't the end of world when everything goes to shit.

Admins will get up for their beds and start clearing issues. Things will be sluggish for a bit, but eventually things work out.

Re:Hmm...

[phyrexianshaw.ca]

Fake it? Not in the last five years!
unless you know of some BGP peers that refuse the standard peering protocol, 1) they are required to only listen to routes from known surrounding peers, 2) will not be listening to what's being advertised by your router unless you have instructed them ahead of time what AS you manage and what prefixes you will be advertising to them.

if for some strange reason, you manage to be adjacent to a backbone CORE router, and wanted to spend a few years moving traffic from core's to edges of the internet, you could start injecting routes for a short span of time after having been trusted and your metric's lowered, (at some point BGP will fail to converge and your advertisements will begin being ignored by the AS)

for research purposes here in Canada, we have access to a major core router, and are able to inject routes to get traffic routed through a particular peer for a few minutes at a time. wirecapping the lines at that router, we can then monitor for organisational security compliance for penetration testing. (you'd be surprised how often usernames and passwords get sent in clear text, or how often people THINK intra building traffic is being encrypted via a VPN only to find out it's badly midconfigured.)

I too am far from all knowing on the ins and outs of global BGP, but every peering agreement I've read (from about twelve countries and almost a hundred cities) have always been the same. "you are required to listen to ASxxxxx for advertisments for this super block, you are required to listen to these private peers with multi-homing agreements, you are required to advertise with the AS number assigned to you only, you are required to advertise only the prefixes you privately manage, and to contact and update the peers directly adjacent to you if assigned a new superblock. etc"

Re:Hmm...

[comm3c]

Fake it? Not in the last five years!

unless you know of some BGP peers that refuse the standard peering protocol, 1) they are required to only listen to routes from known surrounding peers, 2) will not be listening to what's being advertised by your router unless you have instructed them ahead of time what AS you manage and what prefixes you will be advertising to them.

No. Period, fucking no. Most BGP sessions run between customers and carriers are still basically allowing whatever. Even the big boys basically don't care what you advertise. It would cause too many problems to go and begin filtering, so only regions that seem to have routing DBs (RIPE region) are even remotely participating in this. For the most part, thats a few places here and there, but the carriers will let you do what you want.

Don't believe the hype: BGP is still as weak in public IP as it ever has been. The difference is that if you do decide to hijack someone else's prefixes (don't even include bogons, because the carriers will probably let you advertise those!), everyone will know and you will get your upstream looking at you.

Re:Hmm...

[Anonymous Coward]

My networking class instructor in college had a doozy of a story.

He and a coworker were working for a company, and while they weren't supposed to have the passwords for the BGP routers and whatnot, they did as a matter of expediency. (you know, someone not wanting to walk over somewhere to just enter a code, etc). Anyways, the coworker executed a hard re-computation of the BGP routes rather than a soft one, bringing the entire company's network down for about a half hour until everything was recomputed. The only reason they escaped with their jobs was because they weren't supposed to have the passwords, and thus not suspected.

Re:Hmm...

[Bruha]

We've had this issue happen too, thankfully we have an army of lawyers who threatened to bankrupt their upstream providers and called the police on the ISP that was not fixing the issue, it helps when they try announcing IP space that the government uses.

Re:Hmm...

[phyrexianshaw.ca]

What "big boys" are you talking about?

for every major carrier that I've worked with, filtering isn't optional, it's mandatory.

at the tier one level, Qwest, AT&T, Sprint and L3 all dampen their allowable routes to what they know the immediate peers will advertise. at tier two, there will be many smaller ISP's who will haply pass routes to whomever wants to advertise them, but is not going to be listening to BGP messages on customer facing ports. (unless that customer has already made an agreement with that peer to make an AS entry on both sides)

1% of all traffic?

[Evil Shabazz]

1% you say? Ah, so they somehow only affected the non-porn traffic?

Entire .tr DSL got effected

[Ilgaz]

Yesterday, there were a lot of feedback regarding some really mysterious cuts to popular sites. As .tr Govt. is known to censor Internet, people thought something was wrong at the boxes which does the censoring job.

That experiment really went out of hand I think. And, 1% of Internet in 2010 is... Huge. Really huge.

Not surprising

[Fryth]

I can't believe we don't see more of this, considering the trust-based nature of BGP. I'm not saying that's a bad thing, I'm just wondering out loud why this is so unusual.

Re:Not surprising

[rickb928]

You don't see more of it because you don't get told, and you don't look.

Then there are the snit fits peers have to indulge in.

And the occasional stupidity.

Go check out the Internet Taffic Report from time to time. Today it looks like there was significant event. Wonder what happened.....?

Now don't get me started on PMTUD. How do I explain to a user that it is not 'our' network that is the cause, we have MILLIONS of users working just fine, but everyone in their office can't get on because we broke something just to annoy them? And of course, since they can see the same error a different, unrelated site, it MUST BE US. Yeah. I'm the designated PMTUD expert on the team now, because I let their ISP talk itself into the solution. And I can read packet captures. Yay me, think I'm going off decaf for a few days...

The Internet is not perfect.

Re:Not surprising

[phyrexianshaw.ca]

the trusting nature of BGP? if you inject bad advertisement for too long, you'll get marked as damaged, and the BGP AS will begin converging around you. (cutting off any private subscribers you maintain as they no longer have valid routes back to them from the internet.)

in current BGP, you don't GET trusted, you BUILD trust. you're established a very high metric (or weight) for distance routing initially, and as you carry traffic, (or as more and more traffic originates from your network from your subscribers) your metric will be lowered overtime, moving greater and greater volumes of traffic over your infrastructure.

Re:Not surprising

[bill_mcgonigle]

your metric will be lowered overtime

Is this by protocol or manually, by convention?

Re:Not surprising

[phyrexianshaw.ca]

depends on how you like to manage your equipment. most large companies tend to do it manually, while smaller businesses like to do it automatically. On BSD and cisco routers, it's easy to assign the metrics by hand. some ISP's will even buy a lower metric from you (forcing more traffic off their links)

Re: 1% dropping dead

[macraig]

Ok, so what if 1% of all people on the planet just dropped dead? That would be over 60 million people.

That would be a measurable win for global warming, pollution, species extinction, deforestation, you name it... and I don't think it's politically incorrect to point it out. It might be off topic, though.

Re: 1% dropping dead

[XanC]

You first, then.

Re: 1% dropping dead

[macraig]

That can be arranged.

It was a Cisco bug in a specific model of router

[rwyoder]

http://www.cisco.com/en/US/products/products_security_advisory09186a0080b4411f.shtml

Re: 1% dropping dead

[Runaway1956]

It most certainly IS politically incorrect. Since I'm not a politically correct person, it doesn't bother me. Some drone or another will be along shortly to give you hell though.

Well that explains it.

[g0bshiTe]

Last night I was wondering what was going on with my ISP's DNS servers. I have my network pointing directly to their servers for DNS, and I was getting some flaky behaviour.

Re:Well that explains it.

[jack2000]

I don't trust isps with dns. Too much stupidity. My current isp is horrible really. That's why I use google's servers.

and here I thought..

[PrimordialSoup]

skynet is finally here

That Could Explain the Steam Spike...

[Bieeanda]

For those of you who don't use Valve's Steam storefront/game launch application, the app has a graph that shows usage rates at various scales. Typically it shows the last 48 hours, and typically the graph is sinusoidal. On Friday morning, at about twenty to eleven and at the top of a wave, connections plunged from 2.2 million to under 300,000, before leaping straight back up to 2 million-odd shortly after eleven.

Blame Cisco (maybe)

[forgot_my_nick]

According to comments on the linked article the problem was that Cisco CRS-1 routers misinterpreted or didn't understand the modified BGP data and passed on corrupted versions of the BGP data.

No, it's a control-plane problem

[billstewart]

Teh Intertubes aren't blocked, they're just connected up differently. Kind of like if it were trucks and you put up detour signs.

Re: 1% dropping dead

[Anonymous Coward]

Church of Euthanasia?

Re:It was a Cisco bug in a specific model of route

[DeadboltX]

Is this the same vulnerability that was on slashdot over a year ago? http://tech.slashdot.org/article.pl?sid=09/02/22/0310236 The summary tried to make it sound like Mikrotik was to blame, because it sent the bad bgp information, but it was the Cisco that errored out.

Re:It was a Cisco bug in a specific model of route

[rwyoder]

Is this the same vulnerability that was on slashdot over a year ago? http://tech.slashdot.org/article.pl?sid=09/02/22/0310236 The summary tried to make it sound like Mikrotik was to blame, because it sent the bad bgp information, but it was the Cisco that errored out.

No.
That was a configuration error made on a Mikrotik resulting in massive prepending of the BGP path.
This was a flaw in how unrecognized BGP attributes are handled.

Re: 1% dropping dead

[macraig]

And here I thought I knew my geography! I've heard of East Asia, I've heard of Southeast Asia, but I've never heard of Euthan Asia... where the hell is that and why does it have its own church?

Terrible article, bad summary

[Eunuchswear]

What a lot of verbiage to say:

Some routers have bad BGP implementations that handle attributes longer than 255 bytes incorrectly

Some of those routers will drop a BGP connection if thet get such an attribute.

The article makes it sound as if RIPE is in the business of distributing routes to BGP routers.

(See http://www.merit.edu/mail.archives/nanog/msg11505.html for details).

thanks.

[bill_mcgonigle]

good to know.

Insert...

[Cur8or]

Insert silly Skynet/Swordfish joke here

Pictures of the disruption

[Brighten]

Here are some pictures showing the effects of the disruption, including a 6x or more increase in messaging over the "background chatter" on the Internet, and a description of what went wrong.