Slashdot Mirror
The Status of Routing Reform — How Fragile is the Internet?
crimeandpunishment points out the Associated Press's look (as carried by SkunkPost) "at an issue the government has been aware of for more than 20 years, but still isn't fixed and continues to cause Internet outages: a flaw in the routing system that sends data from carrier to carrier. Most outages are innocent and fixed quickly, but there's growing concern the next one could be devastating. A general manager at Renesys Corporation, which tracks the performance of Internet data routes, says, 'It amazes me every day when I get into work and find it's working.'"
...i'm glad I decided to wait for internets2 before i get online.
[posted via FIDOnet]
THL phish sticks
Kind of. However, it has also always been this way, and it has survived so far. All that has really changed is the number of players has increased, and the size of the routing tables are increasing.
It has to work, so a lot of people should notice very quickly if something large goes wrong.
It also cannot very easily be fixed, as many players would have to spend a lot of money for it to change, and there is little financial incentive to chase that ghost.
And you thought IPv6 or DNSSEC adoption was taking a long time... imagine how many decades it would take for SBGP adoption?
First of all, the US federal government shouldn't have the power to do this even in America, and it definitely doesn't have the power to enforce this in the rest of the world.
Secondly, no sane ISP will forward BGP data.
This limits the problem to people with access to core internet routers. Companies that own these routers should only give access to extremely trustworthy people, and even then, they should still only need to access the server when there's a legitimate change. The issue then lies with accidents, which will always happen, no matter what you do, and corruptness. Corrupt ISPs should be removed from the network as soon as they are found to be corrupt.
BGP Filtering. There, fixed that for you.
"In the meantime, network administrators deal with hijacking an old-fashioned way: calling their counterparts close to where the hijacking is happening to get them to manually change data routes. Because e-mails may not arrive if a route has been hijacked, the phone is a more reliable option, says Tom Daly, chief technical officer of Dynamic Network Services Inc., which provides Web hosting and other Internet services."
Route filtering, USE IT!
Especially when peering with Pakistani/Chinese/etc ISPs.
This is why RIRs such as RIPE/ARIN/APNIC have their information publicly available.
So you know which addresses belong to who.
Only accept routes from your BGP peers that you know belong to them.
This also (in addition to hijack prevention) prevents a clueless NOC monkey from another autonomous system from messing up your whole network by announcing a default route.
But there are only 13 internet root servers . . . .
13 root DNS servers...this is a different protocol altogether. I don't pretend to understand real well--VLSM/CIDR confuse the hell out of me, and that's where I gave up trying to understand the nuts and bolts--but there's a very large number of systems whose routes would need to be compromised, and quickly, to make this have an effect that is visible to end users--and even that would be short lived. As the parent put it:
This "hijacking" happens all the time, people immediately see it and fix it and nobody notices.
I don't post AC. I like my -1, Flamebaits. Trump/Sheen 2012 on the Batshit Insane ticket!
The only reason a Major ISP hasn't had a full, network wide outage is simply a lack of desire on the part of the people that would be capable of doing such a thing. In fact, many ISPs do have network wide outages fairly regularly but are able to keep it hidden. Most customers think it was local to them. What makes networks so week? The same thing that caused the oil spill in the gulf. It costs to much to do things correctly. And what are the chances anything bad will happen... right?
What?! Anyone can edit it?! Really???
'It amazes me every day when I get into work and find the Wikipedia front page has not been blanked or filled with goatse porn.'
it's hard to work out if your joking, ignorant or stupid
If you mod me down, I will become more powerful than you can imagine....
Routing reform? The answer is simple. Just fine Cisco $750 for every router until it starts routing correctly (or they go bankrupt and take Federal bailout money in exchange for incorporating federal guidelines in all future router designs; including backdoor, and mandating USGaBGP, US government-authorized BGP, where the government will issue every router operator who pays the fee and follows the rules a digital certificate to use their AS number, and a digital certificate for each IP prefix the router owner obtains, after filling out 100000 reams of paperwork).
There, fixed it for you.
From TFA:
"It's kind of everybody's problem, because it impacts the stability of the Internet, but at the same time it's nobody's problem because nobody owns it," says Doug Maughan, who deals with the issue at the Department of Homeland Security.
So clearly we need one centrally owned routing system under the watchful and benevolent eye of DHS, right? With help from advisors provided by Microsoft and Disney.
Decentralized routing is a feature, not a bug. And although the problems identified in the article are real enough, the implications of this kind of discussion always scare the hell out of me.
The correlation between ignorance of statistics and using "correlation is not causation" as an argument is close to 1.
From the article: "My fear is that innovation on the Internet would slow down if there's a need to go through a central authority," Poll says. "I see little appetite for that in the industry." --- Is there an argument against this (quote above)?
No single point of failure? Correct. Instead it seems to be many points of failure. I am not a networking wiz and I don't even like networking issues, but I have taken a few networking classes and after trying to set up even basic RIP stuff I'm amazed that the internet works at all. It's been a while ago but I recall that even one team in our lab screwing up brought down the whole network.
http://www.rootstrikers.org/
This is ridiculous, I suspect this is FUD created to take control of the Internet. Routing tables are a feature of the Internet that are designed to ensure the Internet doesn't have a single point of failure. Hacked router?, connection hit by bomb?, satellite suffering from solar flares?... change a few routes and it's fixed. Security?... TLS. The moron even suggests that creating a central authority would make the Internet more secure!!! Imagine if you wanted to take out the Internet and it relied on a central authority, hmm, what would you attack, billions of Internet clients, millions of routers, or the one authority?
Or, as Arthur C. Clarke put it, "Any sufficiently advanced technology is indistinguishable from magic."
Actually no, DARPA is, if it was the US Government that was the founder, IPv4 would still be in a committee somewhere :)
Hint: Military trumps government for getting stuff done...
You don't actually know what "the government" is, do you?
The correlation between ignorance of statistics and using "correlation is not causation" as an argument is close to 1.
And that is a big reason why the Internet exterior gateway protocol is not RIP or any other IGP.
A premise of the RIP and other IGP protocols is routers talking to each other trust each other.
With BGP, the premise is the opposite... routers speaking the protocol implement policies against each other: policies regarding what routes they propagate or originate outbound, policies regarding what routes they accept, and policies regarding what incoming routes they propagate.
So networks that don't trust each other only accept appropriate routes from their peer based on AS-path and Prefix-list filters.
Basically almost all networks should treat their peers as untrusted, and list out prefixes of end users.
It doesn't start to get hairy, until you need to peer with a provider (instead of an end-user) and accept all prefixes from them, because you want their customer prefixes, or you want to buy transit from them.
As for ISPs and providers though... failing to filter downstream announces is the exception to the rule.
For it would deprive us of these terrible sensationalist articles. The InterWebz is doomed!
Mistakes will be made. And some people will lose their Internet connectivity (in some form or other) for a period of time.
During that time, the people who control the routers will be working to fix whatever problem happened and the idiots who caused the problem will either learn how to do it CORRECTLY or be fired. Although the executives who insisted on cutting the budget so that they couldn't hire people with the knowledge in the first place will still keep their bonuses and their jobs.
At work, our Internet connection is through Verizon. Within the past two months, we've had 1 day of no connection (and Verizon still denies that there was any problem) and a few days massive packet loss (and still there is no problem noticed by Verizon).
BGP works and works well. But it does require people with the knowledge of how to make it work.
With the FCC stymied in its attempts to regulate the internet, it's going to be basically an ISP fur ball. Layer general greed and self-interest of individual providers on top of load and routing problems, take away the regulators ability to maintain order and you have a recipe for disaster.
I got a bad feeling about this.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
"I am not a networking wiz and I don't even like networking issues" So you tried to setup basic RIP and you are amazed the internet works at all huh.
Well this artical is pure BS, sure you packets go between multiple backbone ISP's and a couple smaller isps on the edge maybe, but the guys that run the bigger ISP's do have rules that govern how they BGP peer with other backbones and peers. They enforce strict BGP filtering, to keep the smaller compaines from causing major issues.
Sure every once in a awhile someone might fat finger some shit and mess something up that will effect 1 of the main backbones, but with more automated tools this happens way less than it used to. Most big backbone ISP's use router hierarchy and pure core routers are protected from anyone configuring them much at all once setup.
I think the system runs well, I am sure it could be made better in many ways, but the issues made here are non issues, the backbones one security would be the main factor here, and that should get only better over time.
Its better there is no central routing authority on the internet. Each company has it in thier best interest that it has the best routes to get to a centain network, and if that company messes its routes up, others should be protected by proper BGP filering. BGP filtering can get pretty complex, on ciscos this can be with prefix based ACL's and also with BGP AS number based ACL's, you can also use BGP communities to keep things nice and neat. If done correctly it can be pretty rock solid, if a rookie does the filtering you can have holes and issues, but a big company like LEVEL3 for instance, should have standards and all this stuff pretty hardened and worked out.
This internet sky is not falling.
if the government gets to "fix" the internet, i may just have to give up slashdot.
If you mod me down, I will become more powerful than you can imagine....
Route filtering. Trust me, if the 12 occasionally scattered folk I work with every day can manage block leaks of inappropriate routes within 15-60 minutes, so can everyone else, and they typically do...generally they're properly filtered to begin with. The open nature of the internet and diversity amongst transit carriers is precisely what contains these leaks to segmented populations rather than causing a massive nationwide failure. The fact that largely Internet standards have been left to technocratic, Balkanized organizations rather than via Congress is what keeps everyone playing nice. The "next one" may be "a big one", but anyone running a truly important network should and will have diverse carriers...anyone critical to the US infrastructure should and does generally run over dark fiber that would not be affected. Not seeing the call to action here, but I have very little faith in the media to actually competently understand and relate this one. HangingChad, exactly: "I got a bad feeling about this"
I've seen alternate routing protocols proposed wherein your traffic has to barter/haggle its way through the network at every hop, as some new troll demands a passage fee for a certain QOS.
These new methods look to me like they would create two issues:
1. Unpredictable permutations of complex, balkanized, and non-local routing strategies. Performance of the system as a whole would be unpredictable and possibly unstable.
2. It really is back to the old circuit-switching network of ma bell, on top of IP. A few nice low-latency end-to-end Concorde-like connections for those willing to fork over the dough, clogging up the routers so all the proletariat traffic suffers in a poverty of routes and bandwidth.
Deep Simplicity at the core of routing protocol is the only thing that will work at the scale of the Internet. Maybe a "voluntary-QOS-downgrade" flag on email packets etc, and a "pretty please low latency" flag on video packets, might work, but these should not have monetary contracts associated with them. They should just indirectly affect the end-consumer's bandwidth bill if anything.
Where are we going and why are we in a handbasket?
Actually no, DARPA is, if it was the US Government that was the founder, IPv4 would still be in a committee somewhere :)
Hint: Military trumps government for getting stuff done...
You don't actually know what "the government" is, do you?
I'd agree with the GP. The military is not directly "the government" (unless you live in a place under martial law). They certainly work for the government, but so do school teachers and I wouldn't refer to a school as being "the government".
I consider "the government" to be "that which governs". The military's job is NOT to govern, but to defend the nation (which often takes a myriad of forms depending on where you live) - the government is responsible for directing the military to do this and may often have their fingers in the "how", but not always, and only very rarely down to an actual implementation level.
My book about LSD and Self-Discovery
Also on facebook as: DroppingAcidDaleBewan
I wonder if something should be done to limit the deployment of straight Ethernet as opposed to OC-[0-9]+, ATM, Sonet, etc... for Tier-1 backbone traffic.
I don't have real numbers or statistics I can back up my claims with, but having experimented with implementing SONET and ethernet VLSI simulations, I'm convinced that SONET maintains a much more reliable connection and is able to recover from glitches MUCH quicker than Ethernet. Sure, we're talking about milliseconds, but over long distances, glitches must be common enough to allow these glitches to screw up UDP traffic on a massive scale.
On the other hand, maybe it's time for a new extension for Ethernet to be made which re-frames Ethernet packets for easy redundancy. So, basically an Ethernet wrapper which simply numbers the packets and passes it over two separate lines or over two different wavelengths in the same fiber. Then the receive discards the packets which come late. It obviously won't resolve bottleneck related packet loss, but it will help to resolve the issue of glitch related packet loss.
Things like trying to force proliferation of BGP (or similar) routing technologies on an international scale would simply be irresponsible. Though, I'd imagine that governments would love it as it would simplify line snooping substantially for them.
Euh... their are more then 13 routes, their are 13 addresses (prefixes) but their are many, many more routes, most of those 13 prefixes are announced in many places it's called anycast and their aren't just 13 servers either. Every one of them is a cluster of machines and as many use anycast their are multiple clusters per 'root nameserver'.
New things are always on the horizon
Can we please have a tag "moronswithnobasicunderstandingofthetechnologyproposestupidsolutions" ? The article is mostly fear-mongering and a a waste of time. Should we be looking at what every idiot on the planet thinks about something he doesn't understand?
If so, can I write something on how bad particle physics is, because there are always problems with the accelerators and they carry a lot of energy and can open black holes?
As on the BGP hijacks, etc. - there are BGPmon and a ton of other projects that track the internet. There are established ways to stop all leaks/hijacks within a hour or two, and there's the way of making the person responsible NOT do that again. Go read on NANOG or a similar list the discussions on the topic, they're far more useful.
it's hard to work out if your joking, ignorant or stupid
Not true. There's a lot of stupid ignorants joking while they work out in my gym.
Yes, but since the Congress controls DARPA's budget directly, they effectively control them: http://www.wired.com/dangerroom/2008/09/darpa-budget-sl/
Dilbert RSS feed
Why don't we let congress fix it ? We'll be back to running RIP on the internet backbone before anyone can say "it doesn't scale".
BGP works, and all secure origin (never mind secure path) bgp announcements require and effect a total government takeover. It basically brings internet routing under government control, and the government (ICANN) key can take any IP offline, through revoking it's authorization, without warning and without recourse.
About the only thing that could remain operational without government fiat would be p2p networks (although thepiratebay would be screwed).
Let's hope there aren't too many democrats here, otherwise I probably shouldn't have said that.
Yea the article (and def summary) are clueless. As a network architect for a tier1, I assure you its pure FUD... EVERYONE uses ACLs on their edge... and definitely prefix limits too.
Ogre Wedding Planners llc.
Where's BGPSEC when you need it?
now we need to go OSS in diesel cars
/don't know if serious
:. Ultimate Control Dedicated/VM Servers
Well this artical is pure BS
"Uh Hacker told Uh Panel Uh thing, and now we're all gonna die". I dunno, I might have appreciated some links to sources discussing the events in more detail or filling in some, any of the gaping informational holes.
"Routing errors also blocked Internet access in different parts of the world, often for millions of people, in 2001, 2004, 2005, 2006, 2008 and 2009." ORLY? Certainly you could name these incidents or link somewhere, or do you expect me to google "routing error 2004" and figure out which event you are talking about? Or are you just pulling dates out of your ass? Occam's Razor suggests the latter theory.
"Soon, even Internet users in the U.S. were deprived of videos of singing cats and skateboarding dogs for a few hours."
8I
Well why didn't you let us know how much was at stake from the get go, then? Holy shitballs, something must be done right away!
People willing to trade their freedom of expression for temporary entertainment deserve neither and will lose both.
As someone who's accidentally announced the entire Internet routing table to an ISP when setting up a dual-homed configuration, I can confirm that good upstream ISPs do BGP filtering. I was trying to troubleshoot what was going on, and the Tech on the other end was helpful enough to tell me that I was sending him the full route table. Fortunately they had filters in place to stop them from going out any further and impacting anything. But I had it clearly demonstrated to me how important filters are on both ends of the connections.
Stop hurting the English language! What has it ever done to you?
key can take any IP offline, through revoking it's authorization, without warning and without recourse.
The only solution to that one I see is: all certificates should be irrevokable.
If you want to stop someone announcing something they have ever been authorized to announce, you have to follow the traditional channels.
English is not my first language. Sorry about that.
New things are always on the horizon
'It amazes me every day when I get into work and find it's working.'"
Sounds like he is ready to start administrating an exchange server.
Having to work for a living is the root of all evil.
xkcd explained this a while ago. Basically, if the internet ever *stops* working, even for a few seconds, alarms go off and people panic and do anything necessary to get it working again immediately. It turns out this is actually a fairly reliable system.
Cut that out, or I will ship you to Norilsk in a box.
Missing the point entirely.
The stories and info posted here are artistic works of fiction and falsehood.
Only fools would take it as fact.
The author does seem to be sensationalizing a bit. Though one can zoom in and criticize the frailty of a given component of the system, the overall system is far less frail. This is analagous to me to hard drives in a sever room. If I have more hard drives (which is the case when distributing your data for resiliency purposes , ie. RAID), there's generally going to be more individual HD failures. However, there's less data loss. Systems that are designed to accommodate & recover from failures (such as the Internet) don't necessarily mean less failures, but hopefully mean less impact on the users of the system when a failure does occur.
Actually, I'd believe it. Or, at least that it's true enough enough of the time to cause a problem.
One, you probably are a super-genius. Relatively speaking. In your area. If you're interested in your work you're in the top 1% at least. If you've got moderately wide training so that you use the right tools instead of beating what you know into the wrong hole you're far ahead of the game.
But it's not just the individual techs. There's easily at least one trained network admin for each ISP, and many work there, but it's amazing how many ways a company can find to sabotage its experts and their best efforts.
One ISP I'm familiar with had problems with spam, or rather, with saying no to money. They had an anti-spam department (one guy, but still enough if they let him work) and they claimed to be trying to stop it but whenever he'd identify a spammer (new customer, thousands of identical spamming outgoing emails...) and cut them off their regional partner would call up bitchy about a lost customer, they'd override the anti-spam dept and turn the account on, and then when the entire organization ended up blacklisted profess ignorance.
So yeah, huge multinational organization screwing things up worse than you can imagine, on a daily basis. Worse, over a $25/month subscriber account.
I've been overridden on purchasing decisions before for trivial amounts and spent more in time directly fixing the shortcoming. For instance, I bought two testing machines and speced $80 nvidia cards (back in ~2000 when they just worked and ATI just didn't), has this changed in favor of crappy $50 ATI cards (not even a good but older like the ones I specced, but the cheapest card that existed just to make crap PCs possible.) Together that saved the company $60 or so, most of which the manager wasted chewing me out for specing gaming cards for testing machines (blah, blah, blah, nvidia=games, blah, blah) and wasted literally THOUSANDS of dollars in fighting with those. (My job involved repeatedly installing various version of Windows (from scratch, couldn't just ghost/dd this part of it) and the ATI cards took more time, reboots, and caused more crashes, all of which had to be investigated to make sure they weren't caused by our product.)
"It's as if a driver had to get from Philadelphia to Pittsburgh without a map, navigating solely by traffic signs he encountered along the way — but the signs weren't put up by a central authority. If a sign pointed in the wrong direction, that driver would get lost."
Such a bad analogy on so many levels. The driver (the traffic) is never really aware of the route they are taking. Ignoring this for a second, a slightly better analogy would be that the driver gets a updated map of the route at each city along the way. And these maps are almost always the same.
An even better analogy is the usual post office one - but that's not as useful for trying to spread panic in the interwebs
This article is total rubbish. Who is this "General Manager". can I have his job cause he is an idiot. DNS is reasonably secure for a dynamic system we can certainly detect poisoning fairly quickly and we have firewalls. If a routing table gets weird it is not such a big end of the world thing... it talks to the next router and things straiten out.