Slashdot Mirror

A farmer in Michigan took out the internet with a backhoe for several hours. His farm is on the I-94 corridor and he accidentally cut through Merit's backbone connection. It made being the LMS administrator for a college with 500+ online courses a lot of fun. Phone rings: "Hi, I can't connect to my online classes!"

I think it's a matter of Cogent trying to strongarm its position. It wouldn't be the first time Cogent has done this and it certainly won't be the last. Doing a Google search for "peering dispute", and not including Comcast (to exclude the Comcast vs. Level3 dispute since it's newer and ongoing), almost every old entry involves Cogent duking it out with someone. They win customers on price, but things seem to be lopsided enough that they get into a scuffle with a number of the other Tier-1 providers.

Mike from HE spells it out pretty clearly from almost 2 years ago on the NANOG list:

http://www.merit.edu/mail.archives/nanog/msg01006.html

I have no reason to think that their stance has changed any.

More detail:
http://lists.ucc.gu.uwa.edu.au/pipermail/lore/2006-August/000040.html

http://www.merit.edu/mail.archives/nanog/1997-04/msg00340.html
http://www.merit.edu/mail.archives/nanog/1997-04/msg00444.html
http://portal.acm.org/citation.cfm?id=347428&dl=ACM&coll=

http://web.archive.org/web/20070328170121/http://www.riverstonenet.com/support/bgp/design/index.htm

More detail:
http://lists.ucc.gu.uwa.edu.au/pipermail/lore/2006-August/000040.html

http://www.merit.edu/mail.archives/nanog/1997-04/msg00340.html
http://www.merit.edu/mail.archives/nanog/1997-04/msg00444.html
http://portal.acm.org/citation.cfm?id=347428&dl=ACM&coll=

http://web.archive.org/web/20070328170121/http://www.riverstonenet.com/support/bgp/design/index.htm

that one didn't work, it was actually http://winmedia.merit.edu/arin

There is no denying, there are more IPv4 /8's allocated this year, then there are left in the pool for the RIR's.

http://www.merit.edu/mail.archives/nanog/msg13209.html

What a lot of verbiage to say:

Some routers have bad BGP implementations that handle attributes longer than 255 bytes incorrectly

Some of those routers will drop a BGP connection if thet get such an attribute.

The article makes it sound as if RIPE is in the business of distributing routes to BGP routers.

(See http://www.merit.edu/mail.archives/nanog/msg11505.html for details).

The Internet was created by Congress in the mid-eighties it is federal property. It's the opposite of a taking to require net neutrality, that is the premise on which the Internet was founded. At the time we fought back a GOP effort to have an all-private internet. Can anyone honestly argue that an all private internet would have grown as fast in the last 25 years as this one has? It would be completely fragmented to begin with, there would be tolls to overcome at every step of the way (pay a toll to leave your house (which we do) then another to reach the next ISP, then another and another....; this was honestly the model the GOP was pushing for). The entire internet would be as successful as Murdoch's pay-wall lamestream media is.

Just a few nitpicks here:

The Internet didn't even allow any commercial traffic until 1992.

The National Science Foundation funded the US Internet backbone (ran by Merit Networks, a collection of universities) until 1995. So, a plan to have tolls at every stop would have, if nothing else, benefited the government the most. If that were the case, do you really think that the government would have sold the backbone off?

Incidentally, the government selling the backbone off is what caused the current situation, because several of the telcos (AT&T and Verizon) own pieces of the US Internet backbone and are using that as leverage.

You hawks would be funny if some of you didn't hold power.

Out of curiosity... does that make that IP space sort of permanently black-listed? e.g., if the "bad guys" go out of business and "good guys" buy the IP space... how do the new owners clear the IP space of its bad name?

Seems like a shame to start throwing IP space away because there's no way to make it clean again.

At times, yes.

See this for a recent incident involving the Atrivo/RBN incident.

There was a recent thread on the topic on the NANOG mainling list, see http://www.merit.edu/mail.archives/nanog/msg02874.html

There was just recently a large discussion about this topic on NANOG. The mailing list archive where the thread begins can be found here: http://www.merit.edu/mail.archives/nanog/msg20241.html

Gee, I wonder why Verizon would think that consumers don't need VOIP? Perhaps competition has something to do with it...

http://www.merit.edu/mail.archives/nanog/msg19609.html

The president of unWired (a much more reputable ISP) has also blocked the same server. A DDoS was apparently attacking said server which wast travelling over both lines. According to this post, the block was due solely to stop the DDoS.

http://www.merit.edu/mail.archives/nanog/msg16816.html: Service to South Santa Clara county [and northern Santa Cruz county] is completely down: Internet,
landline, and cellphones. Both Verizon and AT&T are affected. 911 is also down. My cellphones show one or no bars. Normally they are all four bars.

The idea that all of that is lumped in one fiber bundle is mind boggling.

[apparently verizon was leasing bandwidth on the at&t fiber. ETA for repair sometime between 8 pm and midnight tonight.]

http://www.merit.edu/mail.archives/nanog/msg16843.html has the email archives of the North American Network Operators Group. One comment: That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling. That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue. That there is no over-under wide-area back-up coverage for the cellular canopy ... We posture and orate about being prepared for terrorist attacks and natural disasters, and then events like these reveal the reality: The emperor has no clothes.

The unintended consequence of this is that every user on a system is going to get a fixed ipv6 ip and ipv4 traffic would be gradually phased out. Why bother with the administrative burden of issuing an IP address via dhcp and tracking it, when, you could have an ipv6 theoretically assigned to a customer for the life of a device.

You _ARE_ kidding, right?

Maybe you should check out some information about ipv6 before you make more of a fool of yourself.

There is quite a bit of confusion, and it appears that people like you are the ones that are spreading it. How about just a little ipv6 delivery model to end customers?

The unintended consequence of this is that every user on a system is going to get a fixed ipv6 ip and ipv4 traffic would be gradually phased out. Why bother with the administrative burden of issuing an IP address via dhcp and tracking it, when, you could have an ipv6 theoretically assigned to a customer for the life of a device.

You _ARE_ kidding, right?

Maybe you should check out some information about ipv6 before you make more of a fool of yourself.

There is quite a bit of confusion, and it appears that people like you are the ones that are spreading it. How about just a little ipv6 delivery model to end customers?

There's more than three e-mail messages - that's just the first NANOG thread on it. The second thread is longer.

I suppose that a networking event with one of our upstreams was behind that router?

3/11 (invalid or corrupt AS path)

Or maybe I'm behind that router?

Apparently many Solaris systems restarted. People at NANOG are reporting this. A few banks' systems were rebooted as well: TD, Scotia, American Savings Bank, US Bank, and many more...
I saw many operating systems rebooting, even though this did not happen the last time in 2005.
Good thing I use ZFS on FreeBSD, and after I changed the loader.conf, I have a system that has stayed up for more than 2 months now, including last night.

That was the Kashpureff attack, not the Kaminsky attack. Your understanding of DNS cache poisoning attacks is unfortunately about a decade out of date. All major resolver implementation now do "bailiwick checking" and aren't fooled by crude, cheap tricks as you describe.

The Kaminsky attack does use forged packets, which then poison the cache with bogus NS records in ways that are not blocked by bailiwick-checking. These bogus NS records then "redirect" future queries of names under the same delegation point. Yes, using TCP exclusively would add much more entropy to DNS transactions, and thus make them much more resilient to forgery and thus to Kaminsky attacks.

But, at what cost? TCP is a hog, and typical DNS servers perform many millions of queries a day. Tens of millions and even hundreds of millions, are not uncommon.

Also, the DNS standards explicitly say that TCP is used for ordinary queries only as a fallback in case the response doesn't fit in a UDP packet -- and since the introduction of EDNS0 it's actually becoming quite rare for TCP fallback to become necessary. So the standards would have to be updated, and DNS software would then have to be modified to reflect the new standards. DNSSEC has a huge head-start on your "TCP exclusively" proposal along the standards-approval process.

Lastly, many firewall rulesets wouldn't allow TCP queries and responses as a regular occurrence, so they would need to be updated as well

All of this would take many years to implement. From a cost/benefit standpoint and a how-soon-to-implement standpoint, DNSSEC comes out ahead of "TCP exclusively" and what you get when all is said and done is superior protection against Kaminsky attacks.

"TCP exclusively" isn't a particularly original idea, by the way, see http://www.merit.edu/mail.archives/nanog/msg10298.html (August 9) and the subsequent discussion

More discussion on NANOG Mailing List

10/30/08 Sprint / Cogent

http://www.merit.edu/mail.archives/nanog/threads.html

Tip: The probability of finding more accurate info on NANOG than here seems to be higher.

Check out the NANOG thread on this here: http://www.merit.edu/mail.archives/nanog/msg11573.html.

An informative link about the AS7007 incident: http://www.merit.edu/mail.archives/nanog/1997-04/msg00444.html

Go read the thread on NANOG. Or read the timeline here: http://www.renesys.com/blog/2008/02/pakistan_hijacks_youtube_1.shtml

The way this happened is the result of a fundamental weakness in BGP. A more specific prefix will trump a less specific one, so anyone who has a valid peer can advertise a more specific route and hijack IP space. This is frequently used by Cybercriminals to squat on unused IP space in larger netblocks.

There have been proposals to address this issue for some time. Maybe, now that a major site has fallen victim, something will actually be done about it.

Of course, we could solve the problem the way it was when the Internet was first designed: only allow trusted entities to connect at all. IMNSHO, if the Islamic world don't want to be in the 21st century, that's their choice, but they can't have their cake and eat it too. Unless and until they agree to the basic principles of the Internet: freedom of association and speech, they shouldn't be allowed to connect at all.

This was discussed yesterday, but somehow the mods didn't control the discussion degenerating into a debate about circumcision.

There is a NANOG thread about this. Apparently a more specific IP route was advertised.

As you guys probably know Youtube's IP's are being hijacked. Trace:
~ $ host youtube.com
youtube.com has address 208.65.153.253
youtube.com has address 208.65.153.238
youtube.com has address 208.65.153.251
[Same /24]

701 3491 17557
64.74.137.253 (metric 1) from 66.151.144.148 (66.151.144.148)
Origin IGP, metric 100, localpref 100, valid, external
Community: 65010:300
Last update: Sun Feb 24 11:33:05 2008 [PST8PDT]
3491 17557
216.218.135.205 from 216.218.135.205 (216.218.252.164)
Origin IGP, metric 100, localpref 100, valid, external, best
Last update: Sun Feb 24 10:47:57 2008 [PST8PDT]

So, it seems that youtube's ip block has been hijacked by a more
specific prefix being advertised. This is a case of IP hijacking, not
case of DNS poisoning, youtube engineers doing something stupid, etc.
For people that don't know. The router will try to get the most specific
prefix. This is by design, not by accident. This is a case of censorship
on the internet. Anyways, I hope this doesn't get into a political
situation, and someone stops this.

What action are you going to take? Are you going to filter
announcements from AS17557, or just filter that specific announcement?
Considering youtube is a fairly high-traffic website I think that other
operators are just going to start filtering that AS. This is a great
example of global politics getting in the way of honest corporatism.
This is also an example of how vulnerable the internet is, and how lax
providers are in their filtering policies. I don't know how large
Pakistani Telecom is, but it I bet its not large enough that PCCW should
be allowing it to advertise anything.

They announced a route out of AS 17557 sending all traffic from ANYWHERE on the Internet to a black-hole in Pakistan. The effect was to make YouTube unreachable from ANYWHERE until the route was filtered by the backbone providers. They claimed it was an "oops".

Am I the only one who is fed up with Islamic Medievalism? If they want to live in the stone age, let's send them there. It will also solve a good bit of the world's over-population problem.

According to the NANOG list (North American Network operators Group), Comcast has been discarding emails that include a link created using EasyURL, one of many services designed to provide shortened URLs for email links. This could be an anti-spam policy, as URL forwarding through these services is sometimes used by phishing scams to obscure the link's true destination.

Funny. What makes you think that they have that capability? Even when the traffic is distinct enough to filter, I'd think inspecting it all would take quite a lot more hardware than they're used to using...

You know... A while back I rambled on about lazy ass engineers who have the capability to stop botnet DDoS traffic. Went unanswered, some mumbled those with the capabilities to stop it did nothing. As for the financial fraud occurring, its unfortunate but will likely be resolved too. Its a shame when people go out of their way to make things better only to be trampled upon. Kudos to Castlecop's team for their resiliency. As for the network engineers who peruse this site, this could one day be you too. Think about that before you decide to just brush away calls for assistance when dealing with botnets and attacks.

I think the article may be wrong.

Over on the NANOG mailing list, which has a lot of people from the major U.S. backbones and networks subscribed to it, it is being reported/said that the line was Cogent's, not Level 3's, and that Cogent at one point had an advisory up about it.

Lots of people posted traceroutes that seem to confirm that it was definitely Cogent that took the hit. Packets were basically going all over the place on their network yesterday, and people who had fixed their routers to prefer Cogent over other backbones (apparently Bell) were having some slowdowns as a result.

See http://www.merit.edu/mail.archives/nanog/msg02483. html or here for the thread index (lots of followups).

I think the article may be wrong.

Over on the NANOG mailing list, which has a lot of people from the major U.S. backbones and networks subscribed to it, it is being reported/said that the line was Cogent's, not Level 3's, and that Cogent at one point had an advisory up about it.

Lots of people posted traceroutes that seem to confirm that it was definitely Cogent that took the hit. Packets were basically going all over the place on their network yesterday, and people who had fixed their routers to prefer Cogent over other backbones (apparently Bell) were having some slowdowns as a result.

See http://www.merit.edu/mail.archives/nanog/msg02483. html or here for the thread index (lots of followups).

same song (same verse even) ... read the full thread over on NANOG; especially interesting is Randy Bush's followup, featuring some slides (specifically, slide 20) he presented recently that basically say "yeah, there's a problem; no, the sky is not falling; none of the forced-cutover plans thus far presented have fully taken into account operational and business issues. Careful thought and deliberate action (rather than panic and haste) are needed here to avoid creating problems we'll be living with for the next 30 years." (Apologies if Randy thinks I'm paraphrasing him incorrectly; I doubt he spends much time reading /. though. :))

Here is the issue with that... Networks suck at times. How would you feel if you're paying $60.00 per month but the connection between broadcaster and your provider keeps having issues? Its not similar to cable in the sense that the only issues you would have with cable would be with the provider. You could call them bitch and moan. Try calling them because some backbone went down and see what they'll tell you. So while you wait for someone outside your cable company to fix their network, I will bitch and moan to my cable provider then seek a month of something for free from them, while you wait...

Anyway, it is an article in 1999. IMHO, I like this story more... Metcalf ate his column http://www.merit.edu/mail.archives/nanog/1997-04/m sg00192.html

What can I do, short of keeping an eye on my credit and letting the school get away with yet another blunder?"

One resourceful alum dispensed with hints, threats and allegations, and simply billed OU for the time she spent checking her credit status. Calling the university "fully liable" for her outlay of time, she e-mailed an invoice for three hours of work at her "usual billing rate" of $165 an hour.

In its latest response, OU Legal Affairs Director John Burns has contacted the firm the woman works for, asking for confirmation of her hourly rate.

See this mailing list message, which points to this PDF presentation.

The good news: long term, I think IPv6 is desirable. Thus, I like seeing a large organization pave the way. Let them get the kinks out. Let them find out what all goes wrong. Let them blaze the trail so we can ride on their coattails. Let them incur the big expense.

Several others have already stepped up to the plate and have implemented IPv6. Here are some notes asked when Comcast did their presentation at NANOG about how their IPv6 migration of their cable modem pools worked.

Lol. Here are some more from nanog:

http://www.merit.edu/mail.archives/nanog/msg16183. html

http://groups.google.com/groups/profile?enc_user=q ybeTxcAAAAfIHYUZ1VU5sHfqG_AKbJWly7yRNrpKyy7Nyz7Hby Iyw

http://groups.google.com/group/alt.irc.undernet/br owse_thread/thread/29ac57045fc32f9/44f9a2c8d9bb13f 1#

You'd think that if the e-mails of him were spoofed there would be some evidence complaining about it publically back then and trying to discredit them. Notice how he is so quick to try and deflect attention from the actual questions.

Maybe thats why he created the botnet list, to get more bots?

No - here are the details of the 40,000 or so AS numbers handed out by IANA. There is also a set of weekly statistics posted on NANOG which shows that 21,484 of these AS numbers can be seen in the global routing table. Only 8,867 of these guys advertise a single prefix, so to get 181,747 routes there are a lot of ASes advertising multiple prefixes.

Start here:
http://en.wikipedia.org/wiki/IP_Transit_(Internet)
and follow the links, especially peering and Tier 1.

Once you know the basics, browse/search the NANOG list to learn more:
http://www.merit.edu/mail.archives/nanog/

what went wrong in Level3: http://www.merit.edu/mail.archives/nanog/msg13166. html

* From: Alex Rubenstein * Date: Fri Oct 21 13:45:24 2005 Gary, I understand your statement, but I am sure the gentleman below does not. If you want a story to be done, so that the world can see how something like this can impact thousands of businesses, the best bet would be to help educate this guy so that he has something to write. Are, were you trying to scare him off from doing a story? Personally, I am quote fed up with the issues that the huge providers have and cause, yet never have anyone document it, find out about it, or do anything about it. I laud this guys effort for actually trying to do his job and expose something that needs to be exposed. I am now putting on my level-3 bullet proof jacket, and will be looking over my shoulder for the next 3 NANOGs.

Seems like someone want the media attention, /. is not the worst choice here. Of course, the sensationalism could have been avoided, I can grant you this. Does this warrant cursing, hmmm....

So everybody with an alternate path around Level 3 should have routed around them properly. And yet, they weren't routed around. That's a concern.

If you went down because of this outage, your provider is totally dependent on Level 3, which is not good. This is a useful warning - if you went down, and your operation is important enough that it needs to stay up, you need to look very hard at your provider's upstream connectivity. Better hosting services have connections to several Tier I providers, just in case something like this happens.

Is this the end for Level 3?

No, of course not, you blithering imbecile. L3 had a 2 hour global routing meltdown. Now, it's fixed. Whilst their routes were flapping, other carriers saw transient increases in latency and some problems with reachability, to some sites. However, everything continued to work properly for non-L3 customers. Two hours later L3's routes are back and working properly. End of story, nothing to see here, move along please.

Slashdot editors, do you really expect us to believe that no-one had submitted a more coherent or accurate story than this one? Come on, for heaven's sake.

Anyway, a network engineer's view can be seen in the overnight traffic on NANOG: http://www.merit.edu/mail.archives/nanog/2005-10/ "Tier One ISPs dying" indeed. Worst. Story. EVER.

NANOG has been on fire with posts about this issue over the past few days. The following two from Leo Bicknell do a good job of explaining why this sort of thing would happen, why nobody in particular is The Bad Guy[tm], and why this issue has no relevance to the issue of internet resilience in the case of natural or manmade disaster:

http://www.merit.edu/mail.archives/nanog/msg12302. html
http://www.merit.edu/mail.archives/nanog/msg12350. html

NANOG has been on fire with posts about this issue over the past few days. The following two from Leo Bicknell do a good job of explaining why this sort of thing would happen, why nobody in particular is The Bad Guy[tm], and why this issue has no relevance to the issue of internet resilience in the case of natural or manmade disaster:

http://www.merit.edu/mail.archives/nanog/msg12302. html
http://www.merit.edu/mail.archives/nanog/msg12350. html

Folks on the NANOG list are discussing this rather vigorously at the moment. You can follow the thread here: http://www.merit.edu/mail.archives/nanog/2005-10/

There was a recent discussion on NANOG on this topic which ended with a fairly definitive statement from One Who Knows This Shit (actually it was Dan Golding) that virtually no colleges use SSNs as unique IDs any more; but that they have to maintain *old* data, which *did* use SSNs as UIDs. I'm paraphrasing, badly; go read the archived post.