Slashdot Mirror

← Back to Stories (view on slashdot.org)

Why Google Went Offline Today

Posted by Soulskill on from the ok-who-tripped-on-the-cable dept.

New submitter mc10 points out a post on the CloudFlare blog about the circumstances behind Google's services being inaccessible for a brief time earlier today. Quoting: "To understand what went wrong you need to understand a bit about how networking on the Internet works. The Internet is a collection of networks, known as "Autonomous Systems" (AS). Each network has a unique number to identify it known as AS number. CloudFlare's AS number is 13335, Google's is 15169. The networks are connected together by what is known as Border Gateway Protocol (BGP). BGP is the glue of the Internet — announcing what IP addresses belong to each network and establishing the routes from one AS to another. An Internet "route" is exactly what it sounds like: a path from the IP address on one AS to an IP address on another AS. ... Unfortunately, if a network starts to send out an announcement of a particular IP address or network behind it, when in fact it is not, if that network is trusted by its upstreams and peers then packets can end up misrouted. That is what was happening here. I looked at the BGP Routes for a Google IP Address. The route traversed Moratel (23947), an Indonesian ISP. Given that I'm looking at the routing from California and Google is operating Data Centre's not far from our office, packets should never be routed via Indonesia."

7 of 110 comments (clear)

  1. All your packets are belong to... by Adeptus_Luminati · · Score: 5, Interesting

    ... Network Admins who have no clue. Like when just 4 years ago, Pakistan took down Youtube...
    http://securitywatch.pcmag.com/dns/285152-pakistan-takes-youtube-down

    Clearly this should be on the agenda for the new "Cyber Reserves" of the department of Homeland Security. If Google can be taken down by accident in parts of the world, then it certainly can be taken down on purpose. Route filters are your friends!

    CYBER RESERVES: http://www.techradar.com/news/internet/department-of-homeland-security-recruiting-for-cyber-reserve-1109906

    --
    No trees were killed in the making of this post; however, many trillions of electrons were horribly inconvenienced.
  2. Re:Will DNSSEC help with this? by X0563511 · · Score: 5, Informative

    Nope. DNS doesn't mean shit if the routers are sending your traffic to the wrong place. (DNS points to an IP, which is (supposed to) point to the target machine. If that last part isn't working, the first part won't work no matter what)

    --
    For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
  3. Root cause was PCCW, not Moratel by Aqualung812 · · Score: 5, Interesting

    From TFA:

    Someone at Moratel likely "fat fingered" an Internet route. PCCW, who was Moratel's upstream provider, trusted the routes Moratel was sending to them. And, quickly, the bad routes spread.

    Yes, someone at Moratel screwed up, but this is exactly why upstream ISPs should never allow advertisements from their customers for networks that their customer does not control.

    PCCW is to blame for allowing this to happen. Never trust customers with things that don't belong to them.

    --
    Grammer Nazis - I mod you "troll" unless you actually add something on-topic. Yes, I know I have mispellings in my sig.
    1. Re:Root cause was PCCW, not Moratel by Anonymous Coward · · Score: 5, Interesting

      PCCW is to blame for allowing this to happen.

      Again. They were also the upstream for the Pakistan-takes-down-YouTube fiasco.

    2. Re:Root cause was PCCW, not Moratel by vlm · · Score: 5, Informative

      Yes, someone at Moratel screwed up, but this is exactly why upstream ISPs should never allow advertisements from their customers for networks that their customer does not control.

      Another important point is its twenty freaking twelve and at a "respectable" ISP this was part of my job a decade ago. Too many customers try advertising too much stupid space. Rule number one for a BGP operator... never trust whats incoming from nobody. Rule number two is when you call in for support and 1st level call center tells you to reboot everything, tell them to F off and transfer directly to my desk unless you want to learn the joys of route flap dampening. Rule 2 is hilarious when there's a genuine catastrophic failure and like 30 customers all want to talk to me personally because all their sessions dropped when the Juniper caught fire or whatever it was... so beware.

      There are only three things funnier than a fat finger BGP route advertisement:
      1) Why can't I advertise my old /28 from AT&T on your network? Well dumbass thats their space not "your" /28, and secondly on the civilized internet everyone filters at /24 or bigger to keep out the riff raff so even if I was dumb enough to advertise a subnet of another ISPs space, no one gonna see it past our borders.
      2) Multihomed people who basically accidentally try to turn themselves into a transit network. Oh, you connect to L3? How nice. You don't really want to advertise that the whole freaking internet can route thru you to reach it, do you?
      3) Advertising space in BGP, maybe redistributing a static or null route, doesn't mean you can actually route it on your internal network. OK I see your measly little /20 and now that you let me know to update our filters, we can all see it via us on any looking glass in the world. Yes I'm quite sure it doesn't work and no its not BGPs fault, go fix your internal routing protocol and filters and GTF off my phone so I can go back to sleep. No for the 20th time its not a BGP problem just look at the looking glass I'm not filtering you anymore.

      The primary problem is BGP is a social layer 8 protocol for how network managers... manage. You don't learn that shit in a weekend training class where they teach you the exact syntax of "show ip bgp neighbor" or by memorizing AS path regex syntax or whatever. At least up till I got out of the business half a decade ago, no one was teaching anything like "this is how to use BGP while not making an ass outta yourself" class. No book either. I think "Internet Routing Architectures" and maybe the name Halabi sticks in my mind as a good theoretical book as I recall, but no one had a practical "real" hands on class or book. I suppose I shouldda done something about that but its been a long time now. Then again I've probably forgotten more about BGP that most one week CCNP bootcampers will ever know, so maybe its not too late anyway. Another "in my infinite spare time" project.

      Sorry if I've offended any /.er I've actually talked to on the job who Fed up, nothing personal... But since I carefully identified noone by name, at least no one knows you Fed up. If today I failed to offend anyone who Fed up while I was doing front line BGP support then I'll try harder next time. BGP is kind of the network engineering version of giving little kids boxes of matches. Its surprising more networks don't burn down, but boxes of matches are so blasted useful if you actually know how to use them safely so its not like we'll ever get rid of it.

      --
      "Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
  4. China already did this in 2010 by hydrofix · · Score: 5, Interesting

    China Telecom also hijacked web traffic to US government websites in April 2010 for 17 minutes. At least that incident seems to have been a purposeful disruptions to capture sensitive data and/or try out a novel cyberwarfare tactic.

  5. Re:Happens all the time, just not usually to Googl by Anonymous Coward · · Score: 5, Funny

    Errr, yeah, what about that porn link? That's really... that's awful. I can't believe that they would have that there. Man, porn. Anyway, I've just got to go and do... a thing. Nothing interesting, don't you worry about it, just... Go about your business.