Slashdot Mirror

← Back to Stories (view on slashdot.org)

Google Takes Blame For Internet Disruption Across Japan (theregister.co.uk)

Posted by msmash on from the internet-disruption dept.

An anonymous reader shares a report: Google on Saturday accepted responsibility for the widespread internet disruptions Japan experienced the previous day. The search engine giant apologized for the trouble, saying it was caused by an errant network setting that was corrected within eight minutes of its discovery. Google did not say whether human error or a technical malfunction was to blame. The disrupted services used internet connections provided by NTT Communications Corp. and KDDI Corp., both of which said Friday that the issues were caused by a change in the flow of data traffic. From a report on The Register: The trouble began when Google 'leaked' a big route table to Verizon, the result of which was traffic from Japanese giants like NTT and KDDI was sent to Google on the expectation it would be treated as transit. Since Google doesn't provide transit services, as BGP Mon explains, that traffic either filled a link beyond its capacity, or hit an access control list, and disappeared. The outage in Japan only lasted a couple of hours, but was so severe that Japan Times reports the country's Internal Affairs and Communications ministries want carriers to report on what went wrong.

59 comments

  1. You know you've reached the summit by Dirk+Becher · · Score: 3, Funny

    if you can claim responsibility for everything bad that happens on the net and everyone believes you immediately.

    1. Re:You know you've reached the summit by Anonymous Coward · · Score: 5, Informative

      Hmm, actually we *knew* it was them in the first place the moment it started, most non-joke internet network engineers refuse to fly blind, so there are probes and monitors everywhere in the Internet control plane (DFZ BGP routing). E.g. read: https://bgpmon.net/bgp-leak-causing-internet-outages-in-japan-and-beyond/

      Also, most internet network engineers will place a lot of the blame on *Verizon*, not Google. "route leaks" are a *fact of life*, they will happen at least once to everyone. You *MUST* filter the routing plane to not accept crap from other autonomous systems you peer with, and Verizon *utterly failed* at doing it. Had they filtered, they'd have rejected the bogus routing from Google and avoided most, if not all of the damage.

      So, Google might have publicly taken the blame since it was their operational error that triggered the damage in the first place, but they are at most responsible for half of it... and *everyone in the field* knows it, Google included.

    2. Re: You know you've reached the summit by Anonymous Coward · · Score: 0, Flamebait

      You know you've reached the summit when you spend energy burning white males at the stake of diversity rather than making sure you're not breaking internet.

    3. Re: You know you've reached the summit by Anonymous Coward · · Score: 0

      Taking responsibility when you could just as easily pass blame is a surprising move from a megacorp. Wish the multi billionaires would act like mature adults more often.

    4. Re: You know you've reached the summit by Anonymous Coward · · Score: 0

      They could not reasonably pass blame. "You should have caught our mistake" is no excuse.

    5. Re: You know you've reached the summit by mrclevesque · · Score: 1

      "You know you've reached the summit when you spend energy burning white males at the stake of diversity rather than ..."

      It's not about whites or males, it was about someone sending a disruptive email to everyone in the company.

    6. Re: You know you've reached the summit by lucm · · Score: 1

      That's absurd. First, did you read it before saying that it was a "disruptive" email, or are you just one of those phonies who repeat things they don't know shit about? People who repeat their shallow understanding of biased news reports remind me of that coward security guard in The Stand who runs away from the lab and carries the disease in the wild.

      Second, the guy was fired because the document was made public, not because it was sent to everyone in the company. He was basically thrown under the bus by his employer for the purpose of appeasing the deranged twitter mobs, of which I suspect you are a member.

      --
      lucm, indeed.
    7. Re: You know you've reached the summit by Anonymous Coward · · Score: 0

      Are you kidding? We live in a world where companies deny responsibility for stuff they're more directly responsible for than this on a daily basis.

      Hell, ever called the cable company when your service goes down?

    8. Re: You know you've reached the summit by KGIII · · Score: 1

      Japan accepted 1, out of more than 8000 requests, refugee. That's actually better than the year before.

      Google was just punishing them for their lack of diversity! /s

      --
      "So long and thanks for all the fish."
    9. Re: You know you've reached the summit by Anonymous Coward · · Score: 0

      He did not send it to everyone at the company. He sent it to a rather limited distribution internal discussion list for self proclaimed sceptics.

      His massive error was that your only need the link to the document to access it. It wasn't locker down to be accessible by the list members only.

      Then someone else started sharing it more widely.

    10. Re: You know you've reached the summit by Ebsolas · · Score: 1

      If I had to put this to an analogy I'd say it's like walking through a dog park. If you don't look down you may step in something. Sure it's the responsibility of the dog owner to pick up after his/her dog but you still should be on the lookout.

    11. Re: You know you've reached the summit by Anonymous Coward · · Score: 0

      It doesn't matter if they claim responsibility. Everybody already knew it was them.

      They need to be punished for this. If you or I made a "mistake" like this and caused an outage across a whole country for hours, we'd be put in prison.

    12. Re: You know you've reached the summit by mrclevesque · · Score: 1

      Thanks for the correction and info.

    13. Re: You know you've reached the summit by mrclevesque · · Score: 1

      "First, did you read it ... "

      Yes, I felt it was mostly hand waving about how he blamed others for his attitude.

      "Second, the guy was fired because the document was made public, not because it was sent to everyone in the company ... "

      Thanks for the correction. I don't approve of his firing, at the same time it's what companies usually do, and blaming it on the public is a cop out.

    14. Re: You know you've reached the summit by lucm · · Score: 1

      Have you watched the movie Trumbo? Because that's exactly what's happening.

      --
      lucm, indeed.
    15. Re: You know you've reached the summit by mrclevesque · · Score: 1

      "Because that's exactly what's happening"

      Not by a long shot. For one, Google is a private company, while Trumbo is about the US government.

    16. Re: You know you've reached the summit by lucm · · Score: 1

      First, it's not just Google. It's most tech companies.

      Second, Trumbo wasn't fired by the US government. He was fired by private companies who didn't want to look bad.

      Same shit.

      --
      lucm, indeed.
    17. Re: You know you've reached the summit by mrclevesque · · Score: 1

      "Second, Trumbo wasn't fired by the US government. He was fired by private companies who didn't want to look bad."

      Trumbo was investigative by a government committee of the United States House of Representatives because of his beliefs, he was visited by the FBI, and he was jailed. The MPAA also issued a statement that Trumbo and people like him would not be permitted to work in the industry. That is a long shot from what happened at Google.

      "First, it's not just Google. It's most tech companies."

      I didn't say Google was the only company that fired employees for their attitude. All companies do, and in no way is it mostly in the tech sector.

    18. Re: You know you've reached the summit by lucm · · Score: 1

      It's always exhausting to discuss with aspies but you're clearly not a troll so let me summarize things for you.

      The climate of political correctness in the USA has crossed an important threshold: people being fired publicly not for incompetence or malice, and not even for badmouthing their employer, but merely for voicing opinions internally about subjects that are not even relevant to the business.

      This is similar, although not factually identical (yet) with what happened during the witch trials in Salem and the commie hunt under McCarthy. It's called mass hysteria.

      --
      lucm, indeed.
    19. Re: You know you've reached the summit by mrclevesque · · Score: 1

      "This is similar, although not factually identical (yet) with what happened... "

      No it's not. I think you are pushing the relation way too far. Damore was fired by Google. Trumbo was investigative by a government committee of the United States House of Representatives, he was visited by the FBI, he was jailed, and the MPAA stated that Trumbo and people like him would not be permitted to work in the industry.

      "people being fired ... merely for voicing opinions internally about subjects that are not even relevant to the business"

      Nothing new.

      "He was basically thrown under the bus by his employer for the purpose of appeasing the deranged twitter mobs"

      If you mean a big part of why he was fired was damage control on the part of Google, I agree.

      "The climate of political correctness in the USA has crossed an important threshold ... It's called mass hysteria"

      When I look at the interactions of some individuals, groups, and media, and how they're focusing on and treating these issues, I agree, but when I look more generally I don't think that's the case.

  2. How... by Drethon · · Score: 2

    did a company that does no provide transit services even manage to send a route table that was accepted for use? Just seems like a very exploitable issue there. Does google have authority or permissions that allows this, even though they don't actually have the capabilities?

    1. Re:How... by Anonymous Coward · · Score: 0

      Just seems like a very exploitable issue there.

      Exactly, somebody hacked the internet. All these *proofs of concept* are being tested in the wild. Looks like they work!

    2. Re:How... by Infiniti2000 · · Score: 4, Informative

      TFA explains how, and at the end of the TFA: "There are various proposals to tweak BGP to stop this sort of thing happening, but as is so often the case, implementation is lagging far behind requirement." In short, it's a known problem and eventually maybe someone will care enough to fix it.

    3. Re:How... by Anonymous Coward · · Score: 0

      Yup, trouble is that in the days of ARPANET computers trusted each other.

    4. Re:How... by dgatwood · · Score: 4, Interesting

      How did a company that does no provide transit services even manage to send a route table that was accepted for use? Just seems like a very exploitable issue there

      Multihoming. I'd imagine Google provides transit service, but only for their own IP blocks. Each Google datacenter almost certainly has multiple Internet connections to the world. As a result, they have multiple netblocks provided by multiple ISPs.

      If, through some unlucky DNS accident, a client on ISP A looks up google.com and gets an IP address provided by ISP B, it would take many more hops to reach that server via public Internet routes than by sending traffic to that datacenter's nearest router (on ISP A) and asking that router to forward traffic through the datacenter to the other set of IPs.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    5. Re:How... by Anonymous Coward · · Score: 1

      This happens because their peers were too lazy or otherwise unable to set up and maintain proper route filters for Google's ASNs. Problem is that with Google Cloud and Google Fiber (for business), google actually sometimes *does* provide transit, so it would be a difficult thing for all of their upstream providers to keep continuously updated -- instead they think "Well, they are Google. They are probably doing that all on their end and won't ever fuck this up."

    6. Re:How... by The+Raven · · Score: 4, Insightful

      BGP routing has no authority mechanism. Anyone can publish any route. This is not the first time this has happened, nor will it be the last.

      --
      "I will trust Google to 'do no evil' until the founders no longer run it." Hello Alphabet.
    7. Re:How... by Anonymous Coward · · Score: 0

      So if I want to push routing filters that will prioritize my company's packets, all I have to do is to create a subsidiary ISP so that transit providers become my "peers" and will let me administrate their networks?

    8. Re:How... by Anonymous Coward · · Score: 0

      The explanation is correct, but only makes sense to people who already understand the issue. And it's not "transit" if you advertise a route for your own addresses.

      The problem is that anyone can advertise any route. If you speak BGP, you can advertise routes for any addresses. They don't need to be your own. Only sanity checks at other providers can prevent you from "stealing" traffic, i.e. telling other providers to send traffic for addresses that aren't yours to your network. If you're small fry, your upstream provider will prevent you from advertising addresses that are not your own. Google is not small fry. The bigger you get, the less clear it's going to be what you are legitimately allowed to advertise and what not. Multihoming is one such thing that can make it difficult for an upstream provider to filter correctly, because with multihoming you advertise not just the address space you got from that upstream provider, but also from other providers (or provider-independent address space). But technically all IP address space allocated to an autonomous system is public information, so it should still be possible to filter routes. Providing transit services for other ISPs is another matter, if you don't just do it for subnets of your own address space. But Google does not provide transit.

    9. Re:How... by Anonymous Coward · · Score: 0

      BGP routing has no authority mechanism. Anyone can publish any route

      Route filters. Anyone can publish a route but everybody should ignore it.

    10. Re:How... by Drethon · · Score: 0

      Thanks for letting me be lazy today

    11. Re: How... by buchanmilne · · Score: 2

      Google fibre is definitely a separate AS from Google (Search, Youtube etc.) AS15169, not sure about GCP.

      AS15169 doesn't provide transit, so Verizon could have easily prevented this with just as-path filters (although prefix filters in addition would be better, but may incur more maintenance).

    12. Re: How... by buchanmilne · · Score: 1

      "Multihoming. I'd imagine Google provides transit service, but only for their own IP blocks. Each Google datacenter almost certainly has multiple Internet connections to the world. As a result, they have multiple netblocks provided by multiple ISPs."

      No, multi-homing doesn't require netblocks from multiple ISPs. Doing this is a hack and only feasible for very trivial networks. A much better approach is BGP multi-homing.

      Google (along with thousands of smaller "Enterprise" businesses) have their own autonomous system number(s) and IP allocations, and advertise these IPs to everyone they peer with. Google peers at about 180 internet exchanges, with most retail ISPs. Probably quite soon they won't need to buy transit ...

    13. Re:How... by Anonymous Coward · · Score: 0

      The original TCP/IP stack was meant for a LAN that could be trusted. Then Microsoft added it to Windows 95/NT in order to compete against UNIX.

    14. Re: How... by dgatwood · · Score: 1

      No, multi-homing doesn't require netblocks from multiple ISPs. Doing this is a hack and only feasible for very trivial networks. A much better approach is BGP multi-homing.

      Fair enough, but whether it's a single block owned by Google or one or more blocks owned by one or more ISPs, they still have to use BGP to advertise routes for a block of IPs that doesn't belong to their upstream ISP, so the distinction is largely academic.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    15. Re:How... by Anonymous Coward · · Score: 1

      BGP routing has no authority mechanism

      Why would you want one? All that would result from it is Yet Another Centralized Bureaucracy that would provide security theater to the public, create another barrier to entry for ISPs, and be a huge target for the likes of the NSA, FBI, etc. Nevermind all of those morons wanting to implement an "internet killswitch" or mandate a federal porn filter / Great Internet Wall of the USA.

      Long story short, given the current online political climate, adding new ways to screw up isn't advisable. There's no-one who could be trusted to run it.

    16. Re:How... by GESUS · · Score: 1

      Authority is given by every individual peer, by contract and implementation.

      They are all autonomous from each other like nations.

      There is no one table being sent, there is a bunch on routes in one or more updates. This can be added to the routing database on the receiving side IF THEY CHOOSE. They should have had filters and safe guards like a max amount of prefixes accepted over this peer.
      I would say that Google is taking the high road but the receiving ISPs are really to blame. They did not protect there customers appropriator.
      I know that this sounds as news to the general public and most of geekdom, if you are not working this field you will simply not understand how it works.

      This is not new, even if done by a bug or a fat finger or red eye. The severity and scale is unusual.
      This is all an artifact of the decentralized internet. That and misplaced trust in the awesomeness of Google inc.

      Apply your filters people!

    17. Re:How... by GESUS · · Score: 1

      Very well known and it is fixed. But not in the manner stated.

      The receivers just happen to trust Google to much and did not apply normal filters and checks. It is more like they did not enable the firewall because they could not be bothered to set up the necessary rules.

      That is why there is no fix. BGP is not broken, imperfect for sure. But it does not need to be replaced, the tweaks that are added almost always work with the internal usage side. Like inside an ISP where it is used for all kinds of information distribution.

      Don't blame BGP for what people do wrong.

    18. Re:How... by GESUS · · Score: 1

      Well, in this scenario where talking about Google breaking the internet.
      They with the help of the ISPs did it unintentionally this time.

      But if Google decided to orchestrate an attack on the internet, they could bring it to its knees in so may ways.

      Simply disabling all there local caches would overload most transit lines in the world. That would make everything suck.
      It they turned them into a zombie network, everything would break until they where disabled.

      The internet of the past is gone, it is no longer a mesh of thousands of ISPs and millions of sites. Now it is 70-90% 5-10 big players with there own infrastructure where 2-3 rival or exceed the biggest transit networks. The only upside is that it is in these entities own interest to keep this working optimally. Unlike some telcos that limit there users experiences.

  3. Sad but true... by __aaclcg7560 · · Score: 1

    Google is the new Godzilla, stomping on everyone's Internet connections.

    1. Re:Sad but true... by Anonymous Coward · · Score: 1

      creimer is the new Bigfoot, stomping on everyone's Slashdot user experience.

    2. Re:Sad but true... by Oswald+McWeany · · Score: 1

      Google is the new Godzilla, stomping on everyone's Internet connections.

      Does that make Bing "Mothra"?
      Duck Duck Go is "Megalon"?

       

       

      and Yahoo is "Tickle Me Elmo"?

      --
      "That's the way to do it" - Punch
    3. Re: Sad but true... by Anonymous Coward · · Score: 0

      LOL true that. Mod up.

    4. Re:Sad but true... by msauve · · Score: 1

      Think of the Friend Of All Children, Gamera.

      --
      "National Security is the chief cause of national insecurity." - Celine's First Law
    5. Re: Sad but true... by Anonymous Coward · · Score: 0

      Not new(s), he's been at it for at least a decade.

    6. Re: Sad but true... by Anonymous Coward · · Score: 0

      Not really true; Bigfoot is mythical, we only wish that creimer was mythical as well. Alas, he's far too real.

    7. Re: Sad but true... by Anonymous Coward · · Score: 0

      creimer becomes more obnoxious with each passing year as he grows older and more desperate for revenue streams to supplement the crap pay from his dead end job

  4. Preparations? by Anonymous Coward · · Score: 0

    Google testing mechanisms for removing wrongthink from the Internet?

  5. Distracted by social justice by Anonymous Coward · · Score: 0

    Maybe Google should focus less on rooting out wrongthink from YouTube and more on basic technical competence

  6. So much for "routing around" by mi · · Score: 2

    widespread internet disruptions Japan experienced the previous day.

    So much for the fabulous promise of "routing around damage"...

    --
    In Soviet Washington the swamp drains you.
    1. Re:So much for "routing around" by GESUS · · Score: 1

      Yeah, that does not work when the routing is not the problem.

  7. Wrongthink. by Anonymous Coward · · Score: 0

    Japan must be punished for being racist and not taking refugees.

  8. Finally a Google story makes it through. by Anonymous Coward · · Score: 1

    /. has been strangely silent about Google shenanigans.

    1. Re:Finally a Google story makes it through. by Anonymous Coward · · Score: 0

      Not counting the Damore debacle I take it?

  9. "TIFU by killing Japan's internet connection" by Anonymous Coward · · Score: 0

    Some Google employee probably.

  10. no need to call NTT! Yeah! by Anonymous Coward · · Score: 0

    Ok so it wasn't just me. Internet indeed WAS messed up.

  11. "Don't worry" by Anonymous Coward · · Score: 0

    ... they told me, Google isn't that influential.

    An accident can take down large chunks of an entire first world nation's internet. What will happen when Google decides to do something on purpose? Will entire hosting companies just disappear? Maybe Cloudfare will wake up one morning, and discover that Google has decided to take them off the 'Net for servicing Nazi websites.

  12. Was this Espresso? by TheSync · · Score: 1

    Google has been touting how Espresso "moves BGP to a custom stack running on servers, enabling finer-grained partitioning and much more computation power than available in any Internet router." Theoretically providing real-time changes in edge routing to maximize video streaming performance based on client metrics. Unless there is a bug...

  13. Actual affected users by Anonymous Coward · · Score: 0

    The actual affected users within Japan were mostly NTT OCN customers.

    So nothing of actual value was lost...