Google Suffered a Brief Outage on Monday Which Pushed Some of Its Traffic Through Russia, China and Nigeria; Company Says It Will Do an Investigation (cnet.com)

← Back to Stories (view on slashdot.org)

Google Suffered a Brief Outage on Monday Which Pushed Some of Its Traffic Through Russia, China and Nigeria; Company Says It Will Do an Investigation (cnet.com)

Posted by msmash on Tuesday November 13, 2018 @03:20AM from the stranger-things dept.

Google suffered a brief outage and slowdown Monday, with some of its traffic getting rerouted through networks in Russia, China and Nigeria. From a report: Incorrect routing instructions sent some of the search giant's traffic to Russian network operator TransTelekom, China Telecom (which, as you may recall, has been found of misdirecting internet traffic in recent months) and Nigerian provider MainOne between 1:00 p.m. and 2:23 p.m. PT, according to internet research group ThousandEyes. "This incident at a minimum caused a massive denial of service to G Suite and Google Search," wrote Ameet Naik, ThousandEyes' technical marketing manager, in a blog post. "However, this also put valuable Google traffic in the hands of ISPs in countries with a long history of Internet surveillance. Applications like Gmail and Google Drive don't appear to have been affected, but YouTube users experienced some slowdown. Google noted that the issue was resolved and said it would conduct an internal investigation. Update: Nigeria's Main One Cable Co has taken responsibility for the glitch.

39 of 70 comments (clear)

Min score:

Reason:

Sort:

Re:Just another reason by neilo_1701D · 2018-11-13 03:33 · Score: 2

This is just another in a long string of reasons to slowly back away from google.
Agreed... but to where? Bing search results have become dramatically worse in recent weeks. Yahoo... well, it uses Bing, so say no more. Duck Duck Go? Last time I used it, the results were all over the place.
Cloud Services by StormReaver · 2018-11-13 03:34 · Score: 1

But sure, by all means, put your important information on someone else's servers you have no control over. What could possibly go wrong? Oh, right, all of your important information could be shunted off to your competitors. But that's not a big deal, right?
This is an I.Q. test masquerading as a technical issue.
1. Re:Cloud Services by bill_mcgonigle · 2018-11-13 03:45 · Score: 4, Insightful
  
  Oh, right, all of your important information could be shunted off to your competitors. But that's not a big deal, right?
  Look, I help people set up private servers to keep their data out of "the Cloud" but you can't be wrong about the arguments.
  Event IF this were a BGP hijack rather than a misconfiguration error and even IF they had minted Google.com certs trusted by the default root stores, Chrome would have picked up the pinned-certificate fingerprint mismatches and refused to connect. Everything in Google's suite happens over TLS.
  Yes, this would cause an outage, which costs time and money, but your information does not wind up in the hands of your competitors.
  Make technically valid business arguments - don't spout crazy conspiracy theories.
  
  --
  My God, it's Full of Source!
  OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
2. Re:Cloud Services by StormReaver · 2018-11-13 04:49 · Score: 1
  
  Event IF this were a BGP hijack rather than a misconfiguration error and even IF they had minted Google.com certs trusted by the default root stores...
  And IF Cloud Computer equaled Google, you would have at least a semi-reasonable argument. But this ISN'T just about Google, and you're definitely missing the bigger picture. This is about the very nature of trusting some untrustworthy third party data sieve (be it Google, Microsoft, Amazon, etc.) to go against its own nature and keep your secrets secret. This particular failure is simply an indicator of what HAS happened, what IS happening, and what WILL happen to people stupid enough to trust their data to, "Theeee Clouuuud" (use your Toy Story green alien chorus voice).
  
  Yes, this would cause an outage, which costs time and money, but your information does not wind up in the hands of your competitors.
  You have just failed the I.Q. test.
  Cloud services can fail in a nearly infinite number of ways. Just look at how easy Amazon made it to broadcast secrets to the world. ALL major cloud providers have weak security. One failure could result in publishing names, addresses, and social security numbers to the world (which has happened more than once); and the next failure could result in publishing the latest jet fighter specifications to Russia and China. This failure mode is unique to Cloud computing.
  That you think this is conspiracy theory territory should ban you from ever managing sensitive data, as you have demonstrated an ASTOUNDING lack of common sense. Those damn trees are severely limiting your view of the forest.
3. Re:Cloud Services by bill_mcgonigle · 2018-11-13 11:09 · Score: 1
  
  You seriously can't discuss the different risks of cloud computing separately? That's why you sound like a conspiracy theorist and not an engineer.
  Of course Google is putting your data at risk - that's not relevant to a BGP hijack - it's a separate issue.
  Of course Google's owning Chrome is a special case - that's the case we're discussing here. To try to generalize it to a huge catastrophe is just soapboxing and not useful. Everybody here already knows about those risks - this isn't USA Today.
  
  --
  My God, it's Full of Source!
  OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
This affected me in Canada by Anonymous Coward · 2018-11-13 03:39 · Score: 1

All my smart home devices stopped working for up to an hour on Sunday. I got a panicked phone call from my grandma who couldn't turn off her lights.
I setup my devices on a restricted wifi network because of this kind of stuff. I don't have access to the device to see what it is connecting to, and now we find out it was also routed through potentially malicious nations.
1. Re:This affected me in Canada by Fly+Swatter · 2018-11-13 04:04 · Score: 4, Insightful
  
  There is a lesson to be learned in here somewhere, both for you and for grandma.
Proof by geekymachoman · 2018-11-13 03:39 · Score: 2

Finally proof those pesky Russians are hacking the America and it's freedom.
Chinese ? Well... they all the same kind.
As if we here do not do the same... by bogaboga · 2018-11-13 03:39 · Score: 2, Insightful

...However, this also put valuable Google traffic in the hands of ISPs in countries with a long history of Internet surveillance...
The subtext here is that the USA does not [*cough*] [*cough*], have government funded agencies doing the same. The other day, some government agency was found to be spying on Americans, even when congress [limited] its ability to.
So the summary should have been phrased this way:
..."However, this also put valuable Google traffic in the hands of ISPs and agencies in countries which like the USA, have a long history of Internet surveillance. (Bold mine.).
1. Re:As if we here do not do the same... by theurge14 · 2018-11-13 04:27 · Score: 2, Insightful
  
  Kindly point out any articles about a US government agency hijacking BGP routes.
  Otherwise, save the whataboutism thanks.
2. Re:As if we here do not do the same... by spacepimp · 2018-11-13 05:19 · Score: 1
  
  No one is claiming we don't do this.... In this case the NSA seems like it would enjoy the ability to capture this data in whole. There is no incentive for them to stop this from occurring because now they can legally siphon it up as foreign data and spy on US citizens with hat they claim is less Constitutional rights abusive. In fact there is no evidence that this isn't being encouraged.
3. Re:As if we here do not do the same... by Anonymous Coward · 2018-11-13 06:18 · Score: 2, Insightful
  
  See telco tap points articles of many years ago.
  US government doesn't have to play stupid BGP tricks because the traffic is already traversing links they have 100% access to.
What are you on about by drinkypoo · 2018-11-13 03:51 · Score: 4, Insightful

This is just another in a long string of reasons to slowly back away from google.
If you think this is a google-only problem, you should have your posting rights taken away immediately. This isn't just happening to Google, it's happening to just about everyone. If your traffic isn't encrypted, then this is a great reason to slowly back away from you.

--
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
1. Re:What are you on about by BringsApples · 2018-11-13 06:07 · Score: 1
  
  Google's traffic is encrypted, but what does that matter if it's routed incorrectly? AFAIK, encryption is only a means to detour an attack. However if the stream of data is captured, it can be hacked at until the encryption is solved.
  
  Besides the obvious implications involved with the above, there is also the possibility (get your conspiracy-theorist-bashing-vocabulary prepared) that these things (re-routed to the bad-guy countries) happen by design, in order to blame some out-of-reach entity for whatever shit-storm of fuckery happens next regarding, god only knows what.
  
  --
  Politics; n. : A religion whereby man is god.
2. Re:What are you on about by Known+Nutter · 2018-11-13 07:05 · Score: 1
  
  How long do you believe it takes to crack a proper implementation of AES-256?
  
  --
  Beware of the Leopard.
3. Re:What are you on about by BringsApples · 2018-11-13 11:01 · Score: 1
  
  Dunno, but certainly finite.
  
  --
  Politics; n. : A religion whereby man is god.
4. Re:What are you on about by steveb3210 · 2018-11-13 13:00 · Score: 1
  
  Intractably finite...
5. Re:What are you on about by BringsApples · 2018-11-14 02:57 · Score: 1
  
  To be fair...
  
  Fifty supercomputers that could check a billion billion (1018) AES keys per second (if such a device could ever be made) would, in theory, require about 3×10-to-the-power-of-51 years to exhaust the 256-bit key space.
  The thing to point out there is that exhausting the entire 256-bit key space is one way to skin a cat.
  
  --
  Politics; n. : A religion whereby man is god.
Surely all encrypted? by SmilingBoy · 2018-11-13 04:10 · Score: 3, Funny

All traffic between browser and Google is encrypted. I don't see a real security risk here.
Re:COULD HAVE BEEN A 600LB GUY ON HIS COUCH! by Tablizer · 2018-11-13 04:11 · Score: 2

He described himself, actually.
Anyhow, it seems too easy to blame Nigeria. Most "Nigerian Princes" were not sending email from Nigeria even. They've been unfairly memeitized. Or is it "memeificated"?

--
Table-ized A.I.
pot calling kettle...kettle come in by nimbius · 2018-11-13 04:13 · Score: 4, Informative

in the hands of ISPs in countries with a long history of Internet surveillance.
https://en.wikipedia.org/wiki/...
This might be news to some, but the US is a country with a long history of internet surveillance. AT&T maintains an entire room in one of their san francisco datacenters that does nothing but explicitly snoop traffic for the government.

--
Good people go to bed earlier.
1. Re:pot calling kettle...kettle come in by drinkypoo · 2018-11-13 04:37 · Score: 4, Informative
  
  AT&T maintains an entire room in one of their san francisco datacenters that does nothing but explicitly snoop traffic for the government.
  In fact, only one telecom CEO in America refused to install the feds' equipment where it could be used to spy on Americans. Never Forget.
  
  --
  "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
2. Re:pot calling kettle...kettle come in by fustakrakich · 2018-11-13 05:01 · Score: 1
  
  In fact, only one telecom CEO in America refused to install the feds' equipment where it could be used to spy on Americans.
  Independently verified? Or are we supposed to take them at their word?
  
  --
  “He’s not deformed, he’s just drunk!”
3. Re:pot calling kettle...kettle come in by spacepimp · 2018-11-13 05:22 · Score: 1
  
  I believe you are incorrect. the room that you are discussing is one of many (there are similar ones in NYC) and the intent is to duplicate all data (PRISM) into a separate pipeline for "collection" in bulk.
4. Re:pot calling kettle...kettle come in by spacepimp · 2018-11-13 05:53 · Score: 1
  
  it is well documented. His legal court case was quite revealing.
Just as planned by thunderclees · 2018-11-13 04:22 · Score: 2

Admittedly a large piece of supposition but how hard would it really be for the PRC and the rest of the usual suspects to convince Google and others to have accidents like this. Google, really, really wants to make RB off of Chinese users and so what if the CPC wanted data in return?
Re:Just another reason by alvinrod · 2018-11-13 04:23 · Score: 2

I've started to wonder how useful search engines are anymore since it seems like I invariably end up in the same place 95% of the time. What I probably want most of the time for general knowledge is the Wikipedia article, if it's programming related it's likely to be on Stack Overflow, and for video content I may as well just go straight to YouTube (technically Google anyways, but that's a different argument), and there are a small number of other websites that I frequently end up on after searching for something. It's probably easier to just go to those directly in most cases, but to be fair some of them have awful search (or an utterly fucked site layout with shit navigation) for their own sites so it's just easier to use Google to get to where I want within the site. May as well just buy stuff on Amazon, since the extra hour I might spend searching around to save $.50 just isn't worth my time.

If I'm in a new city and I want to know where to go to eat, it's probably better to just ask a local. Online reviews aren't always that great, and I've eaten at plenty of great little hole in the wall joints that only have one or two reviews total. I think the locals have learned not to review the really good places to keep the tourists away from them. Same goes for other services, where I trust word of mouth or the opinion of someone I know over online reviews.

I still use web search, but I often find myself using queries pretty much designed to give me the Wikipedia page I want instead of just going to Wikipedia to start with. If all web search went down for a week, I might be slightly inconvenienced, but I don't think that my life would be much worse off.
Re: Just another reason by reanjr · 2018-11-13 04:25 · Score: 1

Search engines are super convenient for web developers. Much rather rely on Google to handle that for me than write my own (shitty) site search.
Re:Democrats shouldn't have any problem with this by Anonymous Coward · 2018-11-13 04:39 · Score: 2, Informative

Remind me, which president passed the patriot act? And which president wanted to "close parts of the Internet"? If you think this is a Democrat thing then you really haven't been paying attention.
Re:Just another reason by fustakrakich · 2018-11-13 04:43 · Score: 2

but to where?
Yacy. If more people use it, it can only get better. And it's very resistant to censorship.

--
“He’s not deformed, he’s just drunk!”
Re:Perplexing by hierofalcon · 2018-11-13 04:54 · Score: 1

Bandwidth problems would have made the problem obvious much too quickly.
Poor headline by Dan+East · 2018-11-13 05:20 · Score: 5, Informative

The headline makes it sound like Google had a brief outage and that caused some traffic to be routed through Russia and China. What actually happened is Some Google Traffic Routed Through Russia and China Causing Brief Outage.
But since we're all used to awful headlines here at Slashdot, and we know we can't expect much better from the original source cnet, that's perfectly fine.

--
Better known as 318230.
Re:Just Fix It by sabri · 2018-11-13 05:49 · Score: 1

It is time to replace BGP
Surely, we can use EIGRP for the interwebs.

You, sir, need to go see the network doctor for a good paddling with the cluebat.

--
I'm not a complete idiot... Some parts are missing.
difference is Google catches it, other sites don't by raymorris · 2018-11-13 05:58 · Score: 1

The difference being that other sites don't KNOW when their traffic is being routed through Russia. Google knows, and with certificate pinning and other safeguards, the site just ends up being down rather than having surveillance actually work like it does for other sites.
Effective but not elegant by eastjesus · 2018-11-13 06:15 · Score: 3, Informative

I noticed Google down just as it started and when I checked I found that Spectrum (which still uses rr.com for naming) was sending all Google bound traffic to Tata communications (an Indian Company) which sent it over to Europe on its circuits then Transtelecom in South Africa,which moved it to Chinanet. Traceroute excerpt: 10 0.ae2.pr1.dfw10.tbone.rr.com (107.14.17.236) 66.274 ms 0.ae0.pr1.dfw10.tbone.rr.com (107.14.17.232) 68.537 ms 0.ae4.pr1.dfw10.tbone.rr.com (107.14.19.97) 69.705 ms 11 ix-ae-23-0.tcore2.dt8-dallas.as6453.net (66.110.57.97) 70.130 ms 71.137 ms 70.498 ms 12 if-ae-2-2.tcore1.dt8-dallas.as6453.net (66.110.56.5) 205.871 ms 205.041 ms 207.009 ms 13 if-ae-37-3.tcore1.aeq-ashburn.as6453.net (66.198.154.68) 208.978 ms 207.757 ms 212.871 ms 14 if-ae-2-2.tcore2.aeq-ashburn.as6453.net (216.6.87.1) 211.628 ms 212.403 ms 241.799 ms 15 if-ae-12-2.tcore4.njy-newark.as6453.net (216.6.87.43) 203.197 ms 204.385 ms if-ae-12-2.tcore4.njy-newark.as6453.net (216.6.87.223) 238.450 ms 16 if-ae-1-3.tcore3.njy-newark.as6453.net (216.6.57.5) 234.408 ms 235.627 ms 235.190 ms 17 if-ae-15-2.tcore1.l78-london.as6453.net (80.231.130.25) 239.527 ms 239.084 ms 240.261 ms 18 if-ae-2-2.tcore2.l78-london.as6453.net (80.231.131.1) 240.647 ms 241.425 ms 241.816 ms 19 if-ae-14-2.tcore2.av2-amsterdam.as6453.net (80.231.131.161) 246.783 ms 247.567 ms 246.319 ms 20 if-ae-2-2.tcore1.av2-amsterdam.as6453.net (195.219.194.5) 248.282 ms 167.135 ms 192.261 ms 21 if-ae-6-2.tcore1.fnm-frankfurt.as6453.net (195.219.194.150) 193.772 ms 197.050 ms 200.104 ms 22 195.219.156.146 (195.219.156.146) 213.840 ms 213.268 ms 219.112 ms 23 mskn17ra-lo1.transtelecom.net (217.150.55.21) 271.186 ms 266.862 ms 267.265 ms 24 * * ChinaTelecom-gw.transtelecom.net (217.150.59.249) 280.990 ms 25 * * * 26 * * * 27 * * * 28 * * 154.72.45.166 (154.72.45.166) 466.625 ms There was a period in the middle of that time that Google appeared to be working but traceroute showed everything passing through chinanet and then on to Google, just long latency, but they couldn't keep it up and Google kept going down. There is another article about it at: https://www.dailymail.co.uk/ne... Sorry about the formatting but the /. editor is not accepting my line breaks. Figured the traceroute might be interesting to some even if it looks ugly.
1. Re:Effective but not elegant by Anonymous Coward · 2018-11-13 07:17 · Score: 1
  
  I noticed Google down just as it started and when I checked I found that Spectrum (which still uses rr.com for naming) was sending all Google bound traffic to Tata communications (an Indian Company) which sent it over to Europe on its circuits then Transtelecom in South Africa,which moved it to Chinanet.
  Traceroute excerpt: 100.ae2.pr1.dfw10.tbone.rr.com (107.14.17.236) 66.274 ms 0.ae0.pr1.dfw10.tbone.rr.com (107.14.17.232) 68.537 ms 0.ae4.pr1.dfw10.tbone.rr.com (107.14.19.97) 69.705 ms 11 ix-ae-23-0.tcore2.dt8-dallas.as6453.net (66.110.57.97) 70.130 ms 71.137 ms 70.498 ms 12 if-ae-2-2.tcore1.dt8-dallas.as6453.net (66.110.56.5) 205.871 ms 205.041 ms 207.009 ms 13 if-ae-37-3.tcore1.aeq-ashburn.as6453.net (66.198.154.68) 208.978 ms 207.757 ms 212.871 ms 14 if-ae-2-2.tcore2.aeq-ashburn.as6453.net (216.6.87.1) 211.628 ms 212.403 ms 241.799 ms 15 if-ae-12-2.tcore4.njy-newark.as6453.net (216.6.87.43) 203.197 ms 204.385 ms if-ae-12-2.tcore4.njy-newark.as6453.net (216.6.87.223) 238.450 ms 16 if-ae-1-3.tcore3.njy-newark.as6453.net (216.6.57.5) 234.408 ms 235.627 ms 235.190 ms 17 if-ae-15-2.tcore1.l78-london.as6453.net (80.231.130.25) 239.527 ms 239.084 ms 240.261 ms 18 if-ae-2-2.tcore2.l78-london.as6453.net (80.231.131.1) 240.647 ms 241.425 ms 241.816 ms 19 if-ae-14-2.tcore2.av2-amsterdam.as6453.net (80.231.131.161) 246.783 ms 247.567 ms 246.319 ms 20 if-ae-2-2.tcore1.av2-amsterdam.as6453.net (195.219.194.5) 248.282 ms 167.135 ms 192.261 ms 21 if-ae-6-2.tcore1.fnm-frankfurt.as6453.net (195.219.194.150) 193.772 ms 197.050 ms 200.104 ms 22 195.219.156.146 (195.219.156.146) 213.840 ms 213.268 ms 219.112 ms 23 mskn17ra-lo1.transtelecom.net (217.150.55.21) 271.186 ms 266.862 ms 267.265 ms 24 * * ChinaTelecom-gw.transtelecom.net (217.150.59.249) 280.990 ms 25 * * * 26 * * * 27 * * * 28 * * 154.72.45.166 (154.72.45.166) 466.625 ms
  There was a period in the middle of that time that Google appeared to be working but traceroute showed everything passing through chinanet and then on to Google, just long latency, but they couldn't keep it up and Google kept going down. There is another article about it at: https://www.dailymail.co.uk/ne... [dailymail.co.uk] Sorry about the formatting but the /. editor is not accepting my line breaks. Figured the traceroute might be interesting to some even if it looks ugly.
  FTFY. I guess slashcode doesn't like large blocks of text with loads of carriage returns. Prevents trolls?
too coincidental by AndrewFlagg · 2018-11-13 06:25 · Score: 1

ironically, its a wild wild west our here. i wonder where exactly did the breakdown occur? just reading up on BGP ASNs security and then again that closet in a San Francisco megapop (aka Google) that has no label, no door, no lock, no key -- the black box of sorts. good stuff for Friday read.
The Not-So-Secret Snoop Room by eastjesus · 2018-11-13 09:20 · Score: 1

There have been a number of comments here about the "secret" secure room in San Francisco where Internet traffic is snooped. When General Alexander was head of the NSA (where he built a replica of the Star Trek bridge with taxpayer money for his commmand) he issued the directive to "Collect it all!"

The "room" was in AT&T's facility, not Google's, and tapped a major Internet backbone link. It's been known and documented for years. See the deposition of Mark Klein dated June 8, 2006, formerly of AT&T (class action suit led by EFF vs AT&T: C-06-0672-VRW in US District Court, Northern District of California). He describes the sequence of events, persons, locations, equipment, and details involved in installing the tap of all Internet traffic passing through the fiber lines at the AT&T location at 611 Folsom St., San Francisco, and sending them to room 641A, designated a secret locked "secure room." More technical details, diagrams, and photos here but note that all this info is VERY OLD - 13 years old!

All the major telcos participated in the program back then except Qwest which (as noted by drinkypoo above directly resulted in the destruction of that company (they lost major government contracts) and 6 years in prison for its CEO Joseph Nacchio (for relying on those contracts). Nacchio was finally released from prison in 2013 after serving the full 6 years. The telcos were later granted immunity by congress and other cases have been quashed on grounds of lack of standing because evidence would impinge on national security. Perhaps it seemed all too technical and abstract at the time for people to pay attention but the news has been out there for a long time. We can only imagine where things stand today.
Re:Perplexing by godel_56 · 2018-11-13 10:22 · Score: 1

North Korea doesn't do mail order brides.
I'm sure they would if there was enough money in it. They already do large scale drug dealing and money counterfeiting.