Slashdot Mirror
Why Google Went Offline Today
New submitter mc10 points out a post on the CloudFlare blog about the circumstances behind Google's services being inaccessible for a brief time earlier today. Quoting:
"To understand what went wrong you need to understand a bit about how networking on the Internet works. The Internet is a collection of networks, known as "Autonomous Systems" (AS). Each network has a unique number to identify it known as AS number. CloudFlare's AS number is 13335, Google's is 15169. The networks are connected together by what is known as Border Gateway Protocol (BGP). BGP is the glue of the Internet — announcing what IP addresses belong to each network and establishing the routes from one AS to another. An Internet "route" is exactly what it sounds like: a path from the IP address on one AS to an IP address on another AS. ... Unfortunately, if a network starts to send out an announcement of a particular IP address or network behind it, when in fact it is not, if that network is trusted by its upstreams and peers then packets can end up misrouted. That is what was happening here.
I looked at the BGP Routes for a Google IP Address. The route traversed Moratel (23947), an Indonesian ISP. Given that I'm looking at the routing from California and Google is operating Data Centre's not far from our office, packets should never be routed via Indonesia."
And I thought the internet was a series of tubes...
... Network Admins who have no clue. Like when just 4 years ago, Pakistan took down Youtube...
http://securitywatch.pcmag.com/dns/285152-pakistan-takes-youtube-down
Clearly this should be on the agenda for the new "Cyber Reserves" of the department of Homeland Security. If Google can be taken down by accident in parts of the world, then it certainly can be taken down on purpose. Route filters are your friends!
CYBER RESERVES: http://www.techradar.com/news/internet/department-of-homeland-security-recruiting-for-cyber-reserve-1109906
No trees were killed in the making of this post; however, many trillions of electrons were horribly inconvenienced.
Another networking issue that is probably never going to go away, I'm just surprised it isn't used more maliciously than it is. - HEX
Horror & SciFi Erotic Nudes
Nope. DNS doesn't mean shit if the routers are sending your traffic to the wrong place. (DNS points to an IP, which is (supposed to) point to the target machine. If that last part isn't working, the first part won't work no matter what)
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
From TFA:
Someone at Moratel likely "fat fingered" an Internet route. PCCW, who was Moratel's upstream provider, trusted the routes Moratel was sending to them. And, quickly, the bad routes spread.
Yes, someone at Moratel screwed up, but this is exactly why upstream ISPs should never allow advertisements from their customers for networks that their customer does not control.
PCCW is to blame for allowing this to happen. Never trust customers with things that don't belong to them.
Grammer Nazis - I mod you "troll" unless you actually add something on-topic. Yes, I know I have mispellings in my sig.
This sort of 'feature' did allow me once to escape from a misbehaving ISP holding me hostage and preventing me getting my mail to, for example, change my DNS glue records many many years ago. A helpful friendly new ISP managed to reroute traffic to me via them with a "bogus" routing announcement long enough for me to fix those records and then escape the old ISP when the new records propagated.
Rgds
Damon
http://m.earth.org.uk/
Seriously, a porn link in your sig?
Anyway... clearly Anonymous hasn't learned how to delete BGP filters and inject fake routes yet.
No trees were killed in the making of this post; however, many trillions of electrons were horribly inconvenienced.
China Telecom also hijacked web traffic to US government websites in April 2010 for 17 minutes. At least that incident seems to have been a purposeful disruptions to capture sensitive data and/or try out a novel cyberwarfare tactic.
Errr, yeah, what about that porn link? That's really... that's awful. I can't believe that they would have that there. Man, porn. Anyway, I've just got to go and do... a thing. Nothing interesting, don't you worry about it, just... Go about your business.
And this is why encryption is your friend, because you can't be sure whose networks your packets are going to travel through...
On each protocol, not just DNS lookups - eg HTTPS, SSH etc.
Since when does erotic nudes immediately equal "porn", and clearly you haven't visited the site.
The cyberweapon that could take down the internet
http://www.newscientist.com/article/dn20113-the-cyberweapon-that-could-take-down-the-internet.html
More on BGP Attacks â" Updated
http://www.wired.com/threatlevel/2008/08/how-to-intercep/
http://dankaminsky.com/2008/08/27/the-emergence-of-a-theme/
"Flyin' in just a sweet place,
Never been known to fail..."
Your Mac and iOS devices would also not be able to reach google in this scenario. Maybe that's what he meant....but really, probably just retarded.
Do the editors even read the submissions?
I get the feeling that upstreams should start to not completely trust BGP announcements from peers. I know in my firewalls the configuration knows which networks ought to appear where, and the rules are set to block traffic when that network shouldn't be able to appear on that interface. Perhaps it's time to look into having an administrative communication of which ASes each peer ought to be handling, and having the BGP system at the upstream filter out or ignore announcements for ASes that that peer isn't supposed to be handling. The problem I see with that though is that it works well at the edges, but the closer to the core you get the larger the list of potentially valid ASes and I can see it getting unmanageable pretty quickly. But with the number of these incidents, I think we need to do something to change the assumption that you can unconditionally trust peers to only hand you valid routing data, because that assumption pretty clearly isn't true anymore.
A BGP attack matched with a cert. leak could yield you on a spoof site, and you would provide your credentials to that man in the middle whichever device you are using. Except if you are using google chrome, where you might understand with the pinned certificate error that something is not correct, or with an android device that you were so eager to dismiss...
Here's an iOS virus for you http://www.ibtimes.com/apple-ios-app-store-gets-first-virus-learn-about-app-steals-your-contacts-and-spams-your-friends
As long as those looking to fix the problem don't start by Googling the problem..
That's a Trojan not a virus.
Seriously, a porn link in your sig?
Anyway... clearly Anonymous hasn't learned how to delete BGP filters and inject fake routes yet.
The only reason you replied was to bookmark!
---
ECHELON is a government program to find words like bomb, jihad, plutonium, assassinate, and anarchy.
It was not really the language that concerned him, but probably the lack of being on-topic.
I don't know what says more about the change in the average Slashdot reader--the fact that the summary for this story assumes that the reader doesn't know anything at all about BGP, or the fact that this is the first comment to bemoan that.
We don't do Porn, we try to keep on the erotic art side of things, and thanks for drawing attention to it lots of visitors from your mention! - HEX
Horror & SciFi Erotic Nudes
Well, I'm sure a scheme could be devised, assuming one could reliably trust that they can get to an authoritative source.
The question is really how far into the core can we move such security measures before it implodes. Core routers have to carry the entire routing table for every subnetwork advertised with external BGP in the entire world, and then worry about doing the same for IPv6 as it slowly kicks into gear. They are always in need of an upgrade, even right after an upgrade and sometimes even before any such upgrade exists. Moreover they can legitimately hear advertisements for the same network through multiple interfaces -- and we are not talking multiple as in three, but tens to hundreds given how well meshed things are.
Currently it is jelly-doughnut security: the carriers all just trust each other to not trust the end sites. After establishing a customer relationship, they filter the end-sites, only allowing in the ASN's that the customers have ownership rights to. Once that perimeter is breached it's up to ISPs to react to individual incursions of rogue advertisements after the fact.
Someone had to do it.
...to be fair, you *are* using a TLD explicitly intended for porn.
LegendMUD
The Google logo got caught with its hand in the ballot box cookie jar! It's all over Google's front page!
Oh, really? I thought Route Origin Authorisations were designed to address exactly this issue?
Do you care about the security of your wireless mouse?
No, but I think Route Origin Authorisations can help.
Do you care about the security of your wireless mouse?
And an historical note to what X0563511 said: back in the day, one had go to a Scientologist-ridden place (SRI) to obtain the host address from the IP numbers --- these SRI clowns actually referred to themselves as "physicists" and "scientists" not whackjob scientologists with their weird-assed orgone boxes, or whatever they called those string and tube crap.
Google keeps having error messages and random reloads on Gmail, Adsense, but not in Adwords at least. Their websites are dependent on JavaScript and the scripts can not cope with load errors, so they keep reloading the page until the servers are overloaded. I suspect that they are not aware of this too, becasue I have seen it last month already and they did nothing. I suspect that they run buggy code.
~ Best man at your service.
It's just that the link is right above the "Reply to This [comment]" link. Easy to click by accident. Somebody slashdoting from work may get in trouble or worse. And as for some other comment in this thread, no obviously I haven't clicked on it... I'm at work. Wait let me fix that... I'm at "work". Ok.
No trees were killed in the making of this post; however, many trillions of electrons were horribly inconvenienced.
Keep it in mind. Internet security is done end to end. The old telephony networs are dead, and even them weren't really reliable.
If you have sensitive data to transmit, you shouldn't even think about where it will go. Wherever it is, it is not a safe place, and your data must be protected.
Rethinking email
No wonder no one replies to me, they are all distracted by my link and visit my site instead! A bittersweet win-win situation! As for visiting from work, no one sets their alert threshold that low for even adult material, sure you might get a blocked page but you won't get HR on your back for it. Of course my area and level of IT we're exempt from filtering usually, too many good resource sites get erroneously filtered. (and we implement the filters lol) - HEX
Horror & SciFi Erotic Nudes
If I'm properly filtering at the border, I don't need to filter in the middle
You absolutely have to filter when crossing an international border. National security requires it. Maliciousness can be of a military nature, and you'd better be expecting it. The network admin on the other side may be coerced, an eager participant, or unaware. You ever can't trust what he does or what he says.
If BGP abuse lets China detect a previously-unknown site that communicates with a known US spy agency, China has learned something valuable.
Almost all BGP capable equipment at most exchanges is now able to filter the amount of address blocks each ISP can announce. Once someone starts announcing a whole lot more than the filter is set for, the announcements are ignored and alerts are triggered.
While that mitigates problems, the actual solution is already being put in place. IP address blocks are being assigned to parties and those parties can sign routing announcements for those IP blocks using a PKI system. By having the BGP equipment check each request with the public key of the published "owner" of the block, rogue announcements should be ignored. Not all equipment is capable of this and not all exchanges have made this mandatory, but this will most likely happen in the future. Sure, by stealing keys, finding weaknesses in the implementation of router vendors and such, attacks will still be possible, but admins making mistakes will hopefully not mess up things anymore.
This works perfectly for end points in routes, but I am not certain how routes through someone's AS to another AS are being dealt with. I assume you can tag certain ASes as "transit AS" and accept unsigned routes from them. That would make you still vulnerable for rogue announcements through those ASes, but only if those providers didn't use signed announcements and filters on how many netblocks a peer could announce.
I was promised a flying car. Where is my flying car?
UCLA's Cyclops is a great tool to monitor your own IP space and make sure you know immediately when this sort of this occurs.
Why the hell not? I thought the reason for the internet was to provide a way for data packages to get to their destination without having to have a particular fixed route. Does it says somewhere in the internet standard that data packets *must* be delivered using the shortest route possible?
Here's a summary of the summary: someone broke the internet. That's right, "the internet is broken" actually applies here, lol.
Is that you?
No, not like BOFH in that I never intentionally F-ed anyone up for fun and always operated in everyone's best interest, best as I could. Telecom is a tiny world and I'll be coworkers with "competitors" a hell of a lot longer than I'll be working for any individual employer. Yes was like BOFH in that I was easily capable of anything I wanted to do some of which occasionally got a little "Apocalypse Now heads on stakes", was grouchy almost all of the time, and finally got pretty sick of it. Overall glad I don't do that anymore. Most of my individual memories of it are highly negative, although at the time it was overall kinda fun, donno if that makes any sense. It was fun when I didn't have to shout at people, unfortunately most of my job productivity was shouting at people, is I guess how I'd phrase it. Don't miss rotating pager duty at all.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger