Slashdot Mirror

← Back to Stories (view on slashdot.org)

Chinese Root Server Shut Down After DNS Problem

Posted by timothy on from the need-a-new-source-of-ginseng dept.

itwbennett writes "After a networking error first reported on Wednesday last week caused computers in Chile and the US to come under the control of a system that censors the Internet in China, the 'root DNS server associated with the networking problems has been disconnected from the Internet,' writes Robert McMillan. The server's operator, Netnod, has 'withdrawn route announcements' made by the server, according to company CEO Kurt Lindqvist."

56 of 91 comments (clear)

  1. The great firewall of China by FishTankX · · Score: 1

    For a moment, it stretched around the world. Or, atleast to the Americas.

  2. Even more reason by Finallyjoined!!! · · Score: 1

    To fully implement dnssec.

    --
    If I had an Ass, I'd call it Fanny Bottom, then I could slap my Ass; Fanny Bottom, on the Arse.
    1. Re:Even more reason by rvw · · Score: 1

      Can somebody explain what this all means? What does this root server do, who depends on this, what is the effect of disconnecting it, how will the rest of the world be affected by this?

    2. Re:Even more reason by erroneus · · Score: 1

      I would. But I just finished watching an old Mitch Hedberg special. Now, everything I read, is in, the voice, of, Mitch Hedberg. Damn. Him.

    3. Re:Even more reason by wrencherd · · Score: 1

      I think if you concentrate very hard you could easily substitute Roy Mallard, for higher entertainment value.

    4. Re:Even more reason by DeadChobi · · Score: 1

      I have a lower UID than you and I don't know what a root DNS server does. I do probably know way more physics, mathematics, and philosophy than you so can it. Especially if you're not going to explain.

      --
      SRSLY.
    5. Re:Even more reason by PhrstBrn · · Score: 4, Informative

      One small correction:

      When you ask the root servers (such as a.root-servers.net) for "what is IP for www.google.com", it will respond "go ask a.gtld-servers.net". (each domain has a different server, for instance www.google.co.uk will send you to ns1.nic.uk). Asking a.gtld-servers.net will respond "go ask ns1.google.com", which will then respond with the IP of the domain, which is your answer. The chain could go further if you had "some.very.long.string.of.dots.google.com" and if each one of those nested subdomains were delegated to another DNS server (and were not contained in the zone file for "google.com").

      If the answer is already cached by the DNS server and it is still within the TTL, it will just respond with the IP.

      This is how a DNS caching resolver does it, your workstation is going to be configured with one of these caching resolvers. When you ask a caching resolver, it will do all these things in the background on these server, and just return the client the final answer

    6. Re:Even more reason by gbutler69 · · Score: 1

      Next, your local DNS server (the one your computer asked) recursively asks ns1.google.com for the IP number of "www.google.com".

      Please don't misuse "recursively" like this. It makes the rest of your otherwise intelligent post sound stupid. I think you meant "iteratively" rather than "recursively". A recursive DNS quey goes like this: 1) You ask your local DNS, 2) Your local DNS asks another root (or possibly non-root) DNS, 3) the other DNS asks another 4) the "another" asks "another" 5) Finally, somewhere in the chain, it returns an answer "recursively" through the chain of requests from DNS server to another to you. Normally, DNS looku-up works the way you describe, "iteratively". The "recursive" way is special and requires a DNS server to be configured for "recursive resolution".

      --
      Over-the-top Response Guy! Giving "Over-the-Top Responses" since 1970.
    7. Re:Even more reason by Finallyjoined!!! · · Score: 1

      I have a higher UID than you, though I do know what a root DNS server is. I also bet that I was using a computer before your Father kissed your Mother for the first time. I also probably know way more physics & maths than you. Philosophy is for girls, so you win on that one.

      Yes, the OP should have RTFA, but your point is what, precisely?

      --
      If I had an Ass, I'd call it Fanny Bottom, then I could slap my Ass; Fanny Bottom, on the Arse.
    8. Re:Even more reason by evilviper · · Score: 1

      I don't know what a root DNS server does. I do probably know way more physics, mathematics, and philosophy than you

      That would be wonderful if you were on a "Philosophy news" website... /. is (or at least used-to-be) fairly tolerant of noobs with gaps in their knowledge, but if you don't have a decent background in tech, I don't see why you're here.

      --
      Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
    9. Re:Even more reason by budgenator · · Score: 1

      A root server, serves the DNS querys for a global domain such as .com. how it works is when your computer asks for the addresses for slashdot.org, your ISP probably knows the address because someone else has asked, if not your ISP asks the next higher level which is more likely to know because the answer to more queries. Eventually it get to the root server if the intermediate steps fail. As the answering server gets farther up the longer it takes for you to get the answer. Each query answered has a TTL, time to live, to it so an answer might be good for 24 hours. When a root server is borked, it means that even when they fix it or use a different root server it can take hours or days for all the bad answers to clear out of the DNS system.

      --
      Apocalypse Cancelled, Sorry, No Ticket Refunds
    10. Re:Even more reason by dgatwood · · Score: 1

      The request your DNS server sends to "ns1.google.com" may be iterative relative to the request to the root server, but it is still recursive relative to the original request. Thus, my original statement was completely correct. You just misread it.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    11. Re:Even more reason by dgatwood · · Score: 1

      Right. Sorry, forgot that they stopped serving COM. That only changed a few years ago.

      *does Google search*

      Yikes. Ten years ago. I'm suddenly feeling very old. :-D

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    12. Re:Even more reason by dgatwood · · Score: 1

      Also, maybe in some mega ISPs or some insanely complex intranet environment, you might have a DNS server that queries something other than the root server, but I suspect you could count all such installations worldwide on one hand if you used base 2. In practice, the only servers that support recursion are client-facing servers at ISPs.

      And the root servers never recurse. They didn't even recurse back in the mid 1990s when I was first learning this stuff. As best I could determine, the last root server had its recursion turned off way back in 1995, five years before they moved COM to the GTLD fleet, which AFAIK have never supported recursion.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    13. Re:Even more reason by gbutler69 · · Score: 1

      I never said the root server did answer recursive queries. I just defined what it would be like if they did.

      --
      Over-the-top Response Guy! Giving "Over-the-Top Responses" since 1970.
  3. Google Fights Back by JackieBrown · · Score: 4, Funny

    It had to happen sooner or later...

    1. Re:Google Fights Back by SpzToid · · Score: 1

      So... "like a great many voices cried out in terror before being suddenly silenced."

      But who is Alderaan here, exactly? Isn't China supposed to be The Empire, that just wants its Order? I thought GOOG was the eViL global empire awhile ago but now the rebels control the Death Star? This all so very confusing.

      --
      You can't be ahead of the curve, if you're stuck in a loop.
    2. Re:Google Fights Back by NotBornYesterday · · Score: 1

      "doesn't want to export rare earths, but loves to export lead and melamine"

      Maybe it's time to try some reverse psychology. If we can somehow convince them that we need lead and melamine for our latest high-tech products, but would prefer they keep all that awful neodymium to themselves, I'm sure we can fix the imbalance.

      --
      I prefer rogues to imbeciles because they sometimes take a rest.
    3. Re:Google Fights Back by genner · · Score: 1

      So... "like a great many voices cried out in terror before being suddenly silenced."

      But who is Alderaan here, exactly? Isn't China supposed to be The Empire, that just wants its Order? I thought GOOG was the eViL global empire awhile ago but now the rebels control the Death Star? This all so very confusing.

      It confusing because you didn't make a car analogy.

    4. Re:Google Fights Back by SpzToid · · Score: 1

      Where is BadAnalogyGuy when you really need him?

      --
      You can't be ahead of the curve, if you're stuck in a loop.
  4. route announcements? by bl8n8r · · Score: 1

    So... the chinese DNS server was using BGP? Sorry, not much of a BIND geek. Is this a reference to the Anycast protocol?

    --
    boycott slashdot February 10th - 17th check out: altSlashdot.org
    1. Re:route announcements? by cjcela · · Score: 1

      From www.bgp4.as : The Border Gateway Protocol (BGP) is the routing protocol used to exchange routing information across the Internet. It makes it possible for ISPs to connect to each other and for end-users to connect to more than one ISP. BGP is the only protocol that is designed to deal with a network of the Internet's size, and the only protocol that can deal well with having multiple connections to unrelated routing domains.

    2. Re:route announcements? by pv2b · · Score: 5, Informative

      Here's a graph of the network structure as seen by BGP.

      AS29216 at the right is the AS which I.ROOT-SERVERS.NET is located in. As we can see, it is only reachable through AS8674 (NETNOD-IX).

      Which in turn is reachable directly from a few different AS:es, including AS24151 (CNNIC-CRITICAL-AP).

      My guess is that Netnod simply started filtering out the routes to AS29216 via AS8674 on the BGP session to AS24151.

      The DNS server itself might have been using BGP, it might not have. But in the end every system on the Internet is reachable with some kind of BGP route somewhere.

  5. Chinese tweets by vrmlguy · · Score: 1

    The artilce includes a sample of Twitter tweets, all in Chinese. Unfortunately, just entering the Twitter search URL into Google translator doesn't seem to work, as the "Realtime results for Netnod" (http://twitter.com/search?q=Netnod) are apparently served via JSON or something. Anyone got any ideas?

    --
    Nothing for 6-digit uids?
    1. Re:Chinese tweets by lobsterturd · · Score: 1

      They're in Japanese, and all they're really saying is a summary of the article.

    2. Re:Chinese tweets by bipbop · · Score: 1

      As of this moment, the Japanese tweets are after the "More" link, and all the tweets on the first page of results are Chinese.

  6. Heads should roll by bguiz · · Score: 1, Insightful

    Who knows, in the few days that the Great Firewall of China crossed the Pacific, the kind of damage that could have been done, or perhaps even already been done?

    This should never have been allowed to happen in the first place, and when it had, it shouldn't have been allowed to persist for a few days before being made public and taking action.

    1. Re:Heads should roll by Anonymous Coward · · Score: 1, Funny

      Chillax, it's a firewall, not a deathray.

      But it would be COOL if it were a death ray.

    2. Re:Heads should roll by mysticalreaper · · Score: 1

      Lookups for things like 'www.facebook.com' were returning false answers. Youtube.com and others were affected too.

      So if you got the bad answer from DNS (because you happend to query the Beijing root server), some of your favourite websites would be unreachable.

    3. Re:Heads should roll by mysticalreaper · · Score: 3, Insightful

      This should never have been allowed to happen in the first place, and when it had, it shouldn't have been allowed to persist for a few days before being made public and taking action.

      Well i think this unreasonably harsh. No one had ever seen the great firewall of china affect DNS traffic like this in the past. So no one (not even you) was suggesting that when they set up a root DNS server in Beijing, that it would effectively send out false answers.

      Now, anyone who controls a part of the network you rely on can launch a man-in-the-middle attack, which is what happened here. So to suggest that this should never have been allowed to happen, you would have to be using strong cryptography in some way. DNS has never had that mechanism--but it will soon, cause DNSSEC is coming along.The root servers are deploying it right now, and so are the other Top-level-domains.

      Also, as soon as the I-root server operators realized this problem was occurring, and was outside of their control, they disabled the server. Why do you think that they sat on this problem for a few days, doing nothing about it?

    4. Re:Heads should roll by jafiwam · · Score: 1

      The Chinese should simply be cut off from the internet.

      Anchor-drag their shit and pull up a couple hundred miles of fiber.

      Then keep doing it as they repair stuff.

      "Most favored" seems to be ineffective now days as far as holding their crap back. Maybe it's time to cut them off at their short little knees economically before their expansionist military catches up with their ability to make lead-laden rubber dog crap.

    5. Re:Heads should roll by Anonymous Coward · · Score: 1, Insightful

      I really don't understand where this china-hate is coming from. What did they ever do to you? Let's cut 1.3 billion people off the internet because someone IN ANOTHER COUNTRY WHO IS NOT CHINESE misconfigured a server. Yeah that makes total sense.

      You're a fucking retard.

    6. Re:Heads should roll by Plekto · · Score: 1

      A better solution would be to just block that root server. If China doesn't want to play along nicely, well, they can turn into their own mega-LAN all they want.

      In fact, I'd do one better take ALL of their internet access outside of China offline for them - just flat out cut the connection so that their entire country is in the dark. No news, no information, no business, no nothing. Not even their government and military has any information(aside from maybe a modem or two or satellite new feeds I guess)

      I'd give them about two months before the people in power weren't any more.

    7. Re:Heads should roll by Pteraspidomorphi · · Score: 1

      I think his point is that if China did not modify the responses in first place, this kind of problem would have had absolutely no negative consequences for users until being fixed (since all the servers should return consistent data). I don't hate China myself, but it isn't incorrect to resent those who are intentionally breaking the DNS rather than those who simply made a mistake (or ill-advised decision).

    8. Re:Heads should roll by budgenator · · Score: 1

      Dude, if your that addicted, just hand edit your Hosts file for slashdot, Youtube and facebook or roll your own bind server.

      --
      Apocalypse Cancelled, Sorry, No Ticket Refunds
    9. Re:Heads should roll by jon3k · · Score: 1

      I know it's easy to have the "nuke them from space" policy but honestly the Chinese government is just so fucked up they don't have the appropriate law enforcement or policies to police it. Then you've also probably got some level of government that's involved in a lot of the nasty shit going on. Yes, I realize most spam comes from the US. I don't know about you but the several thousand failed login attempts I see a day aren't coming from ARIN address space. It's all APNIC address space. And it's Chinese, specifically, on every single attempt. Now either the Chinese are totally incompetent at managing hackers living there, they turn a blind eye, or it's flat out state sponsored -- but it's largely irrelevant because none of those are acceptable.

      Half of me would rather be wrong than naive, just cut them off and say "screw you" but the other half of me considers the probably billion or so Chinese people that use the Internet legitimately just like you and me.

      So really, I'm on the fence.

  7. So I guess you could say... by Anonymous Coward · · Score: 5, Funny

    They got to the "Root" of the problem.

    [ducks]

    1. Re:So I guess you could say... by VanessaE · · Score: 2

      Yes, but they had to...ahem...route around for a solution.

  8. Re:Calling it now by TheDarAve · · Score: 4, Funny

    Instead of Germany annexing countries to start a world war, we have China firewalling them? That'd just be an odd way to start a war... "Ha ha! Now you must go through our internet filter!"

  9. What happened? by jbb999 · · Score: 2, Interesting

    All of the articles I've read about this seem to confuse DNS and BGP. My guess is that the IP of one of the root dns servers was being "hijacked" by the Chinese by announcing a route to it and that route was being picked up externally so some people thinking they were using the real dns root were being diverted a chinese root server giving out different IP addresses for lookups on these domains. Does that make sense?

    1. Re:What happened? by Anonymous Coward · · Score: 1, Informative

      No, my understanding is that BGP is used to advertise the IP of the server - they removed the route advertisement to shut the server off from the Internet but BGP wasn't actually causing the problem or compromised.

      It sounds like traffic OUT of the server was being modified in some way, I would doubt the data stored on the server had been modified as that probably flows over a secure connection but actual responses are public communications and the Chinese systems are likely filtering/modifying those so that when you try to visit twitter (or somesuch) it redirects you to a "sorry this page does not exist" site.

    2. Re:What happened? by Stephen+Samuel · · Score: 1

      Something like that .. Netnod apparently claims that the data on their server is accurate, so either China was hijacking the connection generally, or they were filtering the results being returned. This wasn't a problem until the server (and it's hacked data stream) started being accessed by machines outside of China due to a (silly but otherwise benign) routing change.

      --
      Free Software: Like love, it grows best when given away.
    3. Re:What happened? by mysticalreaper · · Score: 5, Informative

      Your suggestion makes sense, but that's not what happened.

      Something like this

      I.root-servers.net (beijing) -> chinese networks -> Chile networks

      So, the real I root server sent correct answers to the querying computer in Chile. But, as the DNS packet travelled across the Chinese network, it was modified, and so the packet received by the Chilean network was false, returning a fake IP address for some domains, like 'facebook.com'.

      This is called a 'man-in-the-middle attack'. The Chinese network, in the middle, is modifying packets.

      Once the I root server operators realized this was happening, they stopped the BGP route announcement from the I root server node in Beijing, so that queries to i.root-servers.net would not be answered in Beijing, but instead by the other i-root nodes. There are 34 currently, so no problems with load would occur shutting off one node.

      Hopefully that makes sense.

      P.S. www.root-servers.org

    4. Re:What happened? by LifesABeach · · Score: 1

      What amazes me about Chinese censorship is that rather than show that the opposite is true, the Chinese government causes those that disagree to not be heard; so much for those in command whose culture values wisdom and patience. Its like watching Sarah Palin read her notes on her hand on topics that my 14 year old daughter could debate either Pro or Con while trying desperately not to look too bored.

    5. Re:What happened? by radtea · · Score: 1

      so much for those in command whose culture values wisdom and patience.

      Chinese culture values wisdom and patience the way Canadian culture values lacrosse. If you didn't know anything about what Canadians actually do, but just read the official literature, you'd think lacrosse was a big deal. It's our national sport! Officially.

      If instead you behaved like an scientist, and looked at the empirical reality of what we do, you'd find this other game called hockey... And then there's this "curling" stuff...

      If you look at actual Chinese history, including recent history, you'll find a culture that values violence, genocide, class hatred, race hatred, torture, imperialism and oppression on a scale that puts it well up with the historical realities of the United States, England and Spain. The difference is that while those other countries have somewhat toned down their bad behaviour in the past fifty years, China is ramping up.

      --
      Blasphemy is a human right. Blasphemophobia kills.
  10. Re:Calling it now by Bacon+Bits · · Score: 1

    I should a lot of people would be very upset by the lack of porn.

    --
    The road to tyranny has always been paved with claims of necessity.
  11. Re:Australia impacted too, not just Chile by fremean · · Score: 2, Informative

    Actually, that does explain a lot of things - all through march I was having issues with Twitter on my Virgin connection yet I could ssh home to my Internode connection and twidge to my hearts content... I complained but they couldn't see a problem (they probably weren't using their own dns servers)

  12. I blame American ISP's by ironicsky · · Score: 3, Insightful

    I blame American and Chile ISP's.
    Why on earth would you query the root server on the other side of the world, especially in an ass backwards country like China when there are plenty of good servers here?
    Shouldn't you query the closest available server, not the furthest?

    1. Re:I blame American ISP's by mysticalreaper · · Score: 3, Insightful

      Basically, your ideas are right. The idea is to query the closest server, for best performance. DNS data is very small, so there's not much financial concern about transmitting data across the world (which happens all the time on the internet)

      Anyway, the logical routing of the internet doesn't always match the physical world. This is routine, and not a problem until DNS traffic crosses the great firewall of China, and is modified, which is what happened here.

      Since this, route announcements have changed, and the Beijing server is not being queried.

      But you are also correct about ISPs. ISPs can control (if they are good) which root servers are going to be queried from their network.

      My overall point is that everything was operating routinely and correctly, until a new kind of DNS problem, not observed in the wild ever before, started happening. It's hard to expect the ISPs to prevent a problem they never knew would occur.

    2. Re:I blame American ISP's by russotto · · Score: 1

      Shouldn't you query the closest available server, not the furthest?

      A host is a host/From coast to coast/And no one will talk to a host that's close/unless the host (that isn't close)/is busy, hung, or dead!
      (From the .signature file of one David Lesher...)

    3. Re:I blame American ISP's by jon3k · · Score: 1

      I agree completely. It's very simple to go into your DNS server root hints file and remove DNS servers you don't want to query. Pick your favorites, specifically ones near you, using anycast ideally, delete the rest -- problem solved.

  13. Re:Australia impacted too, not just Chile by datapharmer · · Score: 1

    Why didn't you just change your dns servers? You can set priority to strict you know.

    --
    Get a web developer
  14. Thailand affected, too by Daengbo · · Score: 1

    My Internet connection in Thailand has had hundreds of 404s for well known sites this week. Waiting a few minutes or forcing a refresh seems to work 70% of the time.

    --
    Put identity in the browser.
  15. Re:Denial of DNS service for evil Chinese? by Daengbo · · Score: 1

    It's the Chinese citizens who apparently don't have any rights. The government is doing whatever it wants.

    From Thailand (also censored, though not as badly).

    --
    Put identity in the browser.
  16. Re:productivity went up! by Anonymous Coward · · Score: 1, Funny

    If you measure productivity in F5 presses, yes.

  17. Re:Bigger News by jon3k · · Score: 1

    They are. Every modernized country does research in this area in anything from cloud seeding to haarp.