Slashdot Mirror

← Back to Stories (view on slashdot.org)

Army DNS ROOT Server Down For 18+ Hours

Posted by Soulskill on from the inter-arma-enim-silent-tubulis dept.

An anonymous reader writes "The H-Root server, operated by the US Army Research Lab, spent 18 hours out of the last 48 being a void. Both the RIPE's DNSMON and the h.root-servers.org site show this. How, in this day and age of network engineering, can we even entertain one of the thirteen root servers being unavailable for so long? I mean, the US army doesn't even seem to make the effort to deploy more sites. Look at the other root operators who don't have the backing of the US government money machine. Many of them seem to be able to deploy redundant instances. Even the much-maligned ICANN seems to have managed deploying 11 sites. All these root operators that have only one site need a good swift kick, or maybe they should pass the responsibility to others who are more committed to ensuring the Internet's stability."

154 comments

  1. Army Intelligence? by toygeek · · Score: 2, Funny

    An Oxymoron indeed!

    --
    Nobodies Prefect
    Tidbits for Techs Technology Blog
    1. Re:Army Intelligence? by Anonymous Coward · · Score: 1, Insightful

      It was probably outsourced to the cheapest bidder. Either that or some incompetent idiots got the winning bid
      by greasing a few palms.

    2. Re:Army Intelligence? by Mr2cents · · Score: 4, Funny

      Don't be so harsh on the US military. They only have a trillion dollar budget, you know? How are you ever going to set up redundant systems if all you get is pocket change? You have to cut corners somewhere. Maybe it's time to increase their funding a bit more.

      --
      "It's too bad that stupidity isn't painful." - Anton LaVey
    3. Re:Army Intelligence? by Anonymous Coward · · Score: 0

      An Oxymoron indeed!

      In fact, forget about the oxy ...

    4. Re:Army Intelligence? by sumdumass · · Score: 3, Interesting

      Actually, given the size and scope of the US military, you are right, 1 trillion dollars is about pocket change to most people.

      I'm for increasing their budget more too. But I'm not sure that this outage wasn't planned. How better to test the ability to withstand a "cyber attack" then to lose your DNS servers and see if the your departments can fully function without them. This ability would greatly decrease the time needed to change to an alternative system if ever needed or more likely regroup resources and work around it. I'm not so sure that this wasn't just a readiness test of some sorts disguised as an accidental outage or something. It would make more sense to make it appear to be a problem server then actually gearing up to work around it that would create another potential target for any attacks.

    5. Re:Army Intelligence? by Runaway1956 · · Score: 2, Funny

      Careful - don't lump all the military together. It's the ARMY under discussion. My navy has problems, to be sure, but my navy can keep a server up and running. Not to mention, the navy wrote the book on repetitive redundancy. I think congress should take the server away from the army, and give to the navy. Overall security should improve, and physical security will most certainly improve. Our marines haven't lost a server yet!

      --
      "Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
    6. Re:Army Intelligence? by Anonymous Coward · · Score: 0

      What about the US Air Force? As everybody knows, the Air Force the the premiere US organ to deal with black holes sucking trough artificial wormholes. They also have the experience needed to deal with spider-like swarm intelligent toys gone haywire and human consciousnesses sucked to a computer systems. Surely they can handle a few redundant servers as well..

    7. Re:Army Intelligence? by Stupendoussteve · · Score: 2, Insightful

      Er... the Navy has outsourced to HP. In fact, to get out of the agreement they are having to pay to even receive information about the network configuration.

    8. Re:Army Intelligence? by delysid-x · · Score: 0

      HP is shit. Don't ever buy from them. More people need to know this.

    9. Re:Army Intelligence? by Anonymous Coward · · Score: 0

      So what would you recommend? Can't say that I've been too happy with Dell or Sun lately.

      And IBM was always my favorite, but with those prices...

    10. Re:Army Intelligence? by Mr2cents · · Score: 1

      How about, hmm, I don't know, reducing the scope? Did you ever see president Eisenhowers' farewell speech? If you didn't, you can always find it online.

      And no, one trillion dollar isn't pocket change. That's an insane amount of money. Just take a calculator and calculate the cost per capita, the "trillion" figure is a unit people aren't very familiar with, and looks deceivingly small. It isn't.

      --
      "It's too bad that stupidity isn't painful." - Anton LaVey
    11. Re:Army Intelligence? by ps2os2 · · Score: 0

      Well do not forget they are still using computers from the 1980's.
      You tell me where the money goes. Throwing more money at anything so antique is a waste of time & money.
      Of course the air force has a super INTERNET command center where they (it seems) daily move satallites around to avoid collisions.
      Sounds more like hype than anything and never underestimate how old equipment is in the Army. I also suspect that the education is a bit lean in this area. 40 years ago I was told I *HAD* to take proficiency test and was force to take it. The questions they had were not even close to reality. Subtract 30 years and that was the kind of questions they were asking. I had no plans for the Army so I just checked of a b c d e on every 5th
      question and walked out.

    12. Re:Army Intelligence? by sumdumass · · Score: 1

      Yea, it's pocket change considering what it's doing. And no, the military does not have a 1 trillion dollar budget, it actually has a much lower budget which was more apparent before Obama took the war spending from off budget expenditures and put it in the budget. He did that so when the war spending ramped down, they could increase other spending claiming it won't cost more because it's coming from the military.

      For the spending, the scope of the military is just fine. Well, that is unless your understanding of the military and what it is doing is limited to soundbites from political groups I guess.

    13. Re:Army Intelligence? by Mr2cents · · Score: 1

      http://en.wikipedia.org/wiki/Military_budget_of_the_United_States#Budget_Breakdown_for_2011

      I know nothing, heh? Fucktard.

      --
      "It's too bad that stupidity isn't painful." - Anton LaVey
    14. Re:Army Intelligence? by sumdumass · · Score: 1

      Read what I said, then read your own fucking link. Read especially the part that lists D.O.D funding at 721.3 billion and says it includes "Overseas Contingency Operations". Then look in the mirror and say fucktard again but really loud this time.

      As I said, the wars are traditionally financed off budget in emergency spending. The only reason the military budget is as high as it is is directly because the war spending has been moved into the budget so the funding can be spent elsewhere after the wars are over. Fuck, why don't you read the entire page you linked to, it actually says exactly what I said.

  2. So the Internet worked as it should... by Anonymous Coward · · Score: 5, Insightful

    So the Internet worked as it should, and routed around this disruption. The other root servers were unaffected, and still functioned fine. So what exactly is the problem?

    1. Re:So the Internet worked as it should... by jayhawk88 · · Score: 4, Funny

      Because it's Saturday, and we don't have anything else to get upset about! WE HAVE TO HAVE SOMETHING TO GET UPSET ABOUT, DON'T YOU UNDERSTAND?! How can I be expected to face the day if I'm not pissed off about something that doesn't directly affect me in any meaningful way?

    2. Re:So the Internet worked as it should... by Anonymous Coward · · Score: 1, Interesting

      Umm--having retired from the military, and having also been a networking professional for fifteen years in education and industry, I have less than a great respect for the products of the U.S. Army Signal School, who happens to operate that server. I was activated for service in Iraq, and watched a fellow captain, a graduate of that school, and someone with at least five years experience, insist that Ethernet Cat 5 had a maximum single link distance of 185 meters. And he designed his network around that premise. Which meant that we spent the next six months after they left cleaning up the mess and troubleshooting the bad links that SOMETIMES worked. Granted, this is not objective but it illustrates the point.

    3. Re:So the Internet worked as it should... by NekSnappa · · Score: 0, Offtopic

      I'm upset that the Jaguar XKRs aren't doing well at Prtite LeMans, and the Europeans seem to be making a comeback at the Ryder Cup.

      --
      I want to shoot the messenger!
    4. Re:So the Internet worked as it should... by OeLeWaPpErKe · · Score: 2, Insightful

      We've all made links in cat5 > 200 meters that work perfectly fine. Granted, perfect reliability is something else, but for a backup link in a datacenter that charges an arm and a leg for fiber connections and < 10% of that price for copper ... I've even been known to stick that link in a 10G copper interface card to see if it'd work (even if it didn't work). But I've had reliable gigabit copper links over > 250meters operational for years.It helps a lot if they're the only ethernet link in a metal cable tray.

      And the opposite as well. Ever had an ethernet link inside a bundle of VDSL links ? The link was barely 30 meters, but the error counters mounted faster than the traffic counters. And the link stayed up, so the routing protocol saw no need to reroute. Now that was a bitch to deal with. Especially since we couldn't replace the cable with cat6.

      If your network design can't deal with signal loss on individual links, especially when known beforehand that said links are located in a warzone, you have other problems than theoretical maximum link distances. And even in general : hardware WILL fail, so prepare for failure instead of investing untold resources in preventing it.

    5. Re:So the Internet worked as it should... by NemoinSpace · · Score: 1

      I think the problem is something not exactly unlike Army's root server was down for 18 hours.
      and all you can say is DON'T PANIC !?
      wait, what was the question again?

    6. Re:So the Internet worked as it should... by Dravik · · Score: 1

      Ever had an ethernet link inside a bundle of VDSL links ?

      Were you running shielded cable? That probably would have helped a lot more than going to Cat 6.

      --
      The purpose of language is communication, If the idea is clear the grammar ain't important
    7. Re:So the Internet worked as it should... by sjames · · Score: 1

      EXACTLY. That's why we have more than just root-server.net.

  3. Maybe.. just maybe.. by js3 · · Score: 1

    They didn't want YOU to access their servers?

    --
    did you forget to take your meds?
    1. Re:Maybe.. just maybe.. by Anonymous Coward · · Score: 0

      Obviously. If you dont have written permission to access their servers you'll
      get a visit from Homeland Insecurity. Hmm, theres someone at the door.

  4. Why is it their problem? by sjs132 · · Score: 2, Insightful

    Because they don't have redundancy? Everyone gets mad because the USA wants to control the internet, but let something go bad and then someone wants to point fingers? Really? I just don't get the mentality of "We want you to do this for free" and then people turn around and B&M about the service being down for a bit.

    --
    --- Relax, that mass muderer is just trying to reduce our carbon footprint, one fetus at a time...
    1. Re:Why is it their problem? by Sprouticus · · Score: 4, Insightful

      It has nothing to do with this being a US Army server. It has everything to do with bad design. The people given the responsibility of a root server should NOT take that responsibility lightly.

    2. Re:Why is it their problem? by WrongSizeGlass · · Score: 1

      If the US wants to "control the internet", which we do 'cuz there's Internet Money to be had, then we have the responsibility to keep the infrastructure up and running. How are we going to combat a 'cyber attack' without redundancy ... I mean really, even /. has a backup site, right?

    3. Re:Why is it their problem? by Anonymous Coward · · Score: -1, Troll

      Oh, I'm sorry! I wasn't aware the Military Industrial Complex's IT ran on fairy farts.

    4. Re:Why is it their problem? by Anonymous Coward · · Score: 0, Troll

      I don't follow your logic. We don't want US to control the internet "for free". We don't want US to control the government AT ALL. But if they don't want to give up their control, they could at least try to be somewhat competent at it.

      Step 1: People say "US Gov/Military shouldn't control the internet". Step 2: US Gov/Military fucks something up.

      By what logic should people not point fingers?

    5. Re:Why is it their problem? by Anonymous Coward · · Score: 0

      And, you know, tax payer dollars here. Let's scream and shout at how much money is being spent and then when the government doesn't spend it, complain some more.

    6. Re:Why is it their problem? by amorsen · · Score: 1

      It would probably be reasonably easy to get someone else to run the H server cluster. The DNS protocol itself limits the number to 13, quite by accident, and there was no grand design when it was decided who was getting them.

      If the US army can't run their server properly, they should offer the slot to someone else.

      --
      Finally! A year of moderation! Ready for 2019?
    7. Re:Why is it their problem? by Thinboy00 · · Score: 1

      Army:"No, we can't let you do that!"
      Me:"Why?"
      Army:"National Security [i.e. PR]. If you don't shut up now, we'll give your name to the FBI!" ...
      Do you really expect any other result?

      --
      $ make available
    8. Re:Why is it their problem? by darkpixel2k · · Score: 0, Troll

      Because they don't have redundancy?

      What do you mean they don't have redundancy? Last time I checked there were something like 13 root servers. The entire purpose of having multiple root servers is to keep the internet up when one or even a few go down.

      --
      There's no place like ::1 (I've completed my transition to IPv6)
    9. Re:Why is it their problem? by JWSmythe · · Score: 4, Informative

          Actually, most of the root "servers" are "anycast" now (9 of 13), so a single site failure doesn't matter. The US DoD runs two (G and H). G is anycast. H isn't. There wasn't clarification to what the issue was. It's easy to be quick to say "oh they suck", but shit happens sometimes. That's part of why we don't run on just one root nameserver. :)

          For all we know, it could have been a planned outage. I kinda doubt it with that size window, but who knows. It was only 1 of 13, which makes it more like 1 of an awful lot since 9 of the "servers" are really servers distributed world wide. I was doing some monitoring a while back, showing how our traffic moved, and that included monitoring the root servers. It made some really screwy routes, where one check would be in the US, and the next one would be somewhere in Europe.

      --
      Serious? Seriousness is well above my pay grade.
    10. Re:Why is it their problem? by amorsen · · Score: 1, Interesting

      I know most are anycast. I still think DoD should give up their slot to someone else, especially since they have 2. There is no reason why any organisation should have two slots; the only reason for that is historical.

      --
      Finally! A year of moderation! Ready for 2019?
    11. Re:Why is it their problem? by Anonymous Coward · · Score: 0

      The logic that says that there is more DNS root hardware outside the US than there are in within it? That most of those that are hosted, operated, and maintained outside the US are so hosted, etc by non-US companies, let alone US government agencies.

      The logic that this "failure" of the US government disrupted exactly zero service to anybody because .. oh shit, I know this is crazy, the fucking system functioned as designed.

    12. Re:Why is it their problem? by philipmather · · Score: 0

      Unless, and this is postulating a fair bit, that organisation wished to ensure the consistency of response between two independently run, organisationally compartmentalized systems by some other separate communications back channel? But yeah it does make you wonder what's going on there doesn't it?

      If a government decided that the Internet had become "critical national infrastructure" and that a reliable, independently verifiable DNS system was a crucial component of that system then they'd not only have at minimum N+1 separate DNS servers but would also consider separating the management and operation hierarchy of those servers.

      If I were designing such a system then (using British institutions)...

      1) I'd have the Foreign Office running one, Nominet another, a coalition of (ISP/hosting) businesses another and then charge CESG with establishing an independent, isolated network for verifying responses and ensuring it's integrity. Obviously you'd DNSSEC the whole thing, have some hosted in your embassies in other countries perhaps, maybe do NTP and/or some PKI system at the same time. When I say "one" I mean an any-cast system backed by distributed servers in reality, hence you'd be protected by a geographically diverse, triply redundant set of N+1 systems run by separate organisations ensuring against infiltration/subversion or some sort of inherent system flaw taking them all out at the same time.
      2) ????
      3) PROFIT!!!

      How hard can it be?

      --
      Regards, Phil
    13. Re:Why is it their problem? by Anonymous Coward · · Score: 0

      One interesting point is that root nameservers are managed so that each one uses different hardware/software/lb techniques. As such, 9/13 being anycast is reasonable, since all of them will not be allowed to be anycast, simply for diversity in setup. Now, to be down for so long indicates that whatever setup IS in place isn't reliable enough, I think that is clear, but I would agree it shouldn't go anycast because so many others are.

    14. Re:Why is it their problem? by mysidia · · Score: 1

      The reundandcy did its job.. There are 13 root servers, and the Army research lab only runs one of them. Individual root servers themselves were never expected to be redundant, that wasn't the design of the domain name system.

      Lets set aside for now the fact that the other root server operators are running additional root servers for profit. As in selling corporations the ability to have their very own local anycasted root server mirror.

      And likely making a ton of cash selling lookup data about queries made to root servers.

      What? You thought the root server operators were altruistic, and 'redundancy' was present in these systems, to improve reliability of the Internet, with no profit to the operators?

      hahaha....

    15. Re:Why is it their problem? by tehcyder · · Score: 1

      I just don't get the mentality of "We want you to do this for free"

      You must be new here. The US Army should provide the internet for free, and make its money by doing live gigs in the Middle East, and selling action figures.

      --
      To have a right to do a thing is not at all the same as to be right in doing it
    16. Re:Why is it their problem? by HappyPsycho · · Score: 1

      They accepted responsibility for a critical piece of internet infrastructure, here is the current list of expectations for said operators http://www.faqs.org/rfcs/bcp/bcp40.html

      What is probably annoying / concerning most people is, if you look at almost every major service that provides service to more than say 1 million plus people there is some sort of redundancy so that even if an entire site is taken out of commission no disruption occurs. The fact that DNS is the poster-child of the anycast technique is not helping their cause.

      The "shit happens" argument affects everyone equally, so why is no one else site down for such a long run for what appears to be a predictable reason (tropical storm damage to fiber lines). The cynical version of this question is "synchronized backhoes anyone?".

      Due to technical limitations within the DNS protocol itself the number of root servers is limited to 13, given that this operator (and supposedly others) seem to not be capable of handling the responsibility of being in the position they are, is it time to give the responsibility to others? From the looks of it, the only reason the research lab is in control of H is that they had facilities during the internet's infancy.

  5. One down, several dozens up by Anonymous Coward · · Score: 2, Insightful

    What's the problem? The point of redundancy isn't to keep all redundant instances up all the time. The system is designed to allow for downtime of quite a few servers.

    1. Re:One down, several dozens up by ani23 · · Score: 1

      Redundancy is good. Just not in comments

    2. Re:One down, several dozens up by PReDiToR · · Score: 1

      Redndancy shouldn't be in comments, although it is good.

      --

      Do not meddle in the affairs of geeks for they are subtle and quick to anger
  6. Lowest bidder by pixiekhatt · · Score: 4, Insightful

    This is what happens when you give contracts to the lowest bidder. The military may have tons of money, but that doesn't mean they spend it wisely. Even if it's not a contracted company taking care of these servers, and it's government employees (there's a difference), a LOT of those employees get their jobs based on keywords and general qualifications and several have a 'I did my time in the military and retired, they owe me this for all the hard work I did before' attitude. Not everyone is like that, and I've met some government employees (in the tech field) who really did know their stuff.. and not all contracts are bad -- but they can turn sour when a company steps in, says they'll do all that and more for this much less, and they really don't know what they're doing. I've seen that happen too. And if it's managed by soldiers.. well. They always told us, you're a soldier first, and a 'whatever your job is' after. Most technically trained soldiers don't know how to do their job well, or even at all. They just tough it out until they're an NCO, and then they're supposed to be a leader and tell their underlings to do the work.

    1. Re:Lowest bidder by Isao · · Score: 4, Interesting

      There are two main approaches to government contracting: Lowest Cost and Best Value. Contrary to popular belief, Lowest Cost is not always the one chosen, by a long shot. I also previously misunderstood "Close enough for government work." Turns out most "government work" has very specific requirements and specifications, or you don't get paid. If you see something different, please call Waste, Fraud & Abuse.

    2. Re:Lowest bidder by hsmith · · Score: 1

      Lowest bidder does not equate getting the contracting job. Not in this day and age.

    3. Re:Lowest bidder by Anonymous Coward · · Score: 1, Insightful

      To tack on, often times those extremely expensive "military spec" tools and such are expensive due to having to meet standards that would normally considered ridiculous. The reason being that it's usually not too hard to head off and buy a new one, but in the middle of a war zone, it's both time consuming and risky to assume you can get a new one.

    4. Re:Lowest bidder by John+Hasler · · Score: 2, Insightful

      > This is what happens when you give contracts to the lowest bidder.

      Because they'd obviously get better results by giving them to the highest bidder...

      Try to get your head around concepts like "requirements", "specifications", and "lowest qualified bid". You not only do not get paid if you don't do the job you agreed to do, you may even have to pay the extra cost of having someone else do it over.

      --
      Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
    5. Re:Lowest bidder by AK+Marc · · Score: 1

      If you see something different, please call Waste, Fraud & Abuse.

      So every government project that runs over budget, I should report it as the fraud it is? There are only a few firms that have mastered the hoops to get a government contract, and they seem to bid low, spend high, and send the bill to the feds. And the feds pay it and come back for more. Are you saying that reporting them for that will have any effect? Because from where I sit, a disproportionate number of contracts are over budget and late and the contractor essentially gets bonuses for underperforming.

      --
      Learn to love Alaska
    6. Re:Lowest bidder by shentino · · Score: 1

      The military industrial complex probably helps conceal a lot of fraud in much the same way that lobbyists and politicians schmooze.

      No politician is going to give up a cushy position at a company by burning a bridge through enforcement.

    7. Re:Lowest bidder by Anonymous Coward · · Score: 0

      The key is to bribe, or blow, the senator controlling funding.

      http://www.sourcewatch.org/index.php?title=Congressional_Bribery_and_a_DC_Sex_Scandal

    8. Re:Lowest bidder by sjames · · Score: 1

      That in connection with sometimes screwy requirements is why the job always goes to large corporations whose primary skill is convincing government agencies that they'd better sign off on the half-assed job that was done.

    9. Re:Lowest bidder by Anonymous Coward · · Score: 0

      There are two main approaches to government contracting: Lowest Cost and Best Value. Contrary to popular belief, Lowest Cost is not always the one chosen, by a long shot. I also previously misunderstood "Close enough for government work." Turns out most "government work" has very specific requirements and specifications, or you don't get paid. If you see something different, please call Waste, Fraud & Abuse.

      "Close enough for government work" doesn't mean the government side. It refers to what the $50 hammer people can actually slip by the inspectors. Which may even include a $50 hammer that meets government specs but could be produced for 75 cents providing you found a loophole where the government forgot to require that the head stay attached in the rain.

      You want real "close enough for government work" take a look at some of what people were selling the Union Army back during the Civil War.

    10. Re:Lowest bidder by Anonymous Coward · · Score: 0

      I worked for the Army in their IT division. I have to say that you are mostly wrong in your assumptions. Their contracts are overseen by EDS which is one of the biggest IT outsourcing companies in the country, or the world for that matter. EDS subcontracts to smaller companies sometimes as well, but the majority are employed through EDS directly. What you're describing is the Civil Service, and yes, its mostly filled with those people, but those people are also not in charge of big projects like this.

      Many of the people who maintain the big-time, high-priority projects like this certainly are among some of the best in their field. Instead of employing people that may know about Active Directory for example, they employ people that originally helped design it. Even the computers are among some of the most locked-down machines I have ever seen in my life. The amount of customization and security checks is truly insane. These "technically trained soldiers" that you mention in reality, don't know anything and they'll admit to that. The powers that be have acknowledged this and have since made the machines extremely difficult to tamper with.

      Lastly, the DoD cut thousands of high-end IT contracts with EDS a year ago due to budget cuts. The money has been refocused on war efforts and less on the IT front.

  7. Not the hugest problem out there.,,, by CajunArson · · Score: -1, Flamebait

    The H-Root server does NOT run Windows, just in case anyone was wondering ;-)

    On a more serious note, while the downtime is bad... there are 13 root servers owned by different organizations (both government & non-government) for a reason.. to provide redundancy. Interestingly, the D-Root at College Park and H-Root at Aberdeen are relatively close to one another geographically. The distributing the H-Root service would be nice, but there are lots of other letters to use in the Root namespace. In short: The Army should probably take some steps to beef things up, but the (usual) mouth-breathing hypersensationalized crap spewed in the summary is mostly for getting ad revenue into Taco's bank account and not a rational evaluation of the situation.

    --
    AntiFA: An abbreviation for Anti First Amendment.
    1. Re:Not the hugest problem out there.,,, by memojuez · · Score: 1

      Mike Muuss must be spinning in his grave.

      --
      Signature applied for, Patent Pending
    2. Re:Not the hugest problem out there.,,, by forkazoo · · Score: 1

      Mike Muuss must be spinning in his grave.

      For the most part, I can confirm that, but there was an 18+ hour period where a percentage of my queries simply reported that Mike Muss was unknown. Odd.

  8. Military Mentality by Anonymous Coward · · Score: -1, Troll

    "I don't give a fuck, I'm still going to get my paycheck, and won't be fired."

    1. Re:Military Mentality by hedwards · · Score: 1

      Where did you hear that? They don't "fire" people per se, but they do have these things called court martials, dishonerable discharge and if I'm not mistaken one can be discharged if one fails to get promoted. I'm sure the details vary somewhat depending upon the branch, but they definitely can get rid of people that aren't living up to their duties. And that's before you get to lesser things like being transferred somewhere miserable.

    2. Re:Military Mentality by Anonymous Coward · · Score: 0

      Ironically this is why there are 13 servers. This doesn't even matter at all.

    3. Re:Military Mentality by Anonymous Coward · · Score: 0

      I didn't hear it. I've seen it. A court martial is simply a trial. A dishonorable discharge is the equivalent of a serious felony. IE murder, rape, grand theft, assault. The details to not very depending on the branch since all branches fall under UCMJ. Being transferred somewhere miserable is never the solution. A prime example of someone who should have been fired, and not transferred was the terrorist who was promoted to the rank of Major and proceeded to shoot a dozen people on FT Hood. To give you an idea of how well the military is ran, use your own image of how well the government is ran, and its about the same. The military is just an extension of the government after all.

      Yes you can be discharged if you don't make rank, but this never happens. You can be discharged for dozens of things. The most common (85%) are failing piss test, and DUI.

  9. Finally.. by Anonymous Coward · · Score: 0

    A story I'm interested in reading... the only problem is there is no article?

  10. Bad design, no actual impact by Sprouticus · · Score: 0

    Having a root server without multiple instances running is horrible. Anyone who has spent an hour studying DNS would understand how bad a decision this is.....

    That having been said, this is one root server amongst many, so the actual impatc is almost zero. If your DNS server is only pointing at 1 rot server, than you are more foolish that the US Army and deserve what you get.

    Still, they need to fix this and move into the 1990's.

    1. Re:Bad design, no actual impact by Anonymous Coward · · Score: 0

      > Having a root server without multiple instances running is horrible

      You're right... that's why there's A, B, C, D, E ...

  11. There are 12 others - pick one. by Anonymous Coward · · Score: 5, Insightful

    Hardware fails. That's just how it is. Even with the highest end hardware available today, outages can happen. This is why there are 13 root servers to start with. So long as they don't all go down at once, all is good. As far as 18 hours to recover, why is that bad? With 12 others to pick from, should this one be a high priority? I think not. Getting one's panties in a bunch because a server fails and takes some time to recover makes you sound like a silly management type. Most of us lived at least a large part of our lives without any root servers - or any servers at all. It's not the end of the world if DNS goes down. It will be ok, I promise.

    1. Re:There are 12 others - pick one. by Iamthecheese · · Score: -1, Flamebait

      Most of us lived at least a large part of our lives without any root servers - or any servers at all. It's not the end of the world if DNS goes down. It will be ok, I promise.

      You are an idiot.

      At one time it wouldn't have been a disaster for DNS to go down. Now we have everything from business to business transactions to stock trading to government bonds to consumer purchases being done online. We have hospitals depending on the internet to get their plasma on time. We have a billion people using social networks for hours. We have farmers using the internet to check the weather, militaries using the internet to transmit vital intelligence, and kids using the internet to call home and say they'll be late.

      From the small to the great the world is online now and even an hour's outage of the internet would be a disaster comparable in economic and social cost to the complete destruction of a small city somewhere in the world.

      --
      If video games influenced behavior the Pac Man generation would be eating pills and running away from their problems.
    2. Re:There are 12 others - pick one. by horatio · · Score: 1
      Too bad you posted as A/C, because you make a good point. Further, quoting the summary:

      ...or maybe they should pass the responsibility to others who are more committed to ensuring the Internet's stability.

      Maybe, before opening your fat mouth and posting on /. something you have no facts on, but seem to confidently be able state that the US Army has "acted stupidly" - you research what went wrong and then pass judgement. The parent is correct - there are 13 root servers so that one or two or three CAN go down - either because of a failure or for maintenance - without killing the whole interwebs.

      --
      There is very little future in being right when your boss is wrong.
    3. Re:There are 12 others - pick one. by forkazoo · · Score: 5, Insightful

      Most of us lived at least a large part of our lives without any root servers - or any servers at all. It's not the end of the world if DNS goes down. It will be ok, I promise.

      You are an idiot.

      At one time it wouldn't have been a disaster for DNS to go down. Now we have everything from business to business transactions to stock trading to government bonds to consumer purchases being done online. We have hospitals depending on the internet to get their plasma on time. We have a billion people using social networks for hours. We have farmers using the internet to check the weather, militaries using the internet to transmit vital intelligence, and kids using the internet to call home and say they'll be late.

      Meh. It's just one of 13 roots. Almost nobody queries it directly. If I have my DNS pointing to my ISP DNS, or to Google DNS, or to my own recursive caching DNS Server which uses one of those as an upstream, all 13 root servers could be down for literally days and it's likely that almost nobody would ever notice. Most DNS servers will retain large caches of most domains. If something freaks out when the roots disappear, a few small ISP's might need to make some quick configuration changes. Some DNS changes wouldn' propagate properly until the DNS root servers were back online. But, frankly, life would go on. Making all of DNS go away would be pretty much impossible, short of taking out every node on the Internet.

      Yes, if *All 13* root servers suddenly died, there would be a few people who would get a late night at the office, but I certainly wouldn't see the effects directly.

    4. Re:There are 12 others - pick one. by Posting=!Working · · Score: 1

      From the small to the great the world is online now and even an hour's outage of the internet would be a disaster comparable in economic and social cost to the complete destruction of a small city somewhere in the world.

      Ummmmm, no. Not at all. Not even close. Not even remotely comparable in any way. Not a comparison that survives even one second of rational thought. Not a sentence that I was able to finish without thinking the very phrase you started your post with.

      --
      This sentence no verb.
    5. Re:There are 12 others - pick one. by Iamthecheese · · Score: 0, Troll

      Then think more rationally. What is the cost of a missed email? How about a thousand of them? If one non-spam email in 10,000 contains an urgent piece of information what's the cost of missing an hour's worth? How about purchases? What if every internet based store, currency trading mechanism, bond exchange and commodity exchange lost an hour's income? How much the cost of 1 billion missed "I know you're there and I support you" connections between friends? How much the cost of 1,000 drivers that can't contact a tow truck, 100,000 telecommuters that can't sign in to work, 1,000,000 phone calls that don't happen, and 10,000,000 attempts to do some bit of research that fail? A million businesses that can't get int touch with a million others?

      --
      If video games influenced behavior the Pac Man generation would be eating pills and running away from their problems.
    6. Re:There are 12 others - pick one. by Enry · · Score: 1

      How many e-mails were dropped as a result of the one (out of 13) DNS server that was down? How many web sites were unavailable? Did you even know there was a problem before seeing it here on /.?

      Redundancy is there for a reason - to make sure that things continue even if one or more systems are unavailable.

      Step back, take a breath, and get on with your weekend.

    7. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 0, Troll

      Maybe, before opening your fat mouth and posting on /. something you have no facts on, but seem to confidently be able state that the US Army has "acted stupidly" - you research what went wrong and then pass judgement.

      What went wrong is that a server that's not supposed to ever go down went down. What went wrong is that there are a number of well-established redundancy schemes that are capable of preventing such an outage. What went wrong is that somebody at the Army who ran this server didn't use a single one of them or didn't do it correctly. So yeah, they acted stupidly. Go ahead and rub your nose in it until you get over your "how DARE you claim incompetence within the Army" offense. I'll wait.

    8. Re:There are 12 others - pick one. by Iamthecheese · · Score: 1

      I'm not talking about the cost of losing a root server, I'm refuting the claim that the internet can go down without significant cost. The "or any server" part.

      --
      If video games influenced behavior the Pac Man generation would be eating pills and running away from their problems.
    9. Re:There are 12 others - pick one. by Bengie · · Score: 1

      18 hours down is only 99.6% uptime averaged over the year assuming no other failures. A well maintained server can have 99.99%-99.999% uptime. They should have a virtual server that can failover to other hardware without an end user noticing. Each server should have multiple network connections in case a NIC or switch fails. Not to mention a major server should have one admin on hand 24/7 and a recovery plan that can get the server back-up-and-running in MUCH less than 18 houirs. We're not talking about some huge disc image to restore, just a large list of DNS records and a stripped down hardened OS with a DNS daemon. A worst case scenario failure should take under an hour.

      I would assume something as this important is housed in a datacenter and not a closet.

    10. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 0

      Hardware fails. That's just how it is. Even with the highest end hardware available today, outages can happen.

      Ummm, no. Many of us have redundant systems for exactly that reason. Highly unlikely for all of them to suffer failure at the same time.

    11. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 0

      A machine that's not supposed to ever go down - there is no such animal. EVERY machine goes down every now and then, maintenance, environment issues, etc. Without knowing the full situation of what happened, how can you claim this was not an intentional outage with any certainty?

    12. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 1, Informative

      It's just one of 13 roots.

      Actually, it's one out of over 200. There are only 13 IP addresses, but behind most of these addresses (anycast) there are multiple sites.

    13. Re:There are 12 others - pick one. by Fumus · · Score: 0, Offtopic

      From the small to the great the world is online now and even an hour's outage of the internet would be a disaster comparable in economic and social cost to the complete destruction of a small city somewhere in the world.

      And I would gladly watch it all crash and burn.

    14. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 1, Insightful

      What is the cost of a missed email?

      1 phonecall

      How about a thousand of them?

      Less spam. (seriously, can you think of anyone else needing to send 1000 mails in one hour?)

      If one non-spam email in 10,000 contains an urgent piece of information what's the cost of missing an hour's worth?

      1. SMTP does not guarantee timely delivery.
      2. Sending an email now does not give you any assurance on when it will be read.
      Anyone using email for time-critical information transport runs a risk -- that can be foreseen.

      How about purchases?

      "Oh noes! Amazon is offline! Where do I shop now?!?!?"

      What if every internet based store, currency trading mechanism, bond exchange and commodity exchange lost an hour's income?

      What if? Seriously: what if???
      Some companies making money don't do so for 1 hour. Oh dear.
      Oh wait, I'm not paid by money-making companies to care about them. So I don't.
      (besides, if all of them are offline, then no one can turn to competition -- I think the effects wouldn't be as severe as when half of them were offline)

      How much the cost of 1 billion missed "I know you're there and I support you" connections between friends?

      1. For ONE FRIGGIN HOUR?!?! Get real. Seriously.
      I've got some great friends, but sometimes whole nights go by without them telling me they're there for me. Or vice versa.
      We tend to respect each other's sleep like that.
      2. If the Internet is the only connection you have to your friends, are they really your friends?

      How much the cost of 1,000 drivers that can't contact a tow truck

      Because internet is down?? How about they phone? Or, you know, talk to people?
      Besides that: the cost is 1,000 drivers having to wait one hour. Which is nowhere near the end of the world.

      100,000 telecommuters that can't sign in to work

      Woohoo!! The first of your arguments that I feel is somewhat legitimate.
      I think they would more or less do the same thing as their in-office colleagues at the same time.
      (I doubt most teleworkers need a permanent internet connection to do any work)

      1,000,000 phone calls that don't happen

      They exit skype and use the damn phone?

      and 10,000,000 attempts to do some bit of research that fail?

      For one hour.
      Seriously, whenever I run into some problem with Ubuntu, I either find a solution in 5 minutes or I easily fail for an hour in doing research.
      No sweat, I'll pick it up later, with a fresh mind. That usually does it.

      A million businesses that can't get int touch with a million others?

      "Oh noes!! No internet, that means we can't contact anyone!!!"
      Unless they have a phone, fax, physical location, post, ...

      You know, there was life before the internet too. And it worked pretty nicely for a while. Even the most conservative estimates place that at roughly 6,000 years.

    15. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 0

      So long as they all don't go down at once

      I think you missed his most important point. And maybe a couple of modders did, too.

      But thanks for playing, anyway. Should have held off on 'idiot' though.

    16. Re:There are 12 others - pick one. by Posting=!Working · · Score: 1

      You left off "for one whole hour" to the end of every sentence. Of those 1,000,000 phone calls, all the important ones would be made once the lines are back up. Ditto the research, the telecommuters, most purchases, email, etc. The tow trucks would get there. The markets wouldn't collapse if you couldn't trade for one hour, in fact, if no one could trade, there would be no change at all.

      It would cost money, but not anywhere near the ridiculous claim you made.. Destroying a city is permanent, you never get it back. The infrastructure alone would cost more than the internet going down for an hour, let alone all the buildings, vehicles, personal possessions, businesses and ecology.

      That's just economic cost, the social cost claim is so far from rational the light leaving rational will not reach it for thousands of years. Hundreds of thousands dead is an insanely large social cost. They're gone, no way to get them back. Tens of millions would be affected by this, the effects would last years/decades.

      --
      This sentence no verb.
    17. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 0

      The destruction equivalent of one small city, spread out over the entire world, would be barely noticeable. The DNS thing would probably be worse. However, I'm still pretty sure it wouldn't be the end of the world, even in a metaphorical sense. If the entire DNS system were out for a month, _then_ we'd have some problems.

    18. Re:There are 12 others - pick one. by pooly7 · · Score: 1

      We have a billion people using social networks for hours.

      Well maybe we'll gain productivity if they get to work instead of wasting time!

    19. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 0

      It's actually more like 99.795% uptime. If we are going to be precise to that degree, might as well be precise. They probably don't have a concrete set in stone SLA with anyone, so a ridiculously solid uptime isn't a legal obligation like you see with other data centers that have such reliable numbers.

    20. Re:There are 12 others - pick one. by Ironsides · · Score: 1

      18 hours down is only 99.6% uptime averaged over the year assuming no other failures. A well maintained server can have 99.99%-99.999% uptime.

      And here you just went and mixed two different time periods. Do you seriously believe a 99.99% or 99.999% uptime is measured over a single year? Lets look at when the previous time the Army rootserver went down. Was it anytime within the last two years? If no, then they have 99.99% uptime.

      --
      Fly me to the moon Let me sing among those stars Let me see what spring is like On jupiter and mars
    21. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 0


      I'm not talking about the cost of losing a root server, I'm refuting the claim that the internet can go down without significant cost. The "or any server" part.

      Who made that claim? Seems like you're building up strawmen

    22. Re:There are 12 others - pick one. by ceoyoyo · · Score: 1

      "Now we have everything from business to business transactions" so sad. Businesses might have to pick up a phone.

      "stock trading" Aw, the high frequency traders will have to take a day off!

      "government bonds to consumer purchases" Government bonds are pretty slow turnover. A one day holiday would be no big deal. If you've just gotta have your bonds, there's always the phone. Or actually going to a bank! As for consumer purchases, a day off from that wouldn't hurt anyone either. And there's always getting off your butt and going to the store.

      "have hospitals depending on the internet to get their plasma on time" No we don't.

      "We have a billion people using social networks for hours." OMG!

      "We have farmers using the internet to check the weather" What did farmers do before the Internet? I suppose TV, radio, newspapers and looking out the window have all been sadly neglected and are now unusable because of the Internet. Oh, no, wait. They haven't.

      "militaries using the internet to transmit vital intelligence" You'd better hope not. If so, they probably deserve to be defeated. Even if they did use the Internet, hopefully someone would be smart enough to type in the actual IP addresses instead of using DNS.

      "kids using the internet to call home and say they'll be late" Right... there are a LOT of kids who call home using the Internet and have absolutely no other way of contacting their parents. Not that parents aren't used to their kids not calling anyway.

      You're pretty young, aren't you?

    23. Re:There are 12 others - pick one. by jimicus · · Score: -1, Flamebait

      How on Earth did you ever get modded insightful?

      What do you think your ISPs DNS server uses to for the first step of resolving any query it doesn't have cached? Psychic powers? Magic fairy dust?

      Having said that, the whole point of having multiple root DNS servers is to ensure the failure of one is nothing more than a minor irritation.

    24. Re:There are 12 others - pick one. by LingNoi · · Score: 1, Informative

      Right at the beginning of the thread you fricken moron.

    25. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 0

      Yeah, and the system is redundant, you silly cunt. There were still TWELVE root servers remaining.

    26. Re:There are 12 others - pick one. by Stupendoussteve · · Score: 1

      Calm down, it's not FaceBook that went down. The redundancy is designed specifically so these machines are allowed to go down from time to time, and designed that way in a time when redundancy behind the address was not very likely.

      The net effect of this was a little more traffic on the other servers.

    27. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 0

      Parent not flamebait. Simple truth.

    28. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 0

      Wouldn't this server going down cause dns lookups to fail globally about 1 out of every 13 attempts? Or do resolvers remember that is is offline and avoid using it?

    29. Re:There are 12 others - pick one. by Pinhedd · · Score: 1

      Do the root servers even act as forward resolvers? I thought the whole point of them was to resolve the name servers for the TLD authorities

    30. Re:There are 12 others - pick one. by stephanruby · · Score: 2, Interesting

      Go ahead and rub your nose in it until you get over your "how DARE you claim incompetence within the Army" offense.

      First, let me start by saying that the guy you replied to was rude, and I don't see why he needed to insult you to make his point. However...

      What went wrong is that a server that's not supposed to ever go down went down.

      Your argument seems circular. Your assumption is that this root server is never supposed to go down. In this physical world, that's a pretty huge assumption to make.

      And no, saying that the server went down is no proof positive that it should never have happened. The fact is, there was redundancy and the redundancy kicked in as it was supposed to. Now we're saying the redundancy can be outside of root, or inside of root, it doesn't really matter. And you're saying that the redundancy has to be ***inside*** of root, there can be no other way.

      Tell us, have you read something that gave you that idea? Me, I'm thinking that you probably read that recent Times (or Newsweek) article, if anything, I do agree that the article only seemed to romanticize and emphasize the importance of root servers, but I'd argue that the article was more a piece of flamboyant story-telling than an actual report on an actual technology. The truth about root servers is far less sexy than what the article did imply. The real truth is that if all the root dns servers went down at the same time, most of the internet and its dns would keep on working pretty nicely. We'd be running on old possibly slightly outdated cached dns information, but that wouldn't really matter -- it's only the end results that would matter anyway.

    31. Re:There are 12 others - pick one. by orin · · Score: 1

      Yet everyone queries it (or one of the other roots) indirectly. http://en.wikipedia.org/wiki/Root_nameserver

    32. Re:There are 12 others - pick one. by Anonymous Coward · · Score: 0

      Holy shit man it's just the internet. People can still use telephones and fax machines. Most big companies have their own backbones that don't rely on the internet at all for internal communications.

      There are always situations where someone digs up and breaks an underground line, Exchange/Domino etc. goes down, AT&T fucks up something downstream, and the list goes on. Any business that is crippled entirely by missed e-mail needs to focus on their own redundancy plans and not rely on the internet for survival. If they depend on it to operate they'll destroyed by their competition at the first failure.

    33. Re:There are 12 others - pick one. by sjames · · Score: 1

      Why, do we just have money to burn?

      I can't speak for everyone, but I didn't even notice that it was down, so how much can it be worth to make sure it never happens again?

    34. Re:There are 12 others - pick one. by sjames · · Score: 1

      Except it won't happen that way. If the DNS is all down, the mail servers will just retry and find it back up later. Email is only required to either deliver the mail or report it undeliverable within FIVE DAYS. If you actually depend on faster delivery, you're using the wrong service.

      If every online store goes down for an hour, do you know what will REALLY happen? Everyone will try to order again later and it will go through just fine. A bit of an anti-climax really.

      Besides that, nothing remotely like that happened. Very few people noticed the "problem" at all.

    35. Re:There are 12 others - pick one. by SuiteSisterMary · · Score: 1

      A well maintained server can have 99.99%-99.999% uptime.

      And I do believe that the DNS Root Server meets that, if not betters it. You'll notice that while the one node was down, the other 12 were up and running. The DNS root system is, when you get right down to it, a 13-node HA cluster. The system stayed up and serviceable, even though one specific node was down. Functioned as designed.

      --
      Vintage computer games and RPG books available. Email me if you're interested.
  12. Really, I'm going to be the first? by Anonymous Coward · · Score: 4, Funny

    They're sticking to their moto and deploying an Army of one.

    1. Re:Really, I'm going to be the first? by Sulphur · · Score: 2, Funny

      When the movie comes out, will it be Stephen Spielberg, James Cameron, or Mel Brooks?

  13. Gnutella, Diaspora like decentralization by unity100 · · Score: -1

    This is what we need for DNS system. p2p dns.

    --
    Read radical news here
    1. Re:Gnutella, Diaspora like decentralization by jojoba_oil · · Score: 1

      I would trust the internet even less if DNS was P2P. I can just imagine trying to look up Bubba: I ask Jack who Bubba is and he says, "a buddy from prison." Then I ask Jill who Bubba is and she says, "My sugardaddy." Sometimes these Bubbas would be the same person, and sometimes they wouldn't.

    2. Re:Gnutella, Diaspora like decentralization by mother_reincarnated · · Score: 1

      This is what we need for the DNS system - a decentralized distributed directory.

      Oh wait that's EXACTLY WHAT IT IS! Notice how there was no outage because the H-root was down.

      And like sibling jojoba points out- p2p DNS would be HORRIBLE from a trust perspective.

    3. Re:Gnutella, Diaspora like decentralization by unity100 · · Score: 1

      fool. all the root servers are under control of one country, and its organizations. does that mean 'decentralized' ?

      --
      Read radical news here
    4. Re:Gnutella, Diaspora like decentralization by andymadigan · · Score: 1

      Did I miss something? Did the U.S. conquer Japan, Europe and the Middle East while I was sleeping?

      *Looks up map of U.S. on Wikipedia*

      Nope, I guess the I, K and M servers are safely controlled by NON-US organizations.

      I'll agree the U.S. probably has too many of them, but aside from the military I'll assume the other orgs know what they're doing.

      Stop spreading FUD.

      --
      The right to protest the State is more sacred than the State.
    5. Re:Gnutella, Diaspora like decentralization by unity100 · · Score: 1

      all of those countries belong to the same bloc in the west, which is controlled by u.s.. the very channels which acta is being pushed over, has the control to the means which control those satellite countries too. the only exception being eu parliament. eu parl is an exception, because, european parties used to send their overly idealistic, 'troublemaking' members there to keep them away from home turf. eu parl gained authority right in the middle of term, and the results naturally been a surprise. next election term, the parties which are generally us puppets are going to pay attention to who they are sending to eu parl.

      its not fud. its politics. and you are unaware of it.

      --
      Read radical news here
    6. Re:Gnutella, Diaspora like decentralization by andymadigan · · Score: 1

      I assure you I am well aware of politics. The U.S. does not control Europe, certainly not Japan or Sweden. They act in their own interests, the fact that the interests of many Western countries are similar does not mean one controls the others.

      --
      The right to protest the State is more sacred than the State.
    7. Re:Gnutella, Diaspora like decentralization by unity100 · · Score: 1

      no, around half of europe is at the moment ruled by u.s. aligned, or puppet parties. in the others, more aligned/puppet parties are trying to get the power, either through civil means or underhanded means. death of almost entire polish top level govt contingency in a crash while landing in a russian airport, should hint what kind of games are going on around europe. or, the sudden and unnecessary agitation of georgia against russia. the other parts of europe are only immune to such extreme theatrics because the power is more settled there in the hands of the puppet parties. the parties themselves and their figureheads, are easily identifiable. blair was a good example. he is still a good example. berlusconi another. all are in the same monkey league, and league's governance is in usa.

      'interests of many western countries are similar' is the biggest bullshit and self-make believe that people fool themselves with. see how interests of western countries made the same :

      http://en.wikipedia.org/wiki/Operation_Gladio

      any country which joined nato, had to get this organization's equivalent.

      --
      Read radical news here
    8. Re:Gnutella, Diaspora like decentralization by Anonymous Coward · · Score: 0

      Well yeah, they're not gonna put a root DNS server in some hostile third world country. Of course they're going to pick a country they know has a power grid and telecommunications network capable of reliably running it, and who's not going to use it for their own crazy agenda.

      Either way the DNS root servers have no impact on politics at all I'm sure.

    9. Re:Gnutella, Diaspora like decentralization by unity100 · · Score: 1

      yeees, and those countries all happen to be american satellites, in any form.

      and, the dns root servers have no impact on politics ... is it ? with all that freedom and free information about, you think private interests, strict regimes are not interested in it ...

      --
      Read radical news here
    10. Re:Gnutella, Diaspora like decentralization by yyxx · · Score: 1

      all are in the same monkey league, and league's governance is in usa

      The US is by far the most powerful of the Western allies, and therefore it sets the agenda. No conspiracy there.

      any country which joined nato, had to get this organization's equivalent.

      They also had to commit troops and put them under US command, they were scrutinized by US intelligence, and they had to allow US troops to be stationed on their soil, imagine that!

      The still wanted to join, voluntarily, because of the Soviet threat and because the US had what it took to hold it off.

      And it was pretty good, because all of a sudden, warring and genocidal empires like France, England, Germany, and Turkey had to cooperate instead of plotting the next world war.

    11. Re:Gnutella, Diaspora like decentralization by yyxx · · Score: 1

      You or anybody can set their DNS service to whatever they like; it's less than a minute. You can even run your own DNS service. Most people just happen to prefer the DNS services running in "American satellites", probably because they also do their shopping, their socializing, and their traveling in those "American satellites".

  14. wow by buddyglass · · Score: 4, Insightful

    Whine much?

    1. Re:wow by Anonymous Coward · · Score: 0

      I can haz some cheese with that?

  15. Could it possibly happen because.. by Anonymous Coward · · Score: 0

    Some muscle bound, Rambo GI Joe type, trampled around inside the server room to play paint ball wargame, and managed to trip on the servers' main power cord accidentally?

    1. Re:Could it possibly happen because.. by Anonymous Coward · · Score: 1, Funny

      I'm betting that some muscle bound, Rambo GI Joe type was trampling around inside of your rectum last night with his third leg.

    2. Re:Could it possibly happen because.. by Anonymous Coward · · Score: 0

      This is the Army Research Lab, not Quantico. Rest assured it was a pimply-faced geek with glasses, a PhD, and no girlfriend who tripped over the server's main power cord accidentally.

  16. Was it the monitoring system? by Antique+Geekmeister · · Score: 2, Interesting

    I've seen numerous instances where the monitoring system, itself, was confused or detached. The results on a chart are then quite confusing, unless you know how to backfill the data in the chart.

    Why, no, I've never been asked to do that for a 99.999% uptime SLA monitored site when some confused person in the offsite monitoring station put a bad IP address in /etc/hosts. No, no, no, couldn't happen.

    1. Re:Was it the monitoring system? by Anonymous Coward · · Score: 5, Informative

      https://lists.dns-oarc.net/pipermail/dns-operations/2010-October/006142.html
      Classification: UNCLASSIFIED
      Caveats: NONE

      > FYI, the H root server is currently experiencing an outage
      > due to a SONET ring outage possibly caused by flooding from
      > the tropical storm on the east coast. No estimated repair time.

      H root returned to service at 12:30 UTC today. Fiber cut due to downed
      utility poles. Repair was delayed due to high water.

  17. The Army Research Lab? by MoldySpore · · Score: 1

    They are too busy getting blocked by my PeerBlock application to deploy more DNS sites.

    --

    "I hope you know how very lucky you are to know me, because I am so incredibly incredible."

  18. And you idiots think the US gov't can do better... by Anonymous Coward · · Score: -1, Offtopic

    micromanaging your personal healthcare choices? LOL, just LOL. Just lost your arm in a car wreck? Slap some duct tape on it and walk it off, son. Gramps has a cold? Better send in the nurse to "fluff his pillow," a.k.a. press it against his face until he stops moving.

  19. "backing of the US government money" by Joce640k · · Score: 4, Insightful

    Rest assured, the government isn't holding back. Those non-redundant Army servers already cost an order of magnitude more then everybody else's redundant servers.

    --
    No sig today...
  20. Re:And you idiots think the US gov't can do better by Anonymous Coward · · Score: 0

    Old troll is old.

  21. Why was this posted? by Anonymous Coward · · Score: 0

    No news outlets have picked up on this, and rightly so, since it isn't a big deal.

    If the other 12 root servers were also down, then it might warrant a story. ;)

    Reminds me of a stupid boss I had that got on my case for not being overly worried when one system in a three-level redundant design went off-line for a few minutes.

    He flipped out screaming that the entire system was all "aye ree" (his pronunciation of "awry"). I guess he had no clue what "redundant" meant, either.

    1. Re:Why was this posted? by 6Yankee · · Score: 1

      Or maybe he was a closet Rastafarian, and was rather animatedly telling you not to worry. "Ev'ryt'ing irie, mon..." :)

  22. Coincidence - Curfew for US Personnel in Germany? by Anonymous Coward · · Score: 0

    Wondering, at least one US Base in Germany had a curfew for all DOD personnel from Oct 1 2300 through Oct 2 0500, with base installation closures from Oct 1 1800 ...

  23. Meant to happen by DrPepper · · Score: 1

    I think you are overreacting a little bit. The expectation always was that one or more root servers would be unavailable at any one time - hence why there are 13 different root server systems available. More than one can be unavailable for days, and due to redundancy and caching it won't affect anything - as expected, nobody has really noticed this blip.

    There should be a good mix of technologies used in the different root server systems - different architectures, OS, etc. Some sites use anycast which gives massive redundancy within that system as well as providing good performance. However other architectures have their place and may be more robust to attack or certain failures. We need the variety.

    So technically it's a shame that H has gone down - they don't seem to have a good track record. Fortunately this time it isn't an issue.

  24. Re:Coincidence - Medal of Honor not available? by Anonymous Coward · · Score: 0

    And during the curfew/outage they could not access Medal of Honor online multi-player... wow that is a strange coincidence...

    Maybe someone in IT thought the AAFES ban meant they had to shut down access, and they was their plan...

  25. Non-story by A+beautiful+mind · · Score: 2, Interesting

    You have to realise that the layout of the root dns server hierarchy is historical. It is composed of organizations that are vastly different now than they were 20 years ago. The H root server people don't seem to care about things very much and there are a couple of other root servers where the organizations operating them don't put too much effort into things.

    Luckily, the internet doesn't really depend on them, as there are a couple of big organizations with heavy investment into making sure the root servers stay accessible all the time, like RIPE or Verisign. They operate thousands of physical machines at dozens of geographically distributed locations, all structured under one ip address, via anycast. This results in the situation where one logical root server outweights the other one in terms of physical boxes at least 100:1, if not more.

    My last information about the Verisign operated root servers from a couple years ago for example is that they are ridiculously overprovisioned, operating well under 1% used capacity, even when subjected to a fairly large DDOS. As far as I know, the common dns servers all support rtt banding, so basically using a random list of dns servers for a given resource that fall below a threshold of latency, therefor they wouldn't really notice the H root being down.

    --
    It takes a man to suffer ignorance and smile
    Be yourself no matter what they say
  26. It's just a drill: Cyber Storm III by Xemu · · Score: 3, Interesting

    Could this simply be a part of the Cyber Storm III information warfare exercise?
    http://www.military-technologies.net/2010/09/29/test-of-first-us-cyber-blitz-response-plan-begins/

    --
    Tell your friends about xenu.net
    1. Re:It's just a drill: Cyber Storm III by turtleshadow · · Score: 1

      Its either planned or SNAFU.

      I'd lean toward planned. Somewhere that has to be some infographics showing the Internet doing its thing in reorganizing small whole in the DNS.

      I've heard stories of .gov with 3 letter names alligator clipping batteries to the powercords of servers in order to move them "uninterrupted" So I think they got the right stuff to keep the hardware going.

  27. Re:Not the biggest problem out there.,,, by gnieboer · · Score: 4, Interesting

    Agreed.

    From the offending server's website: "BRL volunteered to host one of the original root servers ... to provide a root server for the MILNET in the event that MILNET had to be disconnected from the Internet."

    The purpose of the G/H servers is not to support the greater good (that's a side benefit), but to ensure that the MILNET can function if the DoD cuts itself off from the rest of the internet.

    And besides, If my math is correct, there are a total of 205 redundant root sites (http://www.root-servers.org/), so imagine going up asking for funding...
    [IT Guy] "General, we need money to add another redundant root server site, if all the sites go down the internet collapses!"
    [General] "That sounds bad! How many redundant sites are there now?"
    [IT Guy] "Only 205"
    [General]

  28. And which Product? by gearloos · · Score: 1

    ..and which Microsoft Product are you running?

    --
    "Computers are a lot like Air Conditioners" "They both work great until you start opening Windows"
  29. Simple by Anonymous Coward · · Score: 0

    It was down for that long because that is the amount of time it took them to install the new monitoring hardware.

  30. You must be mistaken, check again by UnknowingFool · · Score: 1

    *Unplugs toaster oven and plugs back in server*

    --BOFH

    --
    Well, there's spam egg sausage and spam, that's not got much spam in it.
    1. Re:You must be mistaken, check again by Idiomatick · · Score: 1

      What?! How could you let a fracking toaster into the server room?

  31. Probably not unplanned by RobNich · · Score: 1

    My guess is that since this root server is designed to operate on MILNET after disconnecting from the Internet, they may have been running a drill to do just that. Also, I highly doubt that this is the only root server on MILNET. I expect that they have multiple sites and plenty of redundant locations, but they only give out the Maryland location for security reasons.

    --
    Hello little man. I will destroy you!
  32. Hmmm by BrokenHalo · · Score: 1

    Did anyone actually notice the outage?

    1. Re:Hmmm by delysid-x · · Score: 1

      Not really, there is some redundancy in the system. But 7.5% of the DNS system was down. We should definitely increase the number of people who control the DNS system. Preferably all over the world. Heck, let's give Iran a couple too and be fair.

  33. You already have that problem by Colin+Smith · · Score: 1

    And like sibling jojoba points out- p2p DNS would be HORRIBLE from a trust perspective.

    Actually, what it means, is that we would have to actually fix once and for all, the identity/trust/reputation problem that the Internet already engenders. Unless you use https for everything, signed emails etc you are already trusting people all over the place.
     

    --
    Deleted
  34. A good swift kick? by Klync · · Score: 2, Funny

    > All these root operators that have only one site need a good swift kick...

    Alright, anonymous coward, I nominate YOU to be the one to go and give the US Army a "good swift kick". See ya when you get back!

    --

    ----
    Not to be confused with Col.
  35. Errr.. by Arimus · · Score: 1

    How, in this day and age of network engineering, can we even entertain one of the thirteen root servers being unavailable for so long?

    Just a small, minor issue there... 1 of the root dns went down... only another 12 still up. Not really a problem even if it had been down for a week.

    The whole reason for having 13 root servers is that you can lose a fair few of them before anyone needs to start worrying.

    --
    --- Users are like bacteria -> Each one causing a thousand tiny crises until the host finally gives up and dies.
  36. Re:Coincidence - Curfew for US Personnel in German by Stupendoussteve · · Score: 1

    I'm betting it had more to do with the Tropical Storm that hit the US East Coast, as referenced in the announcement and "back online" emails sent from the US Army. Maybe they're in on the conspiracy.

  37. Fibre was cut by SpaFF · · Score: 1

    I heard through the grapevine that a cable at ARL was cut. I can't find anything to substantiate this other than a slightly related "unscheduled network maintenance" notice here

    --
    -----BEGIN GEEK CODE BLOCK----- Version: 3.12 GIT d? s: a-- C++++ UL++++ P++ L+++ E- W++ N o-- K- w--- O- M+ V PS+ P
    1. Re:Fibre was cut by jaygridley · · Score: 1

      It was a fiber cut. Poor weather caused delays in getting repairs made. See: http://isc.sans.edu/diary.html?storyid=9655 Given the fact that there is more than one root server I find this a bit of a non-story. Obviously the internet continued to function as it should.

  38. Non-Event by Anonymous Coward · · Score: 0

    Your mom was down for 18 hours.

  39. mod parent up - it has actually *gasp* information by shallot · · Score: 2, Funny

    Wish I had mod points...

    Of the 64 comments I see in full, only this one has actual pertinent information about the downtime.

    ...

    I must be new here. :)

  40. Re:Not the biggest problem out there.,,, by thegarbz · · Score: 1

    Exactly. Why do root servers need to be redundant themselves? Aren't they already made redundant by the fact that there's 13 primary servers? Call me crazy but I thought these were setup in a way so they CAN go down without causing as much as a news worthy event.

  41. Maybe use a local root & save yourself traffic by Anonymous Coward · · Score: 0

    I use a local root to reduce outside traffic, load on the roots and to say f-u to the organizations that get a hard on by trying to control the dns system and make money off that situation.

    Maybe people using bind can't/don't know how to do that, but with djbdns it is fast/easy to set up.

    More than a signed root we need a signed root file that can be retrieved from a website/rsync site so that everyone can just run their own root and reduce the load on the official roots down to a trickle but that scheme would be attacked with extreme prejudice by the people with a vested interest in maintaining centralized control of dns and the internet.

  42. Mods on crack by Anonymous Coward · · Score: 0

    Stop using troll and flamebait mods for "I disagree with this post"