Slashdot Mirror
Entire .SE TLD Drops Off the Internet
Icemaann writes "Pingdom and Network World are reporting that the SE tld dropped off the internet yesterday due to a bug in the script that generates the SE zone file. The SE tld has close to one million domains that all went down due to missing the trailing dot in the SE zone file. Some caching nameservers may still be returning invalid DNS responses for 24 hours."
Goat.se
Yeah, been there done that. *My* fumble only brought 10,000 domains down for about 10 minutes, and no one noticed. (I think all the domains hosted only cat pictures anyway.)
Sorry, that's as big a responsibility as any employer has ever deemed suitable for my incompetent ass.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
The downtime lasted 30 minutes, and most domains were probably cached by nameservers anyway.
I once viddied an animated documentary about a small town in Colorado that lost the internet for 22 minutes. It was not pretty. Our hearts and minds go out to you, people of Sweden. I cannot even fathom what that would be like ... I hope the looting and rioting has died down with the restoration of the internet.
My work here is dung.
I seriously hope someone is fired or loses a contract over this. Where was the validation, change control, etc? I would expect that at the TLD level, a change to a configuration file would have to be inspected by someone AND run through some syntax-checking scripts...
As for the person who was modded up for saying "hey, no big deal, fixed in 30 minutes!", not quite. DNS servers (and individual computers!) cache negative results. Anything anyone did a query on during those 30 minutes will be negatively cached by their system and their local DNS server. Granted, a whole lot of local Swedish ISPs and network providers have probably flushed their DNS server caches, but it's still going to seriously impact traffic to many, many sites, especially for everyone outside Sweden.
Please help metamoderate.
...borked!
Uh, it would make no difference.
DNS is hierarchical, and has teh caching.
2 independent groups running DNS would strive to make sure they sync with each other quickly - thus all failures would sync quickly too.
The difference between
- the delay of a correct change propagating across the two firms running DNS
- the delay of an incorrect change propagating within a single DNS
would essentially be zero.
No good things could come from what you propose unless it was specifically designed to have a 24 hour delay or something.
Can't get to milkmaids.se ? Try milkmaids.se via DNS2 to get a 24-hour old version.
This is something the CURRENT DNS system could support - explicitly calling for older versions.
In fact, it might be worthwhile. Somebody write an RFC.
its "no big deal" until you need to know something off the internet right now, high stakes
I need to know what a fourteen year old thinks about copyright law and I need to know it NOW !
If all else fails, immortality can always be assured by spectacular error.
The Internet was started as, and always has been, a "best effort" network. If a packet gets through, great. If not, well, it's not the end of the world. People have tried to code more and more resilient protocols on top to be as robust as possible, but in the end it's a very fragile system that can go down quite easily.
Anything sufficiently "high stakes" shouldn't rely on an unreliable medium.
It still boggles my mind that anyone thought zone files are a good idea. The file format is so damn brittle, that a single byte can spell disaster. On top of that, the hierarchical naming structure presents an inherent systemic risk for all sub-domains as exhibited by this .se fiasco. Nevermind the injection attacks, Pakistan taking out Youtube, and the rest, you have organizations like Verisign which profit immensely off of keeping the system broken. And don't even bother mentioning DNSSEC, as it still doesn't resolve this fundamental issue. The next systemic fuckup will simply be a signed fuckup.
The downtime lasted 30 minutes, and most domains were probably cached by nameservers anyway.
I didn't notice the DNS freak out, but I did notice the internet's smug meter had dropped about 30%.
I am the richest astronaut ever to win the superbowl.
The actual downtime is no big deal, but the reason it happened is. Evidently, the registrar for an entire country's domain likes to roll out changes to the primary zone file without any sort of testing or syntax checking first. Simply having a small network (one or two computers) running a test root server, and running your scripts against that first, would have discovered the bug.
DNS is very simple, but it's just as prone to human error as anything else. If you're responsible for the records of a large number of domains (like, say, an entire country), you probably ought to take some time to develop proper testing and change control procedures before you fiddle with it. It sounds like these guys didn't take it seriously enough and got burned. I hope they'll learn their lesson from this and change their procedures.
Are you my motherboard?
rewriting history since 2109
Wi nøt trei a høliday in Sweden this yer?
See the løveli lakes
The wonderful telephøne system
And mani interesting furry animals
#DeleteChrome
It looks like someone messed up the summary. I'm pretty sure it should be:
Peengdum und Netvurk Vurld ere-a repurteeng thet zee SE tld drupped ooffff zee internet yesterdey dooe-a tu a boog in zee screept thet generetes zee SE zune-a feele-a. Zee SE tld hes cluse-a tu oone-a meelliun dumeeens thet ell vent doon dooe-a tu meessing zee treeeling dut in zee SE zune-a feele-a. Sume-a cecheeng nemeserfers mey steell be-a retoorneeng infeleed DNS respunses fur 24 huoors.
DNS is very simple, but it's just as prone to human error as anything else.
Are you kidding? I've been programming DNS for a long time, and if theirs one thing I learned, its that programmers like me don't make errors.
I am the richest astronaut ever to win the superbowl.
but I did notice the internet's smug meter had dropped about 30%.
Norwegian detected.
Show me on the doll where his noodly appendage touched you.