Slashdot Mirror


CloudFlare Was Hit By Leap Second, Causing Its RRDNS Software To 'Panic' (silicon.co.uk)

Reader Mickeycaskill writes: The extra leap second added on to the end of 2016 may not have had an effect on most people, but it did catch out a few web companies who failed to factor it in. Web services and security firm CloudFlare was one such example. A small number of its servers went down at midnight UTC on New Year's Day due to an error in its RRDNS software, a domain name service (DNS) proxy that was written to help scale CloudFlare's DNS infrastructure, which limited web access for some of its customers. As CloudFlare explained, a number went negative in the software when it should have been zero, causing RRDNS to "panic" and affect the DNS resolutions to some websites. The issue was confirmed by the company's engineers at 00:34 UTC on New Year's Day and the fix -- which involved patching the clock source to ensure it normalises if time ever skips backwards -- was rolled out to the majority of the affected data centres by 02:50 UTC. Cloudflare said the outage only hit customers who use CNAME DNS records with its service. Google works around leap seconds with a so-called "smearing" technique -- running clocks slightly slower than usual on its Network Time Protocol servers.

5 of 119 comments (clear)

  1. Re:Was the Go prog lang at fault? Would Rust help? by Anonymous Coward · · Score: 3, Insightful

    I don't know if you can blame the language, the devs should have added their own checks if the language didn't have a guarantee.

  2. Re:Was the Go prog lang at fault? Would Rust help? by Anonymous Coward · · Score: 3, Insightful

    Why would you even think of switching programing languages due to the simple and sadly common 'bug' of programmers not verifying parameters match a function's documented pre-conditions? My only guess is you're paid to promote Rust. Lazy programmers will write bugs in every language.

  3. Re: Was the Go prog lang at fault? Would Rust help by fubarrr · · Score: 4, Insightful

    >RRDNS is written in Go

    Their bugs are in HR department.

    Who in the world hired people who are dumb enought to use an experimental language in production?

  4. Re:Was the Go prog lang at fault? Would Rust help? by Obfuscant · · Score: 4, Insightful

    The Go documentation clearly says it panics if n = 0.

    And it says it panics if n is less than 0.

    If you write a library function that requires positive input always, and returns positive output always, then use unsigned input and output variables. A good compiler will flag the attempt at sending such a function a signed input as a warning at least. Pedantic compilers will fail -- better than the production program failing.

    And while it seems stupid, the proper action when asked for a random number between 0 and 0 is to return 0, not panic. (I believe the [ on the range means "including", but I could be wrong. If it didn't mean "including", then the documentation should be '(1,n)'.)

    But then, the test cases for the DNS code should have included 0 and negative, so this should have been caught when the function was tested.

  5. Re:Hosts hardcodes avoid DNS issues by Ash-Fox · · Score: 3, Insightful

    Do not trust APK's software, APK is a criminal , he is blatantly violating the Computer Fraud and Abuse Act by posting here as he is banned from Slashdot.

    APK's ban evasion has lead to more restrictive filters being placed on Slashdot that hinder good discussions.

    --
    Change is certain; progress is not obligatory.