Slashdot Mirror

← Back to Stories (view on slashdot.org)

The Linux Kernel Archives

Posted by CmdrTaco on from the huey-lewis-takes-us-back-in-time dept.

Jeremy Andrews writes "KernelTrap offers an interesting look at the history behind the Linux Kernel Archives, home of the Linux kernel. They start from the beginning in 1997, when kernel.org ran on a generic "white box PC" using a shared T1, to the present where it runs on multiple quad Opterons each with 24 gigabytes of RAM, 10 terabytes of disk space, and a gigabit link to the internet. Much of the article is based on an interview with Peter Anvin, also including quotes from Linus Torvalds, Paul Vixie of Internet Systems Consortium, Inc who donates the bandwidth, and Matt Taggart of Hewlett-Packard who donated the hardware."

17 of 154 comments (clear)

  1. Nice to see folks getting credit... by tcopeland · · Score: 3, Interesting

    ...especially having dealt with something like this (on a much smaller scale) recently.

    We were having bandwidth limitations on RubyForge; it was getting up to 80 GB per month at the end of 2004. Mirroring out releases helped get usage back down to 15 GB per month. Many thanks to our mirror providers!

    --
    The Army reading list
  2. Yes but... by PR_Alistair · · Score: 5, Funny
    multiple quad Opterons each with 24 gigabytes of RAM, 10 terabytes of disk space, and a gigabit link to the internet
    ...does it run Linux?
    1. Re:Yes but... by daveKfs · · Score: 3, Funny

      More importantly, does it have a stopwatch?

      --
      i r baboon
  3. Slashdot history! by Dante · · Score: 4, Interesting

    This was a great article! I can attest the there is quite a difference with the new hardware, I got a 500KBps download last night while downloading rc3-mm2.

    Can we please have the same kind of article about slashdot hardware?

    --
    "think of it as evolution in action"
    1. Re:Slashdot history! by VistaBoy · · Score: 4, Funny

      Oh, you mean the Macintosh Plus that CmdrTaco has in the corner of his living room?

  4. Interesting quote by LiENUS · · Score: 5, Interesting

    The 'kernel.org' domain name was picked because by that time in 1997 the more logical seeming Linux dot names were already taken. The Transmeta domain was intentionally not used to avoid creating the false perception that Transmeta owned Linux.

    I wonder what would have happened with Transmeta and Linux if they had used the Transmeta domain to host the kernel archives. Would IBM have gotten involved with Linux? Would SCO have sued Transmeta instead of IBM? Would Linus have left Transmeta?

  5. Slashdotting? by TheKubrix · · Score: 3, Funny

    ...it runs on multiple quad Opterons each with 24 gigabytes of RAM, 10 terabytes of disk space, and a gigabit link to the internet...

    Do I smell a challenge?

  6. load average 1024??!!! by teeker · · Score: 4, Interesting

    Referring to 32-bit systems, Peter noted, "we learned that the Linux load average rolls over at 1024. And we actually found this out empirically."

    Can you even get the server to TELL you what the load is when it's that high?? That's INSANE!

    --
    teeker
  7. Hat's off to HP by oni · · Score: 4, Insightful

    I'm really happy to see HP giving so much support. I'll definitely remember this the next time someone asks my opinion about what server hardware to buy.

    1. Re:Hat's off to HP by Polycom+Sucks · · Score: 3, Informative

      I'll second this. Their support for many things is outstanding. HP really does seem to care about the end users.

      I have seen them support networking equipment purchased second (third?) hand off of eBay without asking "Where did you buy it?" or anything like that.

      In another instance, one of their supprot people searched through several dozen models of laptops and found one in a different product line that had compatible drivers so that an OS besides Windows XP could be used. No other company has come close to providing this level of support.

  8. Slashdot effect minor compared to .tar downloads by moz25 · · Score: 3, Interesting

    It doesn't surprise me that being linked from slashdot is just a minor effect. A kernel package is tens of megabytes, while a single visit will likely consume less than 100KB.

    --
    see a Text Widget
  9. noatime interesting by redelm · · Score: 4, Interesting
    More people should look into `noatime` for file-intensive systems. Peter said all the access time updates doubled his load average, and I've seen worse. Try running the `updatedb` to freshen the locate database. Takes minutes. remount FS noatime, flush buffers with a grepbomb, and it takes seconds. Remount with atime, back to minutes.

  10. Re:here's an idea by diegocgteleline.es · · Score: 4, Interesting

    DONT USE APACHE.

    This was suggested. The kernel.org people didn't seem to have interest in it. Those light http servers are probably good for lots of small static html files. kernel.org is not like that - it needs to serve + 20 MB files and CD ISOs. Your benchmarks don't measure that. I can bet the kernel.org people knows what they're using and they'd have switched if it'd be really useful.

  11. That's what rsync does by trb · · Score: 3, Interesting
    Serving data with http and ftp is is not very CPU intensive, but over time the amount of rsync traffic being fed by the kernel.org server continued to increase, and rsync is CPU intensive. "That's what rsync does" Peter said, "it trades bandwidth for CPU horsepower...

    I don't have occasion to use rsync, and I'm not too familiar with its design, but I think it synchs directories by checksumming the files in them to see if they differ. So Peter is saying above that the server's bottleneck is checksumming. I would think that on a server like this, checksums could be cached - why checksum a stable file more than once? Once you have a checksum for linux-2.6.0.tar.bz2, why calculate it again?

    This would require a bit of bookkeeping when files change, but wouldn't it be worth it on such a busy system? (Or am I confused?)

    1. Re:That's what rsync does by hpa · · Score: 3, Informative

      Fortunately, rsync has a compression exclude list.

  12. Interesting they use Fedora and not RH EL by Glasswire · · Score: 3, Funny

    ...I guess that:
    a) kernel.org doesn't think the 'enterprise readiness' of RH Enterprise Linux is that great, (vs what Fedora offers) even in what should be considered one of the most mission-critical sites in the Linux ecosystem (or that the difference with Fedora is worth paying for)
    b) No one at RH is bright enough to be embarassed by this and offer kernel.org some free licenses...

  13. Re:here's an idea by hpa · · Score: 4, Informative

    We have, indeed, considered that, but it'd not really buy us anything. Earlier Apaches would sit on a lot of memory while serving large files, but current versions just have a thread sitting in sendfile(), which is just about as lightweight as you get.

    Sure, the startup cost of the transaction is higher than for a lightweight HTTP server, but the startup cost of the transaction isn't a big deal for us, and we appreciate the flexibility that Apache offers.