Kernel.org Needs Some Help, Perl Foundation Got Some

Dante wrote in to say "I just read this on the Linux Kernel mailing list, it's from Peter Anvin, one of the ftp.kernel.org maintainers... H. Peter Anvin writes: "The recent troubles we've had at kernel.org pretty much highlight the issues with having an offsite system with no easy physical access. This begs the question if we could establish another primary kernel.org site; this would not only reduce the load on any one site but deal with any one failure in a much more graceful way.

Anyone have any ideas of some organization who would be willing to host a second kernel.org server? Such an organization should expect around 25 Mbit/s sustained traffic, and up to 40-100 Mbit/s peak traffic (this one can be adjusted to fit the available resources.) If so, please contact me."

In related news, mbadolato wrote in to tell us that "there's a press release over at dyndns.org announcing that they've donated $20,000 to the Perl Foundation!

'Thanks primarily to Perl and other Open Source technologies, we are able to provide DNS services to over 180,000 members of the Internet community,' said Tim Wilde, founder and chief executive officer of DynDNS.org. 'This is our way of giving back to some of the people whose tireless devotion to writing quality software has enabled us to provide our services to the Internet community over the past three years.'

The donation page for the Perl Foundation can be found here

Comments

Q: Bandwidth composition

[tweggen]

What is the composition of these bandwidth requirements? I mean, if it's primarily file download, requirements could maybe be met by a decentral system. Just curious...

donation ability

[Anonymous Coward]

Is kernel.org a 501(c)3 org so that whomever decides to donate bandwidth can have 'some' help and not take all the expense of moving that much traffic?

Why not IBM ?

[Krapangor]

They made of lot of claims that they support open source etc.

Freenet is the future?

[John Zero]

This is just another case, where the Freenet Project could help, in the future.

Besides being an anonymous (but authentic) information storage, it is also higly distributed.
In this case, that would mean there would be no "bottleneck", instead, the kernel tar.gz would be distributed, in small blocks.

Too bad it's yet under development, but it's getting better and better.

Re:erm, Google?

[3seas]

Yeah, and they do an excellent job of providing us all with usenet archives +++, free of charge!

Re:The real problem

[NearlyHeadless]

Of course, it goes without saying that everyone should still use their local mirror, particularly as kernel.org will only be accessible to mirrors for the forseeable future

I've thought a bit about this. It seems that every separate open-source project has to set up their own mirrors, there is no automatic system for finding "nearby" connections or for load-balancing, and volunteering for being a mirror can cause you to incur quite a bit of bandwidth costs.

I would be willing to pay a modest fee for downloads. I don't know if many of the other open-source fans (notorious cheapskates) would be willing, but if they were:

• A site like SourceForge could take the lead.
  
• Anybody who wants could have their software hosted there.
  
• Anybody who wants to would create an account to download.
  
• Charges would be proportional to bandwidth used, possibly off-peak times lower.
  
• SourceForge (or whoever) could subcontract connections to Akamai, Connxion, or others.
  
• Possibly a portion of the money collected could go to the creators of the software, the big problem being how to decide how the money gets divided for projects with many contributors or a lot of code re-use.
  
• Anybody could still distribute independently. It would not (could not) be an exclusive arrangement.
  

Mirroring Scheme?

[suwain_2]

The first thing I have to say is largely irrelevant, but quite a good idea, IMHO. Let's move the 'official' list of kernel.org mirrors *off* the kernel.org machine. When you can't load their main page at all, it makes no sense to expect me to use a mirror, since I don't have that list.

That said... An idea struck me. Suppose kernel.org develops a system where incoming requests are sent to a server; the routing is based on the preferences of the admin of that server. For example, let's say I work at a small webhosting company, and have a couple of T3s. (I don't really.) All our servers run Linux, and I want to give back to the community, and show everyone how cool I am. But I'm gonna go out of business if I allow 90 Mbps of bandwidth to be going to kernel.myfakelittlehostingcompany.com, because my customers wouldn't have any bandwidth.

So I decide "Well... I can spare 10 Mbps at the most." I could tell the kernel.org admins this, and when you went to kernel.org, you would be redirected to a site, based on what the mirror sites wanted.

I'm willing to be that companies like OSDN, RedHat, Mandrake, Rackspace, etc. might be willing to let a kernel.org mirror have a small bit of their bandwidth, if they had a way of knowing that it would be controlled.

Re:The real problem

[wampus]

Give that (wo)man a cigar... and some karma!

I would MUCH rather grab linux-base-2.4.18.tar.bz2 and linux-i386-2.4.18.tar.bz2. Possibly break it up even further... I don't need or want SCSI drivers Video4Linux, ISDN support, etc, etc. Why not a simple utility similar to make xconfig that not only configures your kernel, but downloads the neccesary subsystems to your tree. This could also be used to patch for new releases. Just hit the "Upgrade" button and your hard drive grinds for a few seconds and spits out a shiny new kernel.
This could even automate the process of checking signatures, similar to Ximian's Red Carpet. I know I never bother to check when I download from kernel.org or mirrors, but if the utility that downloads automagically does it, those that are worried about j03 h@x0r putting up a rogue mirror will be that much more likely to use the mirrors.

Take a line from FreeBSD

[Dom2]

In FreeBSD, CVSup is used to keep source trees in sync. It's a very efficient way of keeping several hundred megs of source code up to date.

I realise that CVSup is oriented towards CVS trees, which the Linux kernel isn't, but even an rsync server would be better than continuously downloading the patch.

The reason I mention this is because of the support infrastructure available in FreeBSD:

1. Install cvsup (once)
2. edit /etc/make.conf (once)
3. cd /usr/src && make update

CVSup is available at http://www.polstra.com/projects/freeware/CVSup/

Re:The real problem

[axioun]

I believe using mirror sites exclusively is the key, too. This could be done either with the users' knowledge or behind the scenes. The first method would probably be the quickest to implement at this time. It would involve something much like what Tucows has. Have a main site that links to mirrors based on region. This selector system might even temporarily not show links to sites that are currently experiencing high loads so people don't just pick the server on the top of the list and /. it.

This could probably be simplified by creating a small program that automates the server search process, and possibly download and apply patches on the fly. Later we might have a mirror system that is distributed. This latter stage could have a system like Freenet which is P2P or a load-balancing, centralized system.

Another method that just came to mind is to use a system similar to Audiogalaxy or Napster. While you've just downloaded a new kernel, your download software remains running. The software could have a default time to terminate. While the software is running, it acts as a small server, and the "Kernelgalaxy" software (what a fitting name) controls distribution in a distributed manner.