Kernel.org Needs Some Help, Perl Foundation Got Some

Dante wrote in to say "I just read this on the Linux Kernel mailing list, it's from Peter Anvin, one of the ftp.kernel.org maintainers... H. Peter Anvin writes: "The recent troubles we've had at kernel.org pretty much highlight the issues with having an offsite system with no easy physical access. This begs the question if we could establish another primary kernel.org site; this would not only reduce the load on any one site but deal with any one failure in a much more graceful way.

Anyone have any ideas of some organization who would be willing to host a second kernel.org server? Such an organization should expect around 25 Mbit/s sustained traffic, and up to 40-100 Mbit/s peak traffic (this one can be adjusted to fit the available resources.) If so, please contact me."

In related news, mbadolato wrote in to tell us that "there's a press release over at dyndns.org announcing that they've donated $20,000 to the Perl Foundation!

'Thanks primarily to Perl and other Open Source technologies, we are able to provide DNS services to over 180,000 members of the Internet community,' said Tim Wilde, founder and chief executive officer of DynDNS.org. 'This is our way of giving back to some of the people whose tireless devotion to writing quality software has enabled us to provide our services to the Internet community over the past three years.'

The donation page for the Perl Foundation can be found here

Comments

To persuade RH...

[sebol]

To persuade RH, AOL willing to sponsor second kernel.org site.

Good for the Perl Foundation

[Anonymous Coward]

Perl Foundation Got Some

Well it's about time! I couldn't bear to think about those 45 year old GNU hippie geek virgins working at the Perl Foundation anymore.

-Metrollica

Read my UPDATED journals!

Dyn Dns.

[ImaLamer]

Wow! Because of this donation to the Perl guys and gals, my check is in the mail.

I use DynDNS, and have been thinking about sending them *something*. I don't have much, but to see them donate a little something in return is nice. Any donation is cheaper than getting a 'real' domain name. Plus *.ath.cx is kinda cool, I wonder if goatse.ath.cx is available?

I just hope all these donations don't go to stuff like strippers. I could be spending my money on that.

Not just downloads...

[dreamquick]

But I would imagine that everytime a new kernal is released that world+dog go to view the site. I serious doubt that everyone who goes to the site downloads, most just read - lots of people reading still requires a fair old chunk of bandwidth.

As we saw earlier in the week the /. effect seemed to bring that site to its knees, but as regular news sites see linux as more and more relevant to their audience they too will link people in, adding to the problem.

Realistically they are victims of their own success - people want information about the new kernal and doubtlessly they want to download stuff too.

As these once limited interest sites become more mainstream, then it's clear that they need to maintain quality of service, and that means no /. effects which stop people going to your site and could potentially discourage them from going there in the future.

Just my 2c

donation ability

[Anonymous Coward]

Is kernel.org a 501(c)3 org so that whomever decides to donate bandwidth can have 'some' help and not take all the expense of moving that much traffic?

The real problem

[AirLace]

The real problem is not lack of bandwidth. There's plenty of it to go around. What saddens me is that the ISC is throwing away most of $80,000 annually because people can't be bothered to patch their kernel, and instead rely on downloading the full 20MB tarball every time a new kernel is released.

The solution to the problem is really quite simple. As Larry McVoy, who maintains the powerful but non-free BitKeeper RCS system and knows a thing or two about patches, has hinted towards kernel.org may be better off not providing a tarball for each release, instead providing some kind of utility that downloads the latest available full kernel, but only if necessary, plus patches. I'd be all for it. In the meantime, there are a number of incremental patching systems for the Linux kernel that automatically download patches, verify their signatures and patch the kernel which may be worth looking into to save time, bandwidth and resources:

• dlkern
• buildkernel
• lkpatch, which has fallen into disrepair
  

Of course, it goes without saying that everyone should still use their local mirror, particularly as kernel.org will only be accessible to mirrors for the forseeable future.

Re:The real problem

[GrafZahl]

IMHO your last sentence is the key to the solution. Other projects use a system whereby they don't allow direct ftp / http download access. This should be purely for mirror sites.

It would also help, if /. would not put direct links to kernel.org but would instead provide or link to a list of mirrors.

BTW, the following message I just got back from www dot kernel dot org:

The Linux Kernel Archives is currently offline due to a hardware failure. However, mirror sites are receiving updates; please use a mirror site instead.

May be this is the beginning of the end to direct access to them?!

Regards,
REB

Re:The real problem

[cperciva]

kernel.org may be better off not providing a tarball for each release, instead providing some kind of utility that downloads the latest available full kernel, but only if necessary, plus patches

I agree, that's a great idea. But it needs a good name... how about calling it CVSup?

Re:The real problem

[kinnunen]

There should be a 'make update' that atomatically retrieves (from the nearest mirror) the patch, uncompresses it and performs the patching. I can't imagine it would be too hard to code and the ease of use should convince even the "I have a 100M pipe so I don't bother with patches"-people to use it.

Re:The real problem

[Molina the Bofh]

I know system admins (if they can be called such) that don't know how to patch. Granted, it's not an intuitive process.

Also. some are not updating from the last kernel, wich requires more than 1 patch.

I slso believe that such tool, that downloaded as many patches as needed, should be explained and incentivated in the kerne's site motd. If they don't show it on the front page, and say it's an advantage to the user, then few people are going to get it.

Re:The real problem

[Quixote]

ISC is throwing away most of $80,000 annually because people can't be bothered to patch their kernel, and instead rely on downloading the full 20MB tarball every time a new kernel is released.

Another thing: when I download the kernel (as an end-user), why should I have to download Sparc, MIPS, IA64, PPC, etc. sources when all I need is x86? Maybe the kernel sources can be broken apart into individual architectures for the end users (obviously not for the kernel hackers).
Just did a quick check on my 2.4.17-xfs. The "arch" directory, compressed, takes 5.1MB. But the i386 subdirectory takes just 400KB (all figures with tar | gzip -9). I see a potential savings of 4.5MB right here.

Re:The real problem

[NearlyHeadless]

Of course, it goes without saying that everyone should still use their local mirror, particularly as kernel.org will only be accessible to mirrors for the forseeable future

I've thought a bit about this. It seems that every separate open-source project has to set up their own mirrors, there is no automatic system for finding "nearby" connections or for load-balancing, and volunteering for being a mirror can cause you to incur quite a bit of bandwidth costs.

I would be willing to pay a modest fee for downloads. I don't know if many of the other open-source fans (notorious cheapskates) would be willing, but if they were:

• A site like SourceForge could take the lead.
  
• Anybody who wants could have their software hosted there.
  
• Anybody who wants to would create an account to download.
  
• Charges would be proportional to bandwidth used, possibly off-peak times lower.
  
• SourceForge (or whoever) could subcontract connections to Akamai, Connxion, or others.
  
• Possibly a portion of the money collected could go to the creators of the software, the big problem being how to decide how the money gets divided for projects with many contributors or a lot of code re-use.
  
• Anybody could still distribute independently. It would not (could not) be an exclusive arrangement.
  

Re:The real problem

[nehril]

or they could provide patches only for the first few days after a release (forcing the rabid hordes to learn how to patch if they want the goods NOW) and then, at a random later time, post the full tarball. this might cause some percentage of ppl to get into the habit of patching, which should make a significant dent in their bandwidth needs.

Re:The real problem

[Webmonger]

BTW, the following message I just got back from www dot kernel dot org:

The Linux Kernel Archives is currently offline due to a hardware failure. However, mirror sites are receiving updates; please use a mirror site instead.

May be this is the beginning of the end to direct access to them?!

More likely, it's the problem referred to in the email.

Re:The real problem

[axioun]

I believe using mirror sites exclusively is the key, too. This could be done either with the users' knowledge or behind the scenes. The first method would probably be the quickest to implement at this time. It would involve something much like what Tucows has. Have a main site that links to mirrors based on region. This selector system might even temporarily not show links to sites that are currently experiencing high loads so people don't just pick the server on the top of the list and /. it.

This could probably be simplified by creating a small program that automates the server search process, and possibly download and apply patches on the fly. Later we might have a mirror system that is distributed. This latter stage could have a system like Freenet which is P2P or a load-balancing, centralized system.

Another method that just came to mind is to use a system similar to Audiogalaxy or Napster. While you've just downloaded a new kernel, your download software remains running. The software could have a default time to terminate. While the software is running, it acts as a small server, and the "Kernelgalaxy" software (what a fitting name) controls distribution in a distributed manner.

Hmm, I wonder who could help?

[mollusk]

If only there were some organization out there with a vested interest in linux. One that owes its existence to linux. Preferably one with a history of involvement in the linux community. Maybe even some corporation that runs it own websites dealing with open source issues. And while we're wishing, maybe even some entity with experience dealing with massive traffic requirements similar to the dreaded 'slashdot effect.'

Nah, nothing comes to mind. Shame.

Re:Q: Bandwidth composition

[gazbo]

Sounds like a bad idea to me. Centralised control is a good thing on occasion - at the moment we have kernel.org and a list of mirrors. I think this is a sweet spot between being too centralised (bandwidth problems) and too distributed (control problems)

So if we have a decentralised system (I assume you are inferring P2P style? If not, then surely the mirrors system is adequate?) how are we to stop people abusing the system? I could host a trojan version - where do people go to verify the MD5? How many people actually check MD5s? My point is that we need a trusted pool of servers; the truly paranoid can still check their MD5s against those at kernel.org, the rest of us can be assured that this is one of the trusted mirrors, and not a server owned by j03 133+ h4X0R.

Hey - my acct works again!

Why not IBM ?

[Krapangor]

They made of lot of claims that they support open source etc.

That is ALOT of bandwidth...

[A Commentor]

25 Mb/s = 3.125 MB/s = 187.5 MB/min = 11.25 GB/hr = 270 GB/day = 8.1 TB/month

Other solutions? Net-News?

[cpuffer_hammer]

Could some form of broadcast or streaming help?

What about Net-News it is an existing system that could distribute the patch to many of the people within a day.

The new kernel could be released,
mirrors and approved developers could have access to kernel.org for the first 3 days. Then only be patch downloads from kernel.org for the next 4 days.

BUT through net-news and most people would have it in a day.

There are lot's of ways to do this...

[macemoneta]

Asking for a big chunk of bandwidth and centralized management is the problem. It's expensive. Instead:

- Use the existing file sharing networks

- Netnews (I can get the file faster from my ISP's news server than anywhere else), and software like pan makes getting all the pieces trivial.

- Are there any open file sharing projects that we could use? Something that limited to a single download per user wouldn't be onerous. There are lots of cable/DSL linux users.

Suggestions for kernel.org

[sstamps]

1) Only allow access by mirrors and those ACTUALLY working on the kernel (ie, the kernel maintainers).
2) Get more mirrors. We're talking like several thousand here. As an ISP, I know I would not mind hosting a mirror, but I cannot afford $25,000/month in bandwidth. Splitting up the load using a large number of mirrors would make it MUCH cheaper to mirror the kernel files.
3) Use a highly-efficient load-sharing/balancing mechanism to direct people to mirror sites. Make it so the user can browse/select the files from the main kernel.org site, but the downloads are redirected from there to the mirrors.
4) Use a better patch process to reduce the size of the average download: 1) The x.x.0 release is the only full download, 2) use a patch system that downloads all the necessary updates, applies them to the x.x.0 version (or whatever the version the user already has) to get the latest version, and 3) MD5 checksums EVERY file to verify that it was patched correctly.

Take a line from FreeBSD

[Dom2]

In FreeBSD, CVSup is used to keep source trees in sync. It's a very efficient way of keeping several hundred megs of source code up to date.

I realise that CVSup is oriented towards CVS trees, which the Linux kernel isn't, but even an rsync server would be better than continuously downloading the patch.

The reason I mention this is because of the support infrastructure available in FreeBSD:

1. Install cvsup (once)
2. edit /etc/make.conf (once)
3. cd /usr/src && make update

CVSup is available at http://www.polstra.com/projects/freeware/CVSup/

Here's your answer, Peter..

[Bowie J. Poag]

If iBiblio is willing to host Propaganda, i'm sure they're more than willing to host a kernel.org mirror. In my experience, they've been wonderfully good hosts and run a very professional operation. Better still, they aren't hiding alterior motives by hosting free software projects, unlike the two-letter chameleon we've all grown to hate over the past year or two.

As for SourceForge, I wouldnt bother..The company that runs it turned its back on the community that made it's existance even possible. That alone should dissuade anyone. More tangible perhaps would be that the company has only one product (which they cant sell), and only enough cash on hand to last another year at most.

Cheers,

P2P

[burris]

The Linux Kernel and other open source projects should use some of the up and coming peer-to-peer distribution technology to host files. Tools like BitTorrent use the bandwidth of the current downloaders to relieve pressure on the main publishers. DL'ers get pieces of the file in random order and automatically exchange pieces with each other. From the users perspective, they just clicked a link. This technology desperately needs to be used by the Kernel archives, Debian, RedCarpet, etc...

Burris

The Quasi-Ultimate OSS Mirror for Canadians

[Sentry21]

I would suggest Canadians start using the Canadian Communications Research Centre's servers. They do have the bandwidth, especially to University students (CA*Net III and other academic/research networks) who are probably a large amount of users of the Linux kernel.

Incidentally, just some of the files available via rsync from ftp.crc.ca (which, sadly, has an anon-ftp limit of 25 users):

Perl CPAN mirror
GNOME desktop and utilities
Linux HowTo's
KDE desktop and utilities
XFree86
ALSA Linux sound drivers
Debian Linux
Debian Linux ISO images
FreeBSD
Alexy Kuznetsov's IP Routing Tools for linux
Blackdown's port of JAVA for Linux
CRC's Linux Kernel Archive (I wonder if this is different from the standard kernel? they don't say "CRC's" on everything)
CRC's RedHat mirror
CRC's RedHat Contrib (interesting)
Slackware Linux
SUSE Linux
TurboLinux
CRC's VQEG Digital Video Experiments
CRC's XAnim mirror

So if you are Canadian and use any of these software packages (or the others on the page I linked), PLEASE use this site, it's extremely fast on broadband and even more so to university students. I used it for my Debian packages until they dropped the limit on FTP users. Maybe if I ask real nice they'll give me a login....

The site itself is interesting too. Neat stuff.

--Dan