Slashdot Mirror

What I'm really hoping for is that they'd put up compressed archives of their Web history, or perhaps X most popular sites, segmented by domain name. With solid compression across many raw dumps of similar HTML pages (especially different versions of the same page), that could save >99% of bandwidth compared to a scraper script that loops through "http://wayback.archive.org/web/*/$domain/*" ... That would encourage many people to download their favorite sites to burn on DVD, and perhaps client-side software (ex. cache proxies, "timegate" browser plug-ins, personal search engines, etc) could utilize this data as well. It would also add much decentralization and redundancy, if the goal is to ensure data proliferation in case of a cataclysmic event or government censorship of the Internet.

On a less significant note, I find it interesting that they're not running any BT-specific seeding software on their servers (ex. hekate), relying on Web seeds instead. I wonder what exactly the downsides are: limited BT client support, magnet URL's, etc. It's also interesting that they're not using Metalink.

check out metalink. most linux distributions use it for large downloads and software updates.

I would like a solution that combines metalinks (one file that contains multiple urls for a download plus the checksum) with Bittorrent.
A client could start a http download from one server, and a bittorrent that requests pieces for the latter chunks. You can also make multiple http request with a offset these days, on another http server or the same one.

This could even be built in magically into http browsers: if the file size is > 50MB, ask the cloud if there are nodes for the given url. That is provided you have a checksum like with metalinks. Appearantly metalink already features this possibility: http://www.metalinker.org/

it would be cool if they could use metalink, an internet standard for describing files offered in hybrid ftp/http/p2p content distribution systems, already used by a lot of open source projects.

You can use Metalinks for downloads without errors. It can use whole file checksums, or partial file checksums to repair errors.

Not everyone can use torrents, so try out the metalinks. These are XML files that list mirrors & checksums, helping you find a server, and verifying that the download didn't have errors.

You can use DownThemAll! (Firefox extension), KGet in KDE4, GGet in GNOME, aria2, or metalink-checker (among many other Windows/OS X/Linux download clients).

The official .metalink files are available at
http://releases.ubuntu.com/releases/8.10/
http://releases.ubuntu.com/releases/kubuntu/intrepid/

Not everyone can use torrents, so try out the metalinks. These are XML files that list mirrors & checksums, helping you find a server, and verifying that the download didn't have errors.

You can use DownThemAll! (Firefox extension), KGet in KDE4, GGet in GNOME, aria2, or metalink-checker (among many other Windows/OS X/Linux download clients).

The official .metalink files are available at
http://releases.ubuntu.com/releases/8.10/
http://releases.ubuntu.com/releases/kubuntu/intrepid/

A few years ago, I wrote some software for patching and updating a rather large software installation on multiple clients. Even on a LAN, we saw approximately 1 bit error for every 4GB of data transfered over raw TCP/IP. Errors happen.

True reliability over TCP requires strong checksums on top of the weak error-correction provided by the protocol. The bottom line is that HTTP and FTP aren't really suited for transferring more than a few megabytes of data without assistance.

But there is a standard for solving this problem: Metalinker. This defines an XML-based standard for block checksums and multiple sources of a source file. Now if people would actually use it to distribute all their large files...

Besides rsync & torrents, you can also repair files with metalinks, which require nothing extra on the server, and is not blocked like p2p in some places.

This is why so many distributions use them for ISO downloads, so you don't have to restart large downloads from the beginning.

Use the metalinks for the speediest possible download experience. All you need is one of the many available metalink compatible download clients.

http://releases.ubuntu.com/8.04/ubuntu-8.04-alternate-amd64.metalink
http://releases.ubuntu.com/8.04/ubuntu-8.04-alternate-i386.metalink
http://releases.ubuntu.com/8.04/ubuntu-8.04-desktop-amd64.metalink
http://releases.ubuntu.com/8.04/ubuntu-8.04-desktop-i386.metalink

Use the metalinks for the speediest possible download experience. All you need is one of the many available metalink compatible download clients.

http://releases.ubuntu.com/8.04/ubuntu-8.04-alternate-amd64.metalink
http://releases.ubuntu.com/8.04/ubuntu-8.04-alternate-i386.metalink
http://releases.ubuntu.com/8.04/ubuntu-8.04-desktop-amd64.metalink
http://releases.ubuntu.com/8.04/ubuntu-8.04-desktop-i386.metalink

Direct links to the metalinks (As Seen on /.) - these list all the mirrors, checksums, and other info and are used by download apps.

To use them, try DownThemAll! (Firefox extension), aria2 (cross platform, command line), KGet (in KDE4), Speed Download (OS X), GetRight (Windows), Retriever (cross platform, Java) or about 15 other download programs.

http://releases.ubuntu.com/8.04/ubuntu-8.04-alternate-amd64.metalink
http://releases.ubuntu.com/8.04/ubuntu-8.04-alternate-i386.metalink
http://releases.ubuntu.com/8.04/ubuntu-8.04-desktop-amd64.metalink
http://releases.ubuntu.com/8.04/ubuntu-8.04-desktop-i386.metalink

Sounds a lot like Metalink, covered here: http://slashdot.org/article.pl?sid=07/02/25/144209

Some of these issues seem to be addressed (or are being attempted, at least in the early stages) by metalink which was discussed at http://slashdot.org/article.pl?sid=07/02/25/144209 a few months ago, but I don't think people really understood what it was.

Great question! aria2 is the only client (so far, others are in progress) to support segment hashes (aka chunk checksums/repair information) directly in the metalink. It just came out recently. For an example, check out the opensuse metalinks. I think this is one of the most important parts about metalink, files are verified during transfer (and corrupted files are fixed) over plain ftp/http with no server/tracker changes.

http://www.metalinker.org/Metalink_3.0_Spec.odt

http://www.metalinker.org/Metalink_3.0_Spec.pdf

http://www.metalinker.org/Metalink_3.0_Spec.odt

http://www.metalinker.org/Metalink_3.0_Spec.pdf