Metalinks Tries to Simplify Downloads

ant_tmwx writes "Metalinks collect information about files in an XML format used by programs that download. The information includes mirror lists, ways to retrieve the file on P2P networks, checksums for verifying and correcting downloads, operating system, language, and other details. Using Metalinks details the Free Software programs you can use to download them with. There are also clients on Mac and Windows. With a list of multiple ways to download a file, programs can switch to another method if one goes down. Or a file can be downloaded from multiple mirrors at once, usually making the download go much faster. Downloads can be repaired during transfer to guarantee no errors. All this makes things automatic which are usually not possible or at least difficult, and increases efficiency, availability, and reliability over regular download links. OpenOffice.org, openSUSE, and other Linux/BSD distributions use them for large downloads."

Comments

Browser clients available?

[MMC Monster]

Are there clients that integrate (ie: extensions) for Firefox, IE, Safari, and Opera? If there is proper integration with these clients (meaning seamless downloading without opening third party download managers), this might actually go well.

It's bad enough when I tell my dad to download a torrent and he complains that a torrent manager client pops up; especially when he doesn't realize that closing the window may not stop the torrent.

Re:Browser clients available?

[cronius]

From http://www.metalinker.org/implementation.html:

Download Metalinks with these programs...
GetRight (Windows) is a Download Manager that supports Metalink.
FlashGot (Cross platform, Open Source) is a Firefox extension that integrates with around 30 Download Managers and supports Metalink with wxDownload Fast, Speed Download, and GetRight 5.2d and later.
aria2 (Unix/Windows, Open Source) is a high speed download utility that supports segmented downloads, BitTorrent, and Metalink (HTTP/FTP/BitTorrent integrated) from the command line.
Speed Download (Mac) is a slick Download Manager with fast downloads & P2P filesharing. It integrates with popular Mac browsers like Safari, Camino, & Firefox (along with 5 others).
wxDownload Fast (Mac/Unix/Windows, Open Source) supports Metalink.
Free Download Manager (Windows) unreleased BETA supports Metalink.
Orbit Downloader (Windows) is a new download manager with interesting P2P features.
SmartFTP (Windows), an excellent FTP client on Windows, supports Metalink for adding files to a transfer queue and checksum verification (no acceleration).
Phex (Mac/Unix/Windows) is a Gnutella P2P client that can export Metalinks.

...

If you like Metalink, request support in these clients on their forums or bug trackers:

CuteFTP is a Windows GUI FTP client.
Bouncer (Open Source) powers downloads for various projects like Mozilla. A patch for Metalink generation has been submitted by Bram Neijt but not yet integrated.
cURL (Cross platform, Open Source) an interface for libcurl would be cool. Anyone up for writing one?
Shareaza (Open Source) would be great as well.
Opera is a great standards compliant browser that's always adding new features. Current threads: 1, 2, 3
Firefox (Open Source) is pretty neat too. So looks like we need to wait alittle.

Needless extra step?

[grasshoppa]

From their page:

Why should you use it?

Users
Your downloads will be simpler, faster, and more reliable...without you doing anything differently.

Bittorrent already does this just about as effectively as this idea will.

Developers
It's a neutral framework that doesn't favor any one program, Operating system, or group, and is easy to implement.

Once again, bittorrent is just as easy. And its OS agnostic.

Site owners
Resume and recover from single servers going down.

Sorta an issue with bittorrent, but not really. House the seed in multiple locations. Or better yet, have your clients take a copy of the seed and share that with their peers in the case of a downed server.

Downloads can automatically be split between sources (mirrors/P2P) and all downloads will be verified.
More people can get access to your files easier, more reliably, even at the most heavily accessed times.
This means less retries and cheaper bandwidth and support bills. Saving money = good.

Once again, this is where bittorrent shines. A lot of people going after your files? Great, that means it's got a better availability on your torrent, more bandwidth for everyone.

To me, this looks like a solution in search of a problem.

Re:Needless extra step?

[mikkelm]

I don't know about you, but given adequate capacity, I prefer to get my files via FTP or HTTP. Bittorrent is fundamentally unreliable when it comes to download speed and susceptible to packet shaping and the likes. In order to slow this down, theoretically, assuming that all paths are available you'd have to throttle the native P2P as well as FTP -and- HTTP, and I don't know of many ISPs that throttle even FTP, let alone HTTP.

Bittorrent is neat and handy if you don't have the capacity or balls to exclusively host the content centrally. This is not at all Bittorrent, in that it seems to primarily target content already hosted centrally, and allow it the flexibility to spread the content across several geographically seperated sites without the need for a list of localised links, while also providing alternative protocol and multiple-host download methods, as opposed to just using a multitude of random users whom you have no control over to spread your content.

Re:Needless extra step?

[grumbel]

### Bittorrent already does this just about as effectively as this idea will.

If the tracker goes down or there happens to be a lack of seeds the link is dead. With Metalink on the other side a client could automatically use *all* ways to get a file, not just a single tracker or server, but multiple http servers, P2P networks and torrent all at the same time, if one goes down there might still be plenty of others left.

### Once again, bittorrent is just as easy. And its OS agnostic.

But not protocol or server agnostic.

### To me, this looks like a solution in search of a problem.

Ever tried to download a file from Sourceforge or any other server with a dozens of mirrors which you have to manually select? That is exactly the problem that Metalink solves, its a standard way to show where the mirrors are, completly independent of the protocol in use. Thus it allows the client to automatically select them or use them all at the same time for faster download, no more stupid manual mirror selection just to find out that the host is down or slow as hell.

Metalink doesn't try to replace bittorrent, quite the opposite, it tries to provide a way to simply bundle all links that lead to the same file, torrent included.

Re:Needless extra step?

[TheRaven64]

BitTorrent has two major disadvantages:

1. It is fundamentally inefficient. The overall overhead of using BitTorrent on the Internet is greater than direct downloads.[1]
2. It uses the most expensive bandwidth for distribution. Consumer upload bandwidth is the most scarce and about the most expensive bandwidth you can buy

The only advantage of BitTorrent is that you (the content provider) aren't paying for as much of the bandwidth. On the other hand, bandwidth bought in bulk is so cheap that it's really not worth if for most people.

This system, however, would allow me to easily download automatically from the mirror closest to me. It would also be pretty easy using a little ECMAScript to grab the correct mirror using the browser's locale and present a direct download link.

[1] Actually, it's a pretty horrible protocol. It doesn't do anything with location information, so it adds a lot of extra load to the backbones. It uses TCP, so it will never support multicast without a fundamental redesign, and the algorithms for rate distribution have some fundamental flaws.

Re:Who says it can't use BitTorrent also?

[Jonah Hex]

It does not look like it excludes ANY type of file transfer, if your client supports it you can do it is how it looks to me.

Example - MetaLink XML contains the following formats:
5 different HTTP sites
2 FTP sites
3 BitTorrent Trackers
eMule/Edonkey Hash

Example - Client One has implemented:
HTTP, FTP and BitTorrent

Example - Client Two has implemented:
BitTorrent and eMule

Example - Client Three has implemented:
HTTP, FTP, BitTorrent and eMule

I'm surprised it's taken this long to come up with this sort of client independant format.

Jonah HEX

good idea - can be improved

[hey]

This is a good idea. It attempts to formalize something thats been done many times before. We do it manually when download from Sourceforge, Yum has a list of mirrors it does automatically. A standard would be nice. I would like to see a new web protocol for it - ie:

        metalink://host.com/file.ml

Then inside file.ml simply a list of URLs and weights...

        ftp://host1.com/file.rpm 10
        http://host2.com/file.rpm 10
        torrent://host3.com/file.rpm 20
        etc

XML doesn't help.

Your comment doesn't sound right to me...

[__aailob1448]

If it's a pretty horrible protocol, why has it become so popular? Is it possible that all or some of those design "flaws" you mentionned are necessary for it to do what it does? And if not, why have you not created your own, awesomer fileswarming protocol? In fact, why has nobody else?

And I'm really asking here. Not just disguising my attacks as questions.