Metalinks Tries to Simplify Downloads

ant_tmwx writes "Metalinks collect information about files in an XML format used by programs that download. The information includes mirror lists, ways to retrieve the file on P2P networks, checksums for verifying and correcting downloads, operating system, language, and other details. Using Metalinks details the Free Software programs you can use to download them with. There are also clients on Mac and Windows. With a list of multiple ways to download a file, programs can switch to another method if one goes down. Or a file can be downloaded from multiple mirrors at once, usually making the download go much faster. Downloads can be repaired during transfer to guarantee no errors. All this makes things automatic which are usually not possible or at least difficult, and increases efficiency, availability, and reliability over regular download links. OpenOffice.org, openSUSE, and other Linux/BSD distributions use them for large downloads."

Comments

Mirror?

[Lord of Ironhand]

http://www.exampleURL.com/file.metalink

Or, depending on your location, use one of these mirrors:
http://uk.exampleURL.com/file.metalink
http://nl.exampleURL.com/file.metalink
http://de.exampleURL.com/file.metalink

Seriously though, I like the basic idea, but the system does add an extra point of failure.

Re:Mirror?

[Anonymous Coward]

>
> [...] I like the basic idea, but the system does add an extra point of failure.
>

Avoiding problems with dead servers is not the only use of mirrors. Metalink is more used to group all informations to get a file, reliably (hash checks), using your preferred protocol (and, possibly, multiple protocols at once), and as fast as possible (by downloading little, from a lot of places, rather than a lot, from a single place).

Re:Mirror?

[grumbel]

### Seriously though, I like the basic idea, but the system does add an extra point of failure.

Does it? Its not totally fool proof since the original server hosting the link file could still go down, but thats not an extra point of failure, since its no different then the server going down when trying to download the real file from it in the first place. Once you have the Metalink file everything should be fine.

Browser clients available?

[MMC Monster]

Are there clients that integrate (ie: extensions) for Firefox, IE, Safari, and Opera? If there is proper integration with these clients (meaning seamless downloading without opening third party download managers), this might actually go well.

It's bad enough when I tell my dad to download a torrent and he complains that a torrent manager client pops up; especially when he doesn't realize that closing the window may not stop the torrent.

Re:Browser clients available?

[cronius]

From http://www.metalinker.org/implementation.html:

Download Metalinks with these programs...
GetRight (Windows) is a Download Manager that supports Metalink.
FlashGot (Cross platform, Open Source) is a Firefox extension that integrates with around 30 Download Managers and supports Metalink with wxDownload Fast, Speed Download, and GetRight 5.2d and later.
aria2 (Unix/Windows, Open Source) is a high speed download utility that supports segmented downloads, BitTorrent, and Metalink (HTTP/FTP/BitTorrent integrated) from the command line.
Speed Download (Mac) is a slick Download Manager with fast downloads & P2P filesharing. It integrates with popular Mac browsers like Safari, Camino, & Firefox (along with 5 others).
wxDownload Fast (Mac/Unix/Windows, Open Source) supports Metalink.
Free Download Manager (Windows) unreleased BETA supports Metalink.
Orbit Downloader (Windows) is a new download manager with interesting P2P features.
SmartFTP (Windows), an excellent FTP client on Windows, supports Metalink for adding files to a transfer queue and checksum verification (no acceleration).
Phex (Mac/Unix/Windows) is a Gnutella P2P client that can export Metalinks.

...

If you like Metalink, request support in these clients on their forums or bug trackers:

CuteFTP is a Windows GUI FTP client.
Bouncer (Open Source) powers downloads for various projects like Mozilla. A patch for Metalink generation has been submitted by Bram Neijt but not yet integrated.
cURL (Cross platform, Open Source) an interface for libcurl would be cool. Anyone up for writing one?
Shareaza (Open Source) would be great as well.
Opera is a great standards compliant browser that's always adding new features. Current threads: 1, 2, 3
Firefox (Open Source) is pretty neat too. So looks like we need to wait alittle.

Re:Browser clients available?

[Jonah Hex]

I've used both of these, and they are free
# Free Download Manager (Windows) unreleased BETA supports Metalink.
# Orbit Downloader (Windows) is a new download manager with interesting P2P features.
Jonah HEX

Re:Browser clients available?

[ant_tmwx]

DownThemAll a popular Firefox download manager extension has basic support in development.

KGet for KDE4 (their download manager) also has just added initial support.

Both are free open source projects, so help out if you're able.

Re:Browser clients available?

[shakayuu]

The Orbit downloader is better. And here is a step-by-step tutorial for downloading metalink:http://www.orbitdownloader.com/Use-as-Met alink-Downloader.htm/

Gee willakers!

[tomstdenis]

You can do that with computers? Honestly, how hard is it to pick your OS out of a list of download links?

I don't think it's a bad idea, I just don't get why it's on /.

Or in the case of us gentoo folk, just build the damn program and eliminate any doubt :-)

Tom

Re:Gee willakers!

[jZnat]

You know, it might be trivial to change emerge's use of wget to use an application that also supports metalinks (or a wrapper to support wget and a metalinks program) which could be useful...

Needless extra step?

[grasshoppa]

From their page:

Why should you use it?

Users
Your downloads will be simpler, faster, and more reliable...without you doing anything differently.

Bittorrent already does this just about as effectively as this idea will.

Developers
It's a neutral framework that doesn't favor any one program, Operating system, or group, and is easy to implement.

Once again, bittorrent is just as easy. And its OS agnostic.

Site owners
Resume and recover from single servers going down.

Sorta an issue with bittorrent, but not really. House the seed in multiple locations. Or better yet, have your clients take a copy of the seed and share that with their peers in the case of a downed server.

Downloads can automatically be split between sources (mirrors/P2P) and all downloads will be verified.
More people can get access to your files easier, more reliably, even at the most heavily accessed times.
This means less retries and cheaper bandwidth and support bills. Saving money = good.

Once again, this is where bittorrent shines. A lot of people going after your files? Great, that means it's got a better availability on your torrent, more bandwidth for everyone.

To me, this looks like a solution in search of a problem.

Re:Needless extra step?

[mikkelm]

I don't know about you, but given adequate capacity, I prefer to get my files via FTP or HTTP. Bittorrent is fundamentally unreliable when it comes to download speed and susceptible to packet shaping and the likes. In order to slow this down, theoretically, assuming that all paths are available you'd have to throttle the native P2P as well as FTP -and- HTTP, and I don't know of many ISPs that throttle even FTP, let alone HTTP.

Bittorrent is neat and handy if you don't have the capacity or balls to exclusively host the content centrally. This is not at all Bittorrent, in that it seems to primarily target content already hosted centrally, and allow it the flexibility to spread the content across several geographically seperated sites without the need for a list of localised links, while also providing alternative protocol and multiple-host download methods, as opposed to just using a multitude of random users whom you have no control over to spread your content.

Re:Needless extra step?

[grumbel]

### Bittorrent already does this just about as effectively as this idea will.

If the tracker goes down or there happens to be a lack of seeds the link is dead. With Metalink on the other side a client could automatically use *all* ways to get a file, not just a single tracker or server, but multiple http servers, P2P networks and torrent all at the same time, if one goes down there might still be plenty of others left.

### Once again, bittorrent is just as easy. And its OS agnostic.

But not protocol or server agnostic.

### To me, this looks like a solution in search of a problem.

Ever tried to download a file from Sourceforge or any other server with a dozens of mirrors which you have to manually select? That is exactly the problem that Metalink solves, its a standard way to show where the mirrors are, completly independent of the protocol in use. Thus it allows the client to automatically select them or use them all at the same time for faster download, no more stupid manual mirror selection just to find out that the host is down or slow as hell.

Metalink doesn't try to replace bittorrent, quite the opposite, it tries to provide a way to simply bundle all links that lead to the same file, torrent included.

Re:Needless extra step?

[Chris_Jefferson]

The advantage I hope this might provide over bittorrent is making it completely invisable to the user. For a while I've wanted a bittorrent client that was completely invisable to the user. It just goes and downloads and closes itself automatically after it is finished and the ratio reaches 1.0, or the user moves the file. I find explaning bittorrent to my parents annoying, and while they've got it now, it would have been nice if firefox could have simply done it completely invisably from them.

Power users will still want to seed extra, see ratios, check their firewall ports are open, etc. but this kind of completely invsible setup could be what really gets bittorrent onto every website and into every home.

Re:Needless extra step?

[TheRaven64]

BitTorrent has two major disadvantages:

1. It is fundamentally inefficient. The overall overhead of using BitTorrent on the Internet is greater than direct downloads.[1]
2. It uses the most expensive bandwidth for distribution. Consumer upload bandwidth is the most scarce and about the most expensive bandwidth you can buy

The only advantage of BitTorrent is that you (the content provider) aren't paying for as much of the bandwidth. On the other hand, bandwidth bought in bulk is so cheap that it's really not worth if for most people.

This system, however, would allow me to easily download automatically from the mirror closest to me. It would also be pretty easy using a little ECMAScript to grab the correct mirror using the browser's locale and present a direct download link.

[1] Actually, it's a pretty horrible protocol. It doesn't do anything with location information, so it adds a lot of extra load to the backbones. It uses TCP, so it will never support multicast without a fundamental redesign, and the algorithms for rate distribution have some fundamental flaws.

Re:Needless extra step?

[imboboage0]

You hold good points, but then explain to me why downloading a Linux distro (take SLAX for example) is faster over Bittorrent than FTP or HTTP?

Re:Needless extra step?

[mikkelm]

Most likely because the source you're doing your FTP/HTTP downloads from isn't fast enough. Both FTP and HTTP are undeniably more efficient than Bittorrent, even in the unlikely event that the torrent you're leeching from is maxing your connection.

Re:Needless extra step?

[ant_tmwx]

SLAX apparently does not have very fast, or very many, mirrors. Try downloading an ISO from a distribution that has a robust mirror network. If your connection speed is high enough, you'll see a difference.

Re:Needless extra step?

[anethema]

Actually, the torrents you're referring to are old hat.

With DHT and multiple trackers a torrent is pretty much unkillable. Even if you have a lot of peers and no seeds, you usually have the entire file distributed throughout the peers and more seeds will be created.

For example on isohunt.com each torrent has maybe 5 (plus or minus) trackers associated with it. And DHT ensures that even if ALL of the trackers go down, peers will continue to find and download from each other.

It is a fairly robust download method, compared to something like http which relies on one single server or you're hooped.

That being said, more is better, and there is no reason not to augment torrent traffic with http/ftp, etc.

Re:Needless extra step?

[irc.goatse.cx troll]

How come downloading a WoW patch over the built in bittorrent client is slower than loading up firefox and searching for the same patch on a traditional download site, even including the manual searching for the file?

It also depends on your definition of "fast" really. If the files ETA is more than 15 minutes I'm usually going to spend at least the first 5 looking for a better way to download it.

Re:Needless extra step?

[Ilgaz]

Bittorrent can never shine while their official site (bittorrent.com) pushes piracy from main page. It is a great idea but wasted to pirates and thieves in majority.

Not news

[sco08y]

If this standard is accepted by IE and Firefox, *then* it will be news. Until then it's just buzzword compliant crap.

Re:Not news

[mgblst]

No, until then it is an idea, a good idea. If there are problems with it, we have the chance to point them out, before we start convincing the big browsers to adopt this?

I can't see how this is a bad idea, it seems like great to me. Sure bittorrent has its place, but so does this.

And what is wrong with you? Why can't we talk about this idea? Maybe you are on the wrong site, fox news is maybe more what you want.

YUM, Sourceforge, kernel.org, ...

[r6144]

It will be great for sites that have a LOT of mirrors. Particularly sourceforge, of which no single mirror is reliably "fast enough" for me.

Re:YUM, Sourceforge, kernel.org, ...

[jZnat]

Ever tried using the Easynews mirror? Easynews is a dedicated Usenet [a.b] provider with extremely fat tubes, so I think you should get a good speed from them. I typically do...

Re:Who says it can't use BitTorrent also?

[Jonah Hex]

It does not look like it excludes ANY type of file transfer, if your client supports it you can do it is how it looks to me.

Example - MetaLink XML contains the following formats:
5 different HTTP sites
2 FTP sites
3 BitTorrent Trackers
eMule/Edonkey Hash

Example - Client One has implemented:
HTTP, FTP and BitTorrent

Example - Client Two has implemented:
BitTorrent and eMule

Example - Client Three has implemented:
HTTP, FTP, BitTorrent and eMule

I'm surprised it's taken this long to come up with this sort of client independant format.

Jonah HEX

Mac client confusion and Open Standard

[Lord Satri]

Strangely enough, the OpenOffice distribution page links to a pay-for Metalink client for the Mac, but hopefully, just two links above, there's the cross-platform open source client. That's confusing. I clicked on the Mac client, thinking I would end up at a Mac-oriented free (even open source?) client, but no, one must choose the cross-platform. Nothing really wrong there, only that it's confusing.

Unrelated, I saw numerous attempts of such integrated p2p downloads. The part that got me from the Metalink main page: "Metalink is an Open Standard". This makes me believe I will join the bandwagon. And yup, the Wikipedia Metalink page is (surprisingly?) informative.

Mac client confusion: ok, I get it now...

[Lord Satri]

Ok. I just had to follow some more links to realize that the open source cross-platform alternative doesn't offer MacOS X binaries yet, one has to compile from source. That's a showstopper for most Mac users... well, with Metalink getting more attention, I'm not worried, mature clients will come soon.

Location embedding?

[Lord Satri]

I tried to find additional info about location embedding but haven't succeeded so far. From the wikipedia page, the XML code includes url type="http" location="uk" preference="90">http://www.example2.com/example.ex t /url, the part I don't get, as a geospatial professional (see sig), is why the location is encoded as the country code. What are the reasons? Does it make more sense to encode location with a simple lat-lon values (similar to, say, the georss standard do). Some countries being so large, I fail to see the country as a good indicator of distance between computers. What did I got wrong?

Re:Location embedding?

[TheRaven64]

What did I got wrong? You are not thinking in terms of cost. When you buy capacity in bulk, you might get a flat rate, but you are more likely to get one rate for national and one rate for international traffic. In the UK, I believe it's fairly common to get a more expensive rate for transatlantic traffic. If I'm hosting a mirror in the UK, then it will cost me more if Americans use it than people in France. Any finer-grained information doesn't really matter.

Of course, with IPv6 we ought to be able to get rid of this kind of thing, since IPv6 addresses are supposed to encode topology, so you just find the address with the longest prefix match to you and use that.

Re:Location embedding?

[sveinb]

Had the internet had the topology of a mesh, then you would be right. However, the internet has a hierarchical structure where things like cities and nations frequently show up as hubs. As an example: The number of hops from my computer in Oslo to a server in Tromsø (same nation) is most likely smaller than the number of hops to a server in Gothenburg (different nation), even though the geographical distance is shorter to Gothenburg.

Specifications?

[The Real Toad King]

I've looked around the site, and I've found no document for the specification of a metalink file. IMO, this will easily lead to many conflicts with different clients each having their own version of "Metalinks".

Re:Specifications?

[Anonymous Coward]

http://www.metalinker.org/Metalink_3.0_Spec.odt

http://www.metalinker.org/Metalink_3.0_Spec.pdf

Re:Specifications?

[The Real Toad King]

Thank you, but where did you find these on the site? Also, what's with people using PDF files for stuff that could easily go into HTML files?

good idea - can be improved

[hey]

This is a good idea. It attempts to formalize something thats been done many times before. We do it manually when download from Sourceforge, Yum has a list of mirrors it does automatically. A standard would be nice. I would like to see a new web protocol for it - ie:

        metalink://host.com/file.ml

Then inside file.ml simply a list of URLs and weights...

        ftp://host1.com/file.rpm 10
        http://host2.com/file.rpm 10
        torrent://host3.com/file.rpm 20
        etc

XML doesn't help.

Re:good idea - can be improved

[Kjella]

Well, it's certainly missing localization. Now if you've coded up that spec and got that implemented, good luck changing it. I'm quite partial to XML for one thing - you can add tags and have backwards compatibility by ignoring unknown tags. What if you want to add say a standard location for the PGP/GPG signature? Maybe you want the certs to be part of the ML file? What if you want to specify that some sites have access restrictions, like for example the Mandriva club? Sure, you could get an access denied, but maybe someone wants to build a more advanced integration with password management. I'm not saying all of these are great ideas, but well... it just seems horribly limited to restrict ourselves to one set of weigths.

Call me sceptical

[rudegeek]

But "simplify" and "XML format" in once sentence does not always "return true;". If the number of information stored in XML will grow how much CPU time and storage it will require? Wouldn't it be better to get in into database and provide XML based API? Querying XML is a bit slower than asking any DB. Even SQLite.

Re:Call me sceptical

[Jesus_666]

The important part is not how stuff is stored on the server - in fact, that's completely irrelevant. Relevant is only what the client is presented. XML is rather good at storing structured data and most browsers can parse it already. OTOH packing everything into small SQLite database files and sending those to people would not only require every client to link against SQLite but also make it impossible for people to just open the file in a text editor and look at what's inside. Of course you could use direct SQL queries, but then all client would have to support all database softwares and you would give people direct access to your database, which is unwise.

If you don't want to store metalink files on your sever nobody forces you to; you can always generate the metalink file out of whatever your backend uses. Metalinks are the XML-based API you asked for.

Metalinks Tries to Simplify Downloads?

[rueger]

Well, the summary makes it sound divine - one link, one bit of software that accesses P2P, FTP etc interchangeably to maximize download speed.

That seems like a logical growth of Bittorrent.

Trying to figure out exactly what is needed though was another matter. After a half hour and three or four web sites I wound up with the wxDownload Fast Windows download manager and a Metamirrors Firefox plugin.

Is it all working as advertised? Well, stuff is downloading (OpenSUSE 10.2) but I have no idea of it's faster, or even if it's also uploading in P2P fashion.

For God's sake, is it too much to ask that the people behind stuff like this include a simple checklist?

To download using Metalinks you needs
a) A download client (here are links to a few),
b) This Firefox plugin and
c) then do THIS.

Re: Metalinks Tries to Simplify Downloads?

[eddy]

Not sure I understand what kind of checklist you want. That a client is needed ought to be fairly obvious. Your "then do THIS" depends entirely on the client.

You can 'apt-get install aria2' for a cmdline utility in the style of wget, supporting metalinks.

Your comment doesn't sound right to me...

[__aailob1448]

If it's a pretty horrible protocol, why has it become so popular? Is it possible that all or some of those design "flaws" you mentionned are necessary for it to do what it does? And if not, why have you not created your own, awesomer fileswarming protocol? In fact, why has nobody else?

And I'm really asking here. Not just disguising my attacks as questions.

Re:Your comment doesn't sound right to me...

[irc.goatse.cx troll]

Some of those flaws are needed to do what it does, others arn't major enough that anyone would switch -- If someone came out with a p2p app that corrected just those minor things people would say "who cares?" and stick to pirating things with their shiny copy of utorrent.

Really, I've always prefered downloads split between multiple fast mirrors(split http, FTP as done by the actual piracy scene) rather than multiple slow client peers (utorrent, kazaa, any other common 'end user' piracy). I hope this metalink thing takes off as its exactly what I wanted -- I've looked at download pages before where theres this long list of hosts and you're supposed to pick one and I just can't help but think..why only one? Why am I switching from host to host trying to find which one gives me more than 20K/s when I could just be hitting all 5 of these 20K/s hosts at once?

The fact that it uses xml as pointed out elsehwhere means its possible its more buzzwordy and complex than helpful, but the idea is simple enough that it might not be too bad.

Re:Why no segment hashes?

[ant_tmwx]

Great question! aria2 is the only client (so far, others are in progress) to support segment hashes (aka chunk checksums/repair information) directly in the metalink. It just came out recently. For an example, check out the opensuse metalinks. I think this is one of the most important parts about metalink, files are verified during transfer (and corrupted files are fixed) over plain ftp/http with no server/tracker changes.

Magnet has been pretty much doing this.

[WeblionX]

It seems like we might as well just use magnet links, since they can include different hash types of the file and different locations to download from. I already have them for certain larger files on my website and work fine if you have a client that supports it.

Re:Metalink repository?

[ant_tmwx]

Metalink @ Package Resources is the closest thing to a repository. Arch Linux, Centos, Edubuntu, Fedora, Linux Kernel, Kubuntu, OpenOffice.org, opensuse, and Ubuntu metalinks are at http://download.packages.ro/metalink/

Fuck BitTorrent.

[WilliamSChips]

If this application doesn't fuck up my router like BitTorrent does, I'm there.

Faster. Really?

[Nom du Keyboard]

Or a file can be downloaded from multiple mirrors at once, usually making the download go much faster.

M users simultaneously download from N servers individually. Or M users simultaneously download from N servers in parallel. Aside from some load balancing, can you really gain dramatically? The same number of bytes must be transfered from the same number of servers.

It exists !

[DrYak]

For a while I've wanted a bittorrent client that was completely invisable to the user.

And it exists. Either...

• ...as a small torrent client exe-file that is piggy-backed with the actual torrent file and started if the user has no client. Downhill battle's Blog Torrent has such an exemple, called "Easy Download". People just click on the link and the necessary client is automagically installed.
• ...as a on-the-fly P2P-izer, that just works by pre-pending a prefix to the address (just like the Coral cache). Differ is such a system. People put dijjer.org/get/ to the beginning of links. The website will automatically send an installer if needed. This installers set ups some sort of proxy on the users computer. dijjer.org/get URLs are redirected to this local proxy and the proxy it self is in chagre of geting the file and transparently make it available in HTTP as if downloaded by conventionnal means. The good thing is, once the proxy is installed it works with anything, browser, but also media players with HTTP web-streaming.
• ...last but not least, there are browser plugins. Opera, as far as I've heard comes with one, and I'm sure one can find something on http://addons.mozilla.org/

Re:Faster. Really?

[AxelBoldt]

can you really gain dramatically

Yes, in most cases. The bottleneck of a connection is usually near the downloader's end. If such a bottleneck segment is shared by several people, then downloading in parallel (from several servers or even from the same server) helps: the downloader will receive a bigger share of the bottleneck segment.

See here for a little experiment that illustrates the point and that you can reproduce easily.

Re:Faster. Really?

[IcePic]

And if you follow the replies on that link you get to see why server admins don't like clients that grab more resources than necessary.
It works as long as you are the only one doing it, but when everyone starts using segmented downloads, everyone loses. And you not only
go back to as it was before, you also manage to waste more than the same number of poeple would have before, if they would stick to using
one connection per download.

There has been discussions on the openoffice distribution lists about this too. Not everyone is convinced that having each client opens
loads and loads of connections per download is the desired solution in the end.

Metalinks are indeed awesome

[apokryphos]

These worked really great for the openSUSE release. Why is it good? It takes out the hassle of having to track down a working/fast mirror. Want to download a large DVD ISO *quickly*? Then this is the way. The small metalink file will have a populated list of around 50 mirrors. The client your using works with making multiple connections to all of these, so you will pretty much ALWAYS MAX OUT YOUR CONNECTION. And still get a really safe download. Why? Because it checks checksums on, IIRC, 4-megabyte chunks.

Reliable and fast, and it spreads out the weight between mirrors, AND it can work with torrents. Metalinks are indeed awesome. Wget and kget support for it is in the works, so stay tuned. :)