Solutions for Serving Lots of .torrents?

torpor asks: "For 10 years now my friends and I have managed to form a loose musicians collective with the purpose of putting our totally free music/art online for free access. Lately, we've been discussing the approach we can take to utilize BitTorrent to do our file distribution, with the idea being that we can use it to cut our bandwidth costs from that of a single server, and 'spread the bandwidth' among us for hosting files. However, we've found that serving a large number of torrent feeds from one server is not quite so easy as it sounds, as it appears we have to have a single instance -per file- of the torrent server. What solutions are available for Bittorrent-like distribution of media files? We're a small group of budding stars, and we want our tunes out there in the free world. Are we limited to continuing to serve things by HTTP, or are there easy/simple ways of starting a large-quantity of torrent feeds which, perhaps in our fervour, we're overlooking?"

Comments

Don't host it on UBC

[Anonymous Coward]

UBC's site sucks. SFU's is much better

Re:Don't host it on UBC

[Anonymous Coward]

So it is you who's ruining my wireless experience on SFU, you bastard!

What I would do

[Daxster]

Rename all the files to something like "[porn star] hot secks bath tub", and list all the filename conversions on your website with HTML. Then distribute these torrents on popular trackers and let them handle the torrent :)

Torrents suck, but..

[irc.goatse.cx troll]

btlaunchmany can be set to read torrents from a directory. Just throw the .torrent in and it will automaticly start downloading it/seed it when its done. You could easily crontab it and then just ftp upload to the dir. Removing a file is as easy as deleting the torrent.

Im sure theres a pretty web based solution for the cpanel 'admins' out there too, just google around.

How about having no server at all?

[AtariAmarok]

Check into www.ourmedia.org. You might be able to just uploaded your stuff there.

It is just starting out and has problems, but if it shapes up the way they intend, it sounds like it would be perfect for you to host any of your art elsewhere with whatever licensing you want. If this works out, it would be a lot simpler, too. Certainly it would be easy to test. The worst thing I have seen is that it often takes 30+ hours for something you uploaded to be listed.

Re:How about having no server at all?

[torpor]

We want to host ourselves, 100%. The whole point of doing the free art thing is so that we can get involved in the delivery of it as well .. if we put our stuff on someone else's server, we have zero control over it.

But if we start the distribution from a system we own and control and monitor ourselves, it puts us in the position we want to be in, as artists, to keep track of things. Putting it on some other host just gives that host the right to manage our free art accordingly .. thats not the point of our project.

Re:How about having no server at all?

[AtariAmarok]

"The whole point of doing the free art thing is so that we can get involved in the delivery of it as well .. if we put our stuff on someone else's server, we have zero control over it."

Have you looked into ourmedia.org? Their fine print looks to leave you in control of the stuff. You can even delete it. The problems I see with it are technical "does not work as well as it is supposed to" rather than "I give up control of my stuff".

Re:How about having no server at all?

[Linus Torvaalds]

You said:

For 10 years now my friends and I have managed to form a loose musicians collective with the purpose of putting our totally free music/art online for free access.

Then you said:

if we put our stuff on someone else's server, we have zero control over it.

...and:

Putting it on some other host just gives that host the right to manage our free art accordingly .. thats not the point of our project.

I think you are using a different definition of "totally free" to the popular one here. Could you define "totally free" for us so we know what is and isn't acceptable?

Re:How about having no server at all?

[runswithd6s]

Of course, we're assuming that you've read http://www.bittorrent.com/guide.html, which shows you how to do it. Recent versions are trackerless, so setting up the tracker is no longer valid, though if you want to service the 3.x versions, you may still want to set one up.

There's always freshmeat to search for options. EZTorrent appears to be what you want. mod_bt also looks promising if you're using Apache 2.x.

Re:How about having no server at all?

[AtariAmarok]

"I think you are using a different definition of "totally free" to the popular one here. Could you define "totally free" for us so we know what is and isn't acceptable? "

I think it is pretty clear that they want to be the access point for their free stuff. They don't want others to be the access point, due to concerns that others will charge for it, alter it, place restrictions, add odd download requirements, or other odd things that strangers could do with it.

Re:How about having no server at all?

[Andy_R]

This is one of the problems of being a collective - we might not all have the same definition of "totally free" (I use the word 'we' because I'm on a couple of Ampfea.org mailing lists, and I share the occasional bit of my own music there).

I think the intent is to keep everything under Ampfea's direct control. If this needs hardware or bandwidth, we'll have a whip round (and not stick ads on anything) to fund it, if it breaks, Ampfea's tech guys will fix it. In short, if it's in-house, everything will be done by people we know, doing it for the love of our music not for money or fame.

Re:How about having no server at all?

[torpor]

the point is: we want to be the source of our free material, not someone else. it is free, but we want it to be free from us .. not an archive or website we have no control over.

its a collective with the purpose of providing ourselves with the means to deliver free content easily .. and manage it.

BitComet

[Fry-kun]

BitComet can serve multiple torrents with different priorities, all within the same instance of the program

The Matrix?

[mr_rattles]

Am I missing something here? Why is the topic "The Matrix?"

Re:The Matrix?

[bersl2]

A vector of Torrents?

One tip...

[Burianski11]

If you're trying to save on bandwith costs, don't put a link to your site on /.

Re:One tip...

[Andy_R]

Actually, that hits the nail on the head, I'd LOVE to have a link to my latest album in my slashdot sig, and for Ampfea (the collective we are dscussing) to be the place that link goes to. Seamless, easy to use torrent system would be the ideal way to do that without breaking the bank.

Is there an actual problem?

[Otter]

As with a lot of P2P-related Ask Slashdots, it sounds like you have a solution in search of a problem.

The value of P2P comes on several fronts: obviously, it's ideal for illegal sharing (which doesn't concern you), BitTorrent in particular is good for huge legal files with huge demand peaks (e.g. new Linux distribution ISOs and it's good for large distributed bodies of files (like Furthur.net).

In your case, a website and HTTP distribution seems the best way to go, despite its unsexiness. You control the process, so you can track downloads and referrers and you can make sure that things work properly, so your new fans don't suffer. Just bite the bullet and pay a good hosting provider.

Re:Is there an actual problem?

[torpor]

We have our own server, and we have an excellent hosting provider.

We want to use Torrent because it 'spreads the load' easily, without needing a big management back-end to watch our bandwidth usage...

We've used HTTP distribution for 10 years now .. what we want is to make it even easier for members to contribute to the bandwidth bills by simply hosting their own torrent client peers themselves .. either you can make a donation to ampfea, or leave a torrent server running for all to access, on your own local end of things: this turns into a good solution.

If only there were really easy ways to manage the .torrent feed side of things ...

Re:Is there an actual problem?

[Anonymous Coward]

Looks like you are looking into a solution like they use over at http://www.demonoid.com/.

Re:Is there an actual problem?

[dubl-u]

BitTorrent in particular is good for huge legal files with huge demand peaks (e.g. new Linux distribution ISOs and it's good for large distributed bodies of files (like Furthur.net). [...] In your case, a website and HTTP distribution seems the best way to go, despite its unsexiness.

Another advantage of BitTorrent is that other people can easily join in to reduce your bandwidth costs.

When somebody recently a book via BitTorrent, I thought that was pretty cool, and wanted to help out. I joined all the torrents and have just left them running. If a favorite band were to release something this way, I'd do the same thing.

Thanks!!!

[AtariAmarok]

Thanks. After the second message where someone said this was Matrix, I looked closer. That category icon I always thought was a red-and-blue pacifier was actually Morpheus' magic beans!

Re:Thanks!!!

[Chyeld]

I imagine it's a take on the name of a popular multi-protocol P2P client... Morpheus

We Have Our Own Server.

[torpor]

Physically, we have our own server. Totally Managed, 100%. No license agreements, no business relationship: the machine itself belongs to the community.

Thats the point. We don't want unknown-entity/3rd-party Jones to have anything to do with it: we're fully DIY...

Re:We Have Our Own Server.

[AtariAmarok]

OK, I see where you are now. Just was thinking in the direction of a download solution much cleaner than torrent.

Using BitTorrent on Linux

[paul.dunne]

Funnily enough, I wrote up my experiences with BitTorrent only the other day, and it might be of some help to you: Using BitTorrent on Linux.

HE.net is your friend

[Daniel Boisvert]

Sign up for a webhosting account over at Hurricane Electric. They include free torrent hosting. According to their torrent FAQ, they'll eventually limit you to 25 torrents per account, but their servers sit on fat pipes and they won't charge you for the bandwidth used. :)

FYI: I don't work for them or anything; I'm just a very pleased customer.

What is the Matrix?

[Ridgelift]

This may be totally off-topic, but since the Matrix as a cultural focus was destroyed with the release of "The Matrix Revolutions" I wondered how it would be before Taco & the gang decided to recast the Blue & Red Pill icon away from the movies and onto something else.

That day arrived today.

Re:What is the Matrix?

[Anonymous Coward]

This may be totally off-topic, but since the Matrix as a cultural focus was destroyed with the release of "The Matrix Revolutions"...

To be honest, once I saw Monica Bellucci wearing that dress in the prior film, my focus had already shifted away from culture to something else entirely. :)

No problems with that here..

[hacker]

We're not having any trouble serving lots of torrent files from the same instance of the server. To the tune of over 109GiB of Plucker torrent downloads this year.

That doesn't count the downloads over http, rsync, and through our mirrors. Probably another 200GiB there, rough estimate.

I use bttrack and point it to the torrent directory (locked with --allowed_dir of course), and it works great.

I run a separate server on a separate port for various projects, to separate the torrents per-project. No issues at all.

Re:No problems with that here..

[hacker]

Replying to myself here... I just did a quick check and we've served up 25,502,515,260 bytes over http for one of those files (plucker_desktop, Windows build) THIS MONTH on the primary server.

Since we round-robin across 3 mirrors, that could potentially be 75GiB for this month alone for that one single download (10,241,974 byte file).

Its massive, and that's the primary reason we started moving to BitTorrent to help distribute those downloads.

Try Dijjer

[CrosbieFitch]

It may be worth a look.

http://dijjer.org/

neologism!

[Rinisari]

torrentcasting!

Customize a podcasting feed and augment Azureus or something to handle it. Neat!

Re:neologism!

[Oculus Habent]

Look for support in iTunes 5.1! :)

Re:neologism!

[Anonymous Coward]

That's actually pretty insightful, as bandwidth is a major concern for podcasting. I've known a few podcasts that have gone under due to the cost of serving so much data.

Azureus

[Guspaz]

I don't think that solves his problem though, since he would still need once instance per torrent file, unless I'm mistaken.

The answer is an application like Azureus. It handles everything itself. It serves up the torrent files with a built in HTTP server, it acts as the tracker for all torrents, and it acts as the client to seed each torrent. It allows very powerful rules about when to seed which file, how fast/how much, and so on.

So the net result is you can handle the entire hosting procedure with just one application/instance.

More info at http://azureus.sourceforge.net/

Re:Azureus

[Dahan]

No, the whole point of btlaunchmany and btlaunchmanycurses is that you have one process that downloads/seeds a bunch of torrents.

Re:Azureus

[Guspaz]

Ah, not so bad then. However you still need the tracker and web server, and you don't have the option of seeding a torrent trackerless. So you need at least 3 processes, which is admittedly better than one per torrent plus 2 extra.

Re:Azureus

[ksheff]

But how it btlaunchman/btlaunchmanycurses currently works is to have a process per torrent and a process to manage the display of the download stats.

It sounds like the original poster should ask someone at a site like dimeadozen, etree, or tradersden what they are currently using to manage the numerous torrents they are hosting. As far as I know, they still need a seeder process for each torrent. Maybe there is a server version of azureus.

Re:Azureus

[Dahan]

No, that's not correct either. It's a single process, but it's multithreaded. Perhaps you're confused by how LinuxThreads (the old Linux 2.0 thread implementation) created a process per thread. There were some issues with that approach, since processes really weren't meant to be threads. The current Linux thread implementation, NPTL, is much more efficient, and is supported in Linux kernel version 2.6 and later (and I guess some 2.5.x development versions).

Re:Azureus

[ksheff]

So the multiple instances of python that ps reports is a figment of my imagination? maybe I just need new glasses.

Re:Azureus

[kaisyain]

There's no difference between a process and a thread in the ps listing.

Re:Azureus

[Anonymous Coward]

WTF? No difference between the two? Either you're wrong, or that's seriously broken.

Re:Azureus

[cortana]

Or just possibly, you have an imperfect understanding of how the various threading implementations fit together.

Re:Azureus

[Monkelectric]

The answer is an application like Azureus

Has azureus fixed its memory leaks? It has some *severe* problems, so much so that it locks up my X server on a daily basis.

Re:Azureus

[Guspaz]

It fixed the memory leaks from 2.3.x.x, yes. It still takes a lot of memory though (Your definition of a lot may vary), though it makes up for it in power/flexibility.

btlaunchmany.py

[atomic-penguin]

Use the btlaunchmany.py (part of the official distribution) to seed several torrents with just one instance of BitTorrent running. It recognizes most of the same command line options as the vanilla bittorrent client.

How about this...

[Anonymous Coward]

Why not just make a torrent of torrents?

Or is that redundant?

Re:How about this...

[MentalMooMan]

Yes. I thought of this. Why not make a torrent of every other torrent you have, and maybe people really interested in your stuff will grab the whole lot, and decide what to download after they get the bunch.
Perhaps throw in an html page with whatever descriptions/titles you have already done on your site, and link to the local torrents. I think it would work.

(That wasn't very clear was it? Bah.)

Piratebay

[bhima]

I can't believe anyone has said this but you're asking the wrong crowd!

Ask ThePirateBay!!

eztorrent

[mrsam]

When I was in a similar situation, I ended up writing eztorrent. You can find it on Freshmeat.

Publishing BitTorrent content is as simple as copying the files into a directory, and running a single command. Eztorrent automatically creates the matching .torrent files, start the tracker, and start a seeder for each .torrent file.

Files can be added or removed from the torrent directory at any time. A single command adds/removes any .torrent files, and starts or stop the seeders, without any downtime for any other, active torrents.

eztree

[stardancer]

Check out the people sharing live music shows at http://www.dimeadozen.org/ - their tracker is even open source. Could be able to help you, if I understood the question correctly.