Ask Slashdot: Do You Move Legal Data With Torrents?

An anonymous reader writes "We've recently seen a number of interesting projects come from bittorrent.com, including Sync and SoShare. I sometimes use torrents to move several GB of data, especially when pushing large bundles to multiple destinations. It's mostly a hodgepodge of open source tools, though. Apart from anecdotes and info from bittorrent.com, details are thin on the ground (e.g. the Blizzard Downloader). I have two questions for the Slashdot community. 1) Do you use BitTorrent to move data? If so, how? i.e. What kind of data and what's the implementation? 2) If you've looked at torrent clients/tools, what's missing in the open source ecosystem that would make it more useful for moving around large blobs of data?"

Comments

I use it for linux distributions

[Qzukk]

The entire point of swarm topology is to move data to a lot of places at the same time. If you just need to get data from A to B without sharing it with anyone else, rsync it.

Re:I use it for linux distributions

[TooMuchToDo]

While working at Fermilab on the LHC CMS data taking team, I used bittorrent to speed up re-installs of thousands of worker nodes. I was able to saturate 10Gb Ethernet links this way, and could reinstall ~5500 Linux boxes within 10-15 minutes (with only two initial OS source servers).

Yes, Bittorrent is not just for piracy.

Re:I use it for linux distributions

[corychristison]

To be fair, rsync + ssh is equally secure as scp. I actually think rsync uses scp in that situation (please correct me if I am wrong).

The advantage I see rsync having is it is useful for automated (backups) of a large collection of files vs gzipping and copying via scp.

Although git also does a great job of that with concurrent revisioning built in.

It all boils back down to using the right tool for the job.

Re:I use it for linux distributions

[Anonymous Coward]

I nearly jizzed in my pants reading this. Nerdgasm.

Re:I use it for linux distributions

[bobbied]

It all boils back down to using the right tool for the job.

The wise keep as many tools in their tool box as possible. You seem to be among the wise.

Re:I use it for linux distributions

[Sloppy]

rsync? I'd use SCP myself, it's a whole lot more secure and the data goes encrypted.

Speaking as someone who, at any moment, only has to hit up arrow a couple times to see an "rsync .. -e ssh ..." line, I'm telling ya: it's all the same.

Re:I use it for linux distributions

[Tarlus]

Not by default, though. You have to use a flag.

Re:I use it for linux distributions

[Anonymous Coward]

Broadcast / multicast would have been a better solution.

It wouldnt have saturated your pipe, and only taken 10 seconds or so (4GB image * 8 b/B / 10Gb/s * 3 for overhead).

Re:I use it for linux distributions

[TheCarp]

$ env |grep RSYNC
RSYNC_RSH=ssh

Worth putting right in /etc/profile so anyone who doesn't want it can disable it if they want.
It is an entirely sane default.

Re:I use it for linux distributions

[DragonTHC]

That's not the entire point.

If you want to save bandwidth and still distribute your data, then crowdsource your downloads with bit torrent.

Linux, MMO games, game mods, etc. All excellent uses of bit torrent.

Re:I use it for linux distributions

[nabsltd]

The entire point of swarm topology is to move data to a lot of places at the same time. If you just need to get data from A to B without sharing it with anyone else, rsync it.

One huge advantage of bittorrent is that error checking/correcting is built in.

Although you would still need to re-download the block with the error, it's a very small amount of data (usually a few KB). This solves the problem of errors that happen after any data transport verification, as most bittorrent clients can be configured to do a final re-check after the torrent is complete. In addition, the data transport can be encrypted if you need it to be. Although it's not the strongest of encryptions, it would stop most casual snoopers.

Re:I use it for linux distributions

[Isaac Remuant]

Probably the fastest way to get your latest linux distro (and Libre Office).

Re:I use it for linux distributions

[i.r.id10t]

Yup, it is one of the few ways that darn near any user out there can contribute back to Open Source.

I can code, some, but not very experienced with c/c++ - just haven't had the need to do it. I hate writing documentation. I do file bug reports, but the stuff I use is pretty darn stable.

So, to give back, I seed iso images for 24 hours or 100gb in upload of any Linux distro release notice I see here on slashdot, even if I don't use that distro.

Re:I use it for linux distributions

[wagnerrp]

How would you use bittorrent to distribute the OS? Hint... it's something a bit more complex than your typical system BIOS.

Re:I use it for linux distributions

[Anonymous Coward]

"Nearly"? Hand in your geek card.

(But wipe it off first, please. I ain't takin' no chances.)

Re:I use it for linux distributions

[fluffy99]

How would you use broadcast or multicast to distribute an OS? Call me ignorant, but how would you do that in practice?

Pretty easy to setup using Windows Deployment Services
http://www.windows-noob.com/forums/index.php?/topic/452-how-can-i-multicast-an-image-in-windows-deployment-services-windows-server-2008/

Or if Linux is your preference
http://www.udpcast.linux.lu/

Or Clonezilla has a multicast restore function
http://clonezilla.org/clonezilla-SE/use_clonezilla_live_in_drbl.php

Re:I use it for linux distributions

[wagnerrp]

Use multicast for this.

Stop using shitty networking gear that squashes multicast traffic. Use erasure or fountain coding to transmit your datagrams so clients can compute ones they missed. Have clients individually request any remaining lost datagrams over unicast.

Torrents are a very specific solution to a very limited problem in networking; specifically, people being too cheap to invest in proper co-located servers for on-demand traffic. Stop trying to apply it everywhere else.

Re:I use it for linux distributions

[kcbnac]

Here's how Norton did it, back in the late 1990's: http://www.symantec.com/business/support/index?page=content&id=TECH106806

I recall using this in college, in 2003, to reimage our 'learning' workstations. (After we'd break them, like discovering that Windows 98 SE would let you format the OS volume, and not crash.)

Re:I use it for linux distributions

[TooMuchToDo]

You don't multicast images to Linux machines when you're using configuration management tools. PXEBoot->Bare Image Install->Puppet system configuration upon first boot based on machine grouping/criteria.

Disclaimer: I'm OP.

Re:I use it for linux distributions

[grcumb]

$ env |grep RSYNC
RSYNC_RSH=ssh

Worth putting right in /etc/profile so anyone who doesn't want it can disable it if they want. It is an entirely sane default.

I don't think that's required any more - not on Linux or Mac OS X, anyway. I use rsync several times a day and each time it just reads my ~/.ssh/config file for the options, sets up the connection and performs the transfer without any fuss or bother.

I haven't set the RSYNC_RSH env parameter since about 2002.

Re:I use it for linux distributions

[ArsonSmith]

This seems very interesting. I'm doing a much smaller setup now with NFS and running the OS in RAM. Even with only about 40-50 servers I can get boot storms on the pxe boot server. I may have to look into this as an option. If I can get a bittorent client into the initrd...hmm time to start playing.

Re:I use it for linux distributions

[tlhIngan]

There was an article on how Facebook did their software updates. First they compiled their PHP into C, then compiled that to produce a binary that's several gigabytes in size (with all the resources and stuff).

That binary is then sent to their datacenter via Bittorrent so it can replicate itself across all the servers efficiently. The final stage is to bring down the node, swap the binaries and bring it back up.

http://agilewarrior.wordpress.com/2011/05/28/how-facebook-pushes-new-code-live/

There was another article on it from Facebook, but I can't seem to find it.

Re:I use it for linux distributions

[ArsonSmith]

If you download pirated files then the torrents win!!

Re:I use it for linux distributions

[gullevek]

Yes, via the -e command. I wouldn't use rsync anyway different between two hosts.

Re:I use it for linux distributions

[delt0r]

On all my linux installs rsync uses ssh by default. rsh hasn't been installed on anything in years. Is it even included in distributions anymore?

Re:I use it for linux distributions

[dargaud]

vs gzipping and copying via scp

"scp -C" will compress as it transfer, no need to gzip.

Re:I use it for linux distributions

[DrXym]

Although git also does a great job of that with concurrent revisioning built in.

Git is a fine way to manage source code or perhaps semi-infrequently changing files like Word documents. It would be absolutely disastrous to use it to manage large binary blobs or log files or anything changes frequently. The reason for this is that a git project holds every version of the file ever and not necessarily as deltas either. Aside from that, the files have to be added and committed to the server before they're visible for pull which increases footprint size.

And the client side has to clone / pull every blob too. A clone can specify a depth to restrict the history of the clone, but it would still build up a history of cruft when perhaps you only want the latest version.

So while I use git in a lot of scenarios myself (mainly source control) I think it is not necessarily a substitute for either scp or rsync except for relatively sedentary directories where the corresponding git project isn't going to bloat out of control over time.

Re:I use it for linux distributions

[eyegor]

We're currently using the ROCKS cluster distro to run our cluster, but are finding that it's beginning to limit our ability to patch and otherwise maintain our cluster infrastructure. We've adopted cobbler and puppet for some of our HPC assets and will likely switch from ROCKS to more of a home-grown approach to manage our nodes. One thing I dearly love about ROCKS is the Avalanche Installer which uses bittorrent to distribute the image to the nodes when they do their initial build. I've

Are you using that or a similar package to do your node builds?

Re:I use it for linux distributions

[Danathar]

Yea Rocks clusters has been using a modified version of bittorrent to re-install nodes in a cluster for years.

Re:I use it for linux distributions

[Opportunist]

Hey, I practice safe sex. I have a firewall and every porn file gets scanned first before use.

Re:I use it for linux distributions

[Opportunist]

And how typically nerd. The discussion whether what you did was time efficient already takes WAY longer than the 10-15 minutes it took you to install it.

That's why we can't have nice things. They never get finished, we already fuck up everything in the planning phase with endless discussions and e-peen waving.

Re:I use it for linux distributions

[tehcyder]

Likewise.

I've gotten odd looks from my co-workers when I mention that I use bit torrent to download ISOs because it's faster. In most people's minds, bit torrent is synonymous with piracy.

That's because bit torrent is mostly used for piracy.

The percentage of the population that downloads Linux ISOs is tiny.

Re:I use it for linux distributions

[DuckDodgers]

It's common for the person in charge of distributing the operating system images to be very different from the person in charge of the network. Maybe the parent AC has no control over the networking gear.

But further, wouldn't multicast work best only when you're transmitting images to multiple machines at once? If the image has to be retransmitted after each system restart, wouldn't torrents work better? The machine that restarts can just grab everything it needs from all of the other machines running the same image on the same subnet.

Re:I use it for linux distributions

[wagnerrp]

If you're individually restarting machines, then it would be better for those handful of machines to pull an image from the image server, and not disrupt the other compute nodes. If you're restarting a whole bunch of machines, it's better to push the images out to all the nodes simultaneously using multicast, rather than bog down the network with all kinds of redundant unicast traffic.

Re:I use it for linux distributions

[kyrsjo]

Why should he/she come up with a new solution (and the one you propose honestly sounds more complex), which requires different HW (migth be problematic if all the HW isn't in his control), if he/she already have a working solution?

Re:I use it for linux distributions

[TooMuchToDo]

That's exactly what we used for configuration management and distributed/unattended installations :)

Re:I use it for linux distributions

[TooMuchToDo]

Something to think about:

If you're seeing boot storms, get a network switch that is managed and support vlans. Either programatically switch VLAN access on the switch, or use trunking support in Linux, so you have a production VLAN and an installation VLAN. This should segment your network to the point where boot storms are no longer an issue.

Re:I use it for linux distributions

[pnutjam]

Here's a question: To know which bits have changed, doesn't it need to compare the two files. How does this result in bandwidth savings?

Re:I use it for linux distributions

[TooMuchToDo]

We used ROCKS, and started running into the same problems you did. We were mid-process of moving to our own custom built solution when I left to work somewhere else.

Re:I use it for linux distributions

[3dr]

It doesn't need to compare the file contents bytewise. Each end of the rsync connection generates hashes of blocks of the file, and the hashes that do not match indicate underlying file differences. If a 1MB chunk in the middle of a 100MB database file generates different hashes, then only that 1MB chunk is transferred. That's how bandwidth savings are guaranteed.

Re:I use it for linux distributions

[denmarkw00t]

The wise keep as many tools in their tool box as possible. You seem to be among the wise.

The wise build workspaces with proper space arrangements and storage for their tools - only a fool puts all of their tools in one box.

Re:I use it for linux distributions

[ArsonSmith]

The OS is actually redeployed on every startup and only ever stored in system ram ramdisk. The image is very small and it hasn't yet caused an issue. I can just see an issue coming when we go from the current 32 nodes to the planed 3-400 by next year. The NFS infrastructure can handled what we have now with little impact. OS and apps are 1.2G, although I've been trying to trim that down a bit too.

Re:I use it for linux distributions

[toddestan]

I've gotten odd looks from my co-workers when I mention that I use bit torrent to download ISOs because it's faster. In most people's minds, bit torrent is
synonymous with piracy.

By ISO, I assume you mean pirated copies of Windows, right?

Re:I use it for linux distributions

[KingBenny]

lots of stuff, mostly demo's, games and linux isos yea, i usually let it run until i need the space back or it's format c: time, lots of games use it, since these days 10gb for a single mmo or similar is not a rarity anymore (something local ISP's dont seem to get since i sometimes get warnings about too much traffic on this 'unlimited downloading' pack subscription when i check out some stuff on steam. It's probably most useful since the load is distributed and that probably save money serverside. Also the clientside doesnt need to worry about a broken http download so yea, i see lots of uses

Re: I use it for linux distributions

[MattMason]

This is awesome! I work at BitTorrent and would be great to hear more about this. Hit me on email - Matt at BitTorrent dot com

Re:I use it for linux distributions

[Man Eating Duck]

Here's a question:
To know which bits have changed, doesn't it need to compare the two files. How does this result in bandwidth savings?

It only needs to compare hashes. Rsync goes a step further and uses the ingenious concept of rolling checksums to determine if only parts of a file has changed.

Re: I use it for linux distributions

[TooMuchToDo]

Email away!

Re:I use it for linux distributions

[WhiteDragon]

Here's a question:
To know which bits have changed, doesn't it need to compare the two files. How does this result in bandwidth savings?

No, because it only sends a hash of the data. The other side computes a hash of it's data, and if the hashes match, the transfer is complete. If The hashes don't match, there is a rolling hash that can verify a partial match, and send only the changed data. There's more info here

Re:I use it for linux distributions

[WhiteDragon]

How would you use broadcast or multicast to distribute an OS? Call me ignorant, but how would you do that in practice?

I used to work in a computer lab at the university I went to. We used Ghost to do exactly that. All the computers in the lab (a few hundred PCs) were booted up off a special floppy (or later, cd) that started up Ghost in listen mode. Then, the central server used multicast to send the OS image to all the clients at once. It took less than an hour, and that was with 100Mb/s Ethernet.

Re:I use it for linux distributions

[TheCarp]

Well I will be damned, the man page agrees with you, this would tend to imply the -e ssh is obsolete too.

For remote transfers, a modern rsync uses ssh for its communications, but it may have been configured to use a different remote shell by default, such as rsh or remsh.

Absolutely

[Anonymous Coward]

All of my Torrents are legal data. What else would I use Bittorrent for besides Linux distros and Humble Bundle games?

Re:Absolutely

[kamapuaa]

Amateur electronica, generated in Garageband?

Re:Absolutely

[Hentes]

Eclipse, of course. Although I'm unsure whether the submitter meant "moving data" as using bittorrent for file transfer, not downloading.

Re:Absolutely

[Nadaka]

Distributing home made erotica of course!

Re:Absolutely

[UltraZelda64]

Porn?

Re:Absolutely

[Opportunist]

C'mon, that's cheating. "Porn" is a valid answer for all questions that somehow deal with the internet.

Re:Absolutely

[poofmeisterp]

Amateur electronica, generated in Garageband?

Nah nah, Amateur Garageband, generated in electronica. Way more transfers.

No - Resources

[Synerg1y]

Moving large data requires resources. In the case, of bittorent most things don't qualify because it's a distributed network, if 10 people in the office have the file and all know how t seed / use bittorent, you'd still be throttled by your bandwidth. Bittorent has however time and time again shown that it's distributed architecture can get something out to the masses very effectively.

Re:No - Resources

[Grishnakh]

I don't see why Bittorrent wouldn't work for an office of 10 people; its strength is copying data over distributed networks. For an office with 10 people, assuming they're on wired Ethernet links rather than WiFi, all 10 of those links will be connecting to one or more switches, which are able to handle full 100Mb or 1Gb speeds, duplex, between the computer and the switch, simultaneously. Using BT to copy the data might be a little slower than just using scp if it were only one PC, but the total transfer time for all 10 will be less.

Re:No - Resources

[sourcerror]

If the 10 people are in the same collision domain, torrenting won't make things faster. Actually it can slow things down with unneccesary collisions.

Re:No - Resources

[RuaisLampSilog]

Yes and No. Moving data from A to B using scp is ok if your link is stable. At my work, to download the 12 Gbytes of that other crappy OS, there is an SMB share. I have tried many times to copy 4 Gbytes dvds over it, most of the times with no luck. Even when there is a single source and a single destination torrent provides error recovery, and consistency not matched by other protocols. Other nice feature is that any client can become source of new files. Let's face it: rsync is beautifull but setting up an rsync server is not for newbies. On the other hand, any newbie can create a torrent file and let others download from him, with a lot of benefits. And the last is that most torrent clients have bandwidth limiting options. Even when you can say rsync how much you want it to use, to be able to change it on the fly sometimes is priceless. So yes, I think torrent in some scenarios, even with single source and client, is the way to go.

Re:No - Resources

[darkain]

Serious question... How long have you been using a network HUB instead of a network SWITCH?

Re:No - Resources

[MightyYar]

Collisions? Are switches that expensive?

Re:No - Resources

[StuartHankins]

cwRsyncServer (I'm not affiliated with them) makes setting up an rsync server in Windows straightforward. It's only 5 minutes or so in Linux. Bandwidth limiting is a --bwlimit switch. I do notice that in general rsync in Windows is slower than on Linux or OSX.

Re:No - Resources

[Drakonblayde]

You're confusing broadcast domain with collision domain.

On a switch, a collision domain is limited to the link itself, ie, collisions aren't possible. However, all ports on the switch (or all ports in the same VLAN, if the switch is manageable) share the same broadcast domain, which merely allows for all ports to contribute to saturation, but there still won't be any collisions

Re:No - Resources

[dead_user]

I wasn't even aware they still SOLD hubs. I haven't seen one in years.

Re:No - Resources

[bruce_the_loon]

Probably the same guys that never upgrade from their 486DX120 because it's fast enough to run Lynx.

Re:No - Resources

[henrypijames]

But if only I have a file and want to distribute it to those 10 people in my office, using BitTorrent is much faster than copying it to them one after another -- precisely because of the bandwidth throttle on my end.

Re:No - Resources

[Whiteox]

On the other hand, any newbie can create a torrent file and let others download from him, with a lot of benefits.

I can't.
I've tried numerous times:
1. Create torrent (easy)
2. Wait (easy)
3. No-one downloads torrent (fail).

Apart from P4P, I've given up on uploading torrents as NO-ONE IS INTERESTED.

Yours,
Failed Newbie

Re:No - Resources

[Synerg1y]

You're correct there, it would take a little longer for anybody to get it as the file seeds, but the 10th person would get it a ways faster and at around the same time as the others. However, traditionally this problem is handled via a file share server, but of course not everybody has / wants one.

Re:No - Resources

[poofmeisterp]

If the 10 people are in the same collision domain, torrenting won't make things faster. Actually it can slow things down with unneccesary collisions.

Is this 1994?

Re:No - Resources

[poofmeisterp]

Probably the same guys that never upgrade from their 486DX120 because it's fast enough to run Lynx.

bah dump bump, tsssing!

Nope.

[Rinikusu]

I mean, other than the Blizzard stuff, no, I don't use bittorrent at all unless I'm downloading movies (usually) or software (sometimes).

Rapidshare/megaupload/etc work much better for my one-off transfer needs, while I leave media distribution to the masses via Youtube, Vimeo, Bandcamp, and media collaboration to Dropbox and sneaker-usbdrive-net (especially for big projects).

Yes, I have

[EkriirkE]

Many times a person is searching for a program to do something by keyword instead of software title, and for free. Torrent sites are a common place to go for something free. I just generate a .torrent for my software(s) and upload it to a few big trackers and the others seem to pilfer it from there. Just make sure the filenames and titles are relevant. It's like SEO, but TTO: Torrent Tracker Optimization.

In a word? YES!

[bobbied]

I specifically do not torrent anything that has copyright issues but I do seed a number of Linux distributions and development tools which do not prevent distribution in their licenses. Downloading anything using torrent is effectively distribution of the material too, so you had better KNOW that the license allows you to make copies and give them away.

You folks that torrent movies and stuff that is not in the public domain are crazy in my book.

Move Legal Data With Torrents?

[MarkvW]

Only a complete fucking moron would move legal data with torrents.

A lawyer is obligated to preserve his clients' confidences. When you store your information on somebody elses server or servers you are giving up custody and control over some of those confidences. In that situation you are entirely dependent upon the strength of your encryption.

That encryption might be good today or tomorrow, but how good will it be five years from now or ten years from now when quantum computing or the next best thing becomes available for codebreakers.

Don't risk a lawsuit from a pissed client!

Re:Move Legal Data With Torrents?

[godrik]

In this context, I believe legal meant "not illegal". Or "data I own" or "data that nobody will sue my ass off for moving them around"

Re:Move Legal Data With Torrents?

[treajones]

I don't think that is the Legal data they are talking about.........

Re:Move Legal Data With Torrents?

[Anonymous Coward]

Okay, I saw the joke, and it was good and all, but maybe shorten it a bit. Like, take out the encryption sentence. Being concise helps!

Re:Move Legal Data With Torrents?

[Anonymous Coward]

#accidentaljoke

Re:Move Legal Data With Torrents?

[Sloppy]

Respectfully, I believe the submitter meant "legal" as opposed to "illegal"

Aren't you worried about getting the deaf penalty for using their network racehorses to trade TV shows containing sax and violins?

Oh.

Never mind.

Re:Move Legal Data With Torrents?

[UltraZelda64]

What are they gonna do? Fire a machine gun by your ears a few minutes until you can't hear any more as a sentence? Yeah, that would have the potential to make you go deaf...

Re:Move Legal Data With Torrents?

[Zontar The Mindless]

Dude. Take a Valium.

Re:Move Legal Data With Torrents?

[xyzyxx]

I think you are misinterpreting the meaning of the word "legal".

Re:Move Legal Data With Torrents?

[poofmeisterp]

In this context, I believe legal meant "not illegal". Or "data I own" or "data that nobody will sue my ass off for moving them around"

Oh, so THAT'S why Slashdot has a moderation unit called "Funny". NOW I get it!

Re:Move Legal Data With Torrents?

[poofmeisterp]

Dude. Take a Valium.

Are you torrenting any?

*snerk*

Can you use it to move a 300 megabyte hosts file?

[Anonymous Coward]

inquiring minds want to know.

Linux ISO's mostly

[dlapine]

At work I need to install several different types/versions of linux OS's for testing. I always torrent the ISO as a way of "paying" for the image that I'm using.

A few years back, we did some experimenting with torrents over the Teragrid 10GBe backbone, to see how well that worked over the long haul between IL and CA. With just 2 endpoints, even on GBe, it wasn't better than a simple rsync. We did some small scale test with less than 10 cluster nodes on one side, but still not as useful as a Wide Area filesystem we were testing against. Bittorrent protocols just aren't optimized for a few nodes with a fat pipe between them.

I am interested in looking at the new Bitorrent Sync client to see how thanks for our setup. We have many users with 10's of TB's of data to push around on a weekly basis.

Yep- Linux

[markdavis]

Just about the only time I use torrents is when downloading Linux distributions- Mageia, Fedora, CentOS, etc. Occasionally iso's for grub magic, ultimate boot CD, and such. All of that legal. And I usually leave it up at least long enough that my share ratio is 100% (1.0).

Re:Yep- Linux

[nabsltd]

Occasionally iso's for grub magic, ultimate boot CD, and such. All of that legal. And I usually leave it up at least long enough that my share ratio is 100% (1.0).

For such tiny things (latest UBCD is 486MB), I pretty much seed forever, as it doesn't really cost me anything. I'm at 67:1 on the latest UBCD, and over 400:1 on some other much smaller torrents.

I leave them running because I don't artificially limit my upload rate on a per-torrent basis, only as a total for all torrents (and that limit is quite high, as I have Verizon FiOS). It really bugs me when I try to download an older torrent and the only seeds are uploading at a few KB/sec. Even on a 100MB torrent, it can take 5-6 hours to download.

Re:Yep- Linux

[StuartHankins]

Thank you for seeding :) It's underappreciated.

Re:Yep- Linux

[jones_supa]

Just about the only time I use torrents is when downloading Linux distributions- Mageia, Fedora, CentOS, etc. Occasionally iso's for grub magic, ultimate boot CD, and such. All of that legal. And I usually leave it up at least long enough that my share ratio is 100% (1.0).

Even better is if you aim for a share ratio of 2.0. With a share ratio of 1.0, you only "give back what you take", so the swarm stays as strong as it was. That's good. But sending back an "extra copy" gives your contribution to make the swarm stronger.

occasionally

[BenSchuarmer]

mostly for music that's under creative commons license and the occasional Linux download.

Re:Patent Trolls?

[bobbied]

Does this seem like fishing expeditions by patent trolls?

No need to go fishing here.. Just find the torrent tracker and connect... Volia, you have a list of everybody who is distributing the material by IP address. It's all about tracing down the IP's and sending out the collection letters and cashing the checks from there, assuming you actually have the permission of the copyright holder to do so...

World of Warcraft

[steamraven]

If I remember correctly, some online games use P2P to distribute updates legally (though they might not use bittorrent).

Re:World of Warcraft

[Black Parrot]

If I remember correctly, some online games use P2P to distribute updates legally (though they might not use bittorrent).

Before I moved to Steam, torrents were the *only* way I could get updates for Company of Heros.

Re:World of Warcraft

[Luuseens]

Indeed, World of Warcraft used to (still does?) provide the updates and installs as a thin torrent client. You could even extract the torrent file from the binary and use a different torrent client to download the update files if you wanted to.

Some

[GameboyRMH]

Linux distros, free movies, free games...

I tried to switch to Deluge but it couldn't handle a file with a Japanese character in its name...other than that, only things that I think many torrent clients could use is the ability to accept magnet downloads through a drop folder somehow, and searching & better sorting/filtering options for downloaded torrents.

Re:Some

[David_W]

I tried to switch to Deluge but it couldn't handle a file with a Japanese character in its name...

FYI, IIRC, it only has issues adding a .torrent with Japanese in the filename; if you rename the file and add it, it can download the files "within" the torrent just fine even if they have Japanese in them. And yes, this is odd.

Re:Some

[GameboyRMH]

Huh thanks I'll have to try that, that may be the problem and it was the only holdup that kept me from switching. IIRC it added the torrent successfully but then couldn't find the files that were already downloaded.

Democracy Now

[Anonymous Coward]

is a daily TV news show distributed by torrents:

http://ewheel.democracynow.org/

I download it every day.

We deploy VM images to our developers over BT

[carlhaagen]

We use the OpenBitTorrent tracker and the Transmission client to deploy and acquire virtual machine hard drive images among our developers. The obvious reason is that it's much faster for our developers to help eachother out with shoveling the data around rather than the developers having to get all the data from one and the same link (read: the main server). Compare it to a person reading a novel, once, out loud, to a group of people, instead of reading it in private over and over to each and every attendee.

Re:We deploy VM images to our developers over BT

[wagnerrp]

That's a horribly inaccurate analogy. Bittorrent is like a restaurant. If you have two people holding a conversation in a restaurant, everything is fine. Even if you have several pairs talking, it's really not a problem. When you have a full restaurant and everyone is holding their own independent discussions, no one can hear a damn thing. If your network is using a hub, you would be stupid, but that's what you would get. Assuming you're using switches like a rational person, chances are not everyone is on the same switch. If they were, you would have sufficiently few developers that they would have no problem just sequentially pulling the new images off a shared server. So, further assuming you have developers networked through a branching topology, bittorrent is not topology aware. It will not attempt to favor those peers close to it on the network, and thus will quickly clog your trunk lines between switches.

Your analogy is an accurate description of broadcast or multicast networking. Why aren't you using multicast networking? Have your developers run a client that joins a certain multicast group. At a certain, scheduled time of the day, the server pushes new images onto that group, and to any client listening on that group. The client then reconstructs the image, and stores it on the hard drive of each developer. One person talks; everyone receives. No physical link ever has to transmit more than that one original copy of the file. Anyone whose machine is not available for the update can then download it manually from the central server at their leisure.

gittorrent

[lkcl]

the one thing that would help enormously would be to have git be *truly* peer-to-peer distributed. not "yeah shure mate you can always git pull and git push, that's distributed, and you're a peer, right, so... so... git is peer-to-peer and distributed, so what are you talking about you moron??" but "at the network level, git pull and git push have a URL type that is **TRULY** peer-to-peer distributed. to illustrate what i mean, i would like to be able to do the following - with all that it implies:

git clone magnet://abcdefg0123456789/gittorrent.git

if you're familiar with magnet links, you'll know that there is *no* central location: a DHT lookup is used to find the peers.

now, what wasn't clear to the people on the git mailing list when i last looked at this, was that it is possible to use bittorrent to do git pack objects, by creating a file named after the pack object itself. and what wasn't clear to sam (the last person who tried to put git over bittorrent) was that you *MUST NOT* make use of bittorrent's "multiple file in a torrent" feature, because bittorrent divides up its data into equal-sized blocks that *do not* line up with the files that are in them, which is why when you download one file in a torrent you almost always end up with the end of its preceding file and the start of the one after it, as well.

the idea i came up with is that you create *multiple* torrents - one per git object (or git pack object). if you want to pull a tree, you create a torrent containing *one file* which is the list of objects in that tree; gittorrent would then know to map each of those objects onto yet *another* torrent (one per object), repeat until all downloading happily. gittorrent objects are of course named after the hash, so you can pretty much guarantee they'll be unique.

and, adding in a DHT (a la magnet links), you are now no longer critically dependent on something like e.g. github, or in fact any server at all.

to answer your question in a non-technical way, mr anonymous, i think you can see that i feel it would be much more useful to have development tools that use bittorrent-like protocols to share files-as-revision-controlled-data (and, if you've seen what joey hess is doing with bittorrent you'll know that that's a hell of a lot - including storing home directories in git and doing automatic distributed backups)

Re:gittorrent

[Qzukk]

Intriguing idea, but I tried subscribing to your newsletter and I keep getting the Sept. 24, 1998 edition over and over. The problem with using bittorrent for this is distributing NEW data. If the protocol could cope with a seed appending data to the torrent without having to create a whole new .torrent file, then this could be awesome. As it is, you're just changing the problem from "how do I send out new versions of files when I commit something" to "how do I send out new versions of the .torrent files every time I commit something"

Re:gittorrent

[jones_supa]

I tried subscribing to your newsletter and I keep getting the Sept. 24, 1998 edition over and over.

A new meme is born!

Re:gittorrent

[akanouras]

I see your flamebait already fooled some mods. However, you're supposed to attack the idea, not the man. And you failed spectacularly at that.

The key component in what he's proposing would be the DHT; not even necessarily the same DHT as used by standard BitTorrent clients. The use (or not) of the BitTorrent specification and its intricacies would be just an implementation detail.

You're free to be one of the "can't be done" people; it would be wise not to advertise it too much around here however; as it's not become Reddit yet.

"how do I send out new versions of the .torrent files every time I commit something"

I think I've had enough. Off to BT101 with you.

Re:gittorrent

[Qzukk]

However, you're supposed to attack the idea, not the man.

Care to point out where I insulted the man? All I did was torrent his newsletter, but every time I connect to the swarm, I never get any new data, I always get the same data again. This has been the way bittorrent worked since its inception (so I was off by 3 years). After all, if anything changes then the hashes in the .torrent file will tell the client it's corrupt and the client will try to change it back.

The use (or not) of the BitTorrent specification

If you "(or not)", then you're making it something other than what he's proposing.

Off to BT101 [wikipedia.org] with you

OK, I read it. I'm not seeing how DHT will solve the fact that the first time you commit a change to your code, the bittorrent client will detect it as corruption and replace the files with the original version from the swarm since the modified files won't match the hashes in the .torrent. Using a magnet link just hides the .torrent from you, the first commit will still need to somehow add new hashes (god forbid you do a rebase and change ALL the hashes).

The least invasive way I can think of to do this with some semblance of security would be some sort of public/private key arrangement that would identify the author of the .torrent file and allow that person to distribute replacement .torrent files through the swarm. When I commit a new version of foo.c, my signed update goes out and everyone updates their .torrent file with the new foo.c hash, and starts getting the new foo.c file. Anyone joining the swarm with an old .torrent gets notified about the updates and new hashes. Meanwhile, someone attempting to add evil.c to the torrent would not be able to get anyone else to recognize it as a valid block.

Re:gittorrent

[magic maverick]

I might switch to Git (from Bzr) if this sort of thing existed. I would love for a way to anonymously (pseudo-anonymously) host and distribute my software, including changes as I make them. I'm not interested in signing up for Github or similar though, and so I'll continue to use Bzr because it's what I use...

Re:gittorrent

[benob]

I think you've just found how to save rss.

Re:gittorrent

[fulldecent]

So the magnet url changes with each commit?

Re:gittorrent

[akanouras]

First of all, I apologise for the tone of my post... lkcl's posts frequently attract trolls and, posting at a late hour again, my hasty judgement got the better of me. :-(

When you give your bittorrent client a magnet link with a "btih" (BitTorrent Info-Hash) component, it starts looking up the DHT (asking other peers it's already made contact with) for peers sharing that infohash's torrent. As soon as it finds at least one, it connects to that peer ("joins the swarm") using the BT protocol and asks for the torrent's metadata (the.torrent file's info section). It is at this point where it will pop up a window asking you where to save the torrent and/or which files to download from it.

Given the above, one way that "gittorrent" could work would be the following:
1. Each Git object (file/file tree/commit) would be a separate "torrent", identified by its hash. Information about which peers have which hashes (i.e. objects) would be stored in the DHT. Nothing new for DHT so far. With this in place, you could checkout any individual git file/tree/commit and its history (each git commit references its parent).
2. Here we introduce a DHT protocol extension (let's call it "get_hash_addenda") using which you could get information stored in the DHT (this is the major difference with .torrent files) about newer commits. With this in place you would also find other users' "forks". This would also be useful for ordinary torrents by the way (get subtitles/new episodes for this torrent).
3. With these in place, you now have a (very slow) "gittorrent" implementation. Additional extensions/local DHT data caching/assumptions/whatever) would be used to speed the whole thing up. Existing BT trackers (fast peer lookup for a given infohash) would work as is. Existing git daemons would also work, and could also be extended to speak DHT and the BT protocol. All these things map well to existing BT concepts.

The protocol(s) used for transferring objects between peers could be the BT protocol (provided we treat each git object as a separate torrent, as lkcl recommended), and/or any protocol already supported by Git (as an analogue of "web seeds").

Example bare bones magnet links:

magnet:?xt=urn:git:<object id>
magnet:?xt=urn:git:<long hex number representing a user>/<repo>[/<branch|tag name>]

or even magnet:?xt=urn:git:<user>/<repo>/<branch|tag name>
if a way of having non-spoofable and globally unique nicknames is found (probably in an FCFS fashion).

A nice research project on extending BT (which already somewhat implements the "get_hash_addenda" functionality) is Tribler.

The earlier "gittorrent" effort that lkcl criticised can be found here. A cursory glance reveals an emphasis on trackers and almost no mention of DHT, probably due to it being written in 2008.

Disclaimer: All this is from a layman's point of view and horribly inaccurate. The DHT protocol extension especially could probably be avoided using some convention. I've been reading up on how BitTorrent and DHT works lately and, honestly, I can't blame you: apart from the BEPs it's basically UTSL.

Re:gittorrent

[lkcl]

I'm not seeing how DHT will solve the fact that the first time you commit a change to your code, the bittorrent client will detect it as corruption and replace the files with the original version from the swarm since the modified files won't match the hashes in the .torrent.

ok. there's a couple of solutions here. one is to query a number of peers for the same object, obtain its MD5 (or other) checksum and validate them aand the other iiiis...

The least invasive way I can think of to do this with some semblance of security would be some sort of public/private key arrangement that would identify the author of the .torrent file and allow that person to distribute replacement .torrent files through the swarm.

yes. exactly. git allows you to GPG sign tags. the GPG-signed tags would be the key point around which you would verify that you (ultimately) got the right objects. and it's the GPG-signed tags that would allow you to decide to fork an entire project, or upgrade an entire gnu/linux distro, by simply setting a new target to pull and verify against.

the question which i have yet to resolve is: what the hell do you do about all the intermediate commits, intermediate objects etc. etc.? enough idiots trying to corrupt the system would result in quite a lot of bandwidth wasted before you got to the point where the git tagged branch could be verified by MD5 summing.

what i don't quite understand, though, is why git over http (or any other network protocol) doesn't have the same issue. or, is it the fact that there is only one central control typically for a particular branch (or tag) that makes this moot? so you know that you're only going to ever be pulling one git pack-object from that one server, and having done so you're now up-to-date so can do the checksum, bam, done.

I use the bittorrent protocol sync my computers

[runner_one]

I have three computers. Home desktop, Work Desktop, and, a laptop.
I use this newly released piece of software to keep them all in sync. I added a server to the mix as a backup, and now all my data is on four computers. The peace of mind given by having my data automatically mirrored in four locations and the resulting lowered chances of loosing all my data enables me to sleep better at night.

Re:Patent Trolls?

[larry bagina]

It's a honeypot to trap people that can't tell the difference between copyrights and patents.

Damn right

[EmagGeek]

Linux ISOs
VM Images
Backup Images
Home Movies
etc...

Re:Damn right

[tehcyder]

"Home Movies". Nudge nudge. Wink wink. Know what I mean? Say no more, say no more. "Home Movies" eh? Are you keen on videos? Eh? Eh? Got any videos of your wife naked in your "Home Movies"? Like to buy some?

Re:Damn right

[poofmeisterp]

"Home Movies". Nudge nudge. Wink wink. Know what I mean? Say no more, say no more. "Home Movies" eh? Are you keen on videos? Eh? Eh? Got any videos of your wife naked in your "Home Movies"? Like to buy some?

Oh, come on that's totally inappropriate.

You torrent that stuff at the office.

Game download

[Spy Handler]

I got a game called War Thunder via bit torrent. What you do is download a small installer program from the publisher's website and run that. The installer automatically connects to BT seeds and peers and downloads the actual game itself.

There is no other way to get War Thunder. I suppose since they're a small publisher, their web server can't handle distributing the 13 GB game file to tens of thousands of users.

Re:Game download

[damnbunni]

I hate it when developers do this. Offer a torrent as your default download, sure, but put a direct download buried somewhere in your support pages.

Some people (like me) have net connections that just crap out on torrents. I can download a large file reasonably well, but the same file in a torrent will take weeks.

And some people (a majority, in some nations) have caps, and a torrent-based downloader eats into that quite a bit.

Multi site sync

[silas_moeckel]

I have a fairly large and growing (3.7TB) dataset that needs to be replicated nightly to a bunch of different sites. By the nature of the dataset nothing is ever removed or changed just new files added. It needs to be copied out to a half dozen locations that have as much outbound bandwidth as the primary. So a cron job sets up the torrent every night and all the remote sites pull data from the primary and reshuffle it between themselves.

Re:Multi site sync

[wvmarle]

I may miss something, but isn't this the ultimate job for rsync, to copy only the new bits? Or do you take a diff and distribute only that?

Re:Multi site sync

[keshet]

Let me guess.. bitcoin blockchain?

Re:Multi site sync

[hab136]

Rsync would also work, but BitTorrent will be faster.

> a half dozen locations that have as much outbound bandwidth as the primary

Assume all the sites have 10 MB/s inbound and outbound. To transfer 3.7 TB (3,879,731 MB) would take 107 hours per site, 646 hours total.

Using BitTorrent, all 6 sites will take somewhere between 107 and 646 hours - probably around 300 hours, because after one site receives a block, it can send it to the other sites, without using the primary's bandwidth.

The daily diffs will of course be smaller - but the same principle applies. BitTorrent will also verify the existing data and then just transfer the new bits, same as rsync.

Re:Multi site sync

[silas_moeckel]

Since there are no deletes or changes to existing files it just remakes a torrent with all the new stuff. The far end sees the existing bits checks to make sure it matches the source and skips it.

Rsync would require the same data to be sent 6 times from the primary or a convoluted process to syncing from P to A then P to B, while A to C and finally P to D A to E B to F or similar. In any case it's still 3 times as long as a torrent. At 40 ish million files rsync takes 10 minutes or so just verify that 2 sites match over a local gigabit connection (the hardware is meh e3 with 4x sata 3tb in raid 5).

Re:Multi site sync

[wvmarle]

Your comment would be true if they were really pumping around 3.7 TB every day. However, they don't (or at least: they shouldn't have to):

By the nature of the dataset nothing is ever removed or changed just new files added.

This is why I was thinking of rsync. Only transfer the changed bits. Now indeed one could use a combination of a diff tool (check what's new) plus torrent to distribute the updates... whether that extra effort is worth it depends of course on the total amount of data to transfer.

Move my legal data?

[Jeremiah Cornelius]

I let my Lawyers handle that. It's what they're paid for, isn't it?

NO

[ADRA]

Bit torrent is a good data -distribution- tool, not a data -mover-, and it would be lousy to play that role. There are at least a dozen possible open solutions for moving data from point to point, but I have no idea why you'd use a protocol/tool stack that are designed for broadcast/graph distribution to do so.

An off the top list:
      1. NFS
      2. SMB
      3. FTP
      4. SFTP/SCP/rsync
      5. HTTP/HTTPS
      6. sz/rz
      7. iscsi
      8. DFS
      9. AFS
      10. UFTP/XFTP

The question should really be what exactly do you see ad being deficient about all these protocols that deems it necessary to re-invent the wheel yet again?

Re:no

[nabsltd]

I dont have anything worthy of having to mass source it, and my 3$ a month "unlimited bandwidth" website has taken 400 gigs in a month downloads before without a sweat.

That's an awesome price, even if the overall speed isn't that great.

Everything I have found that gives you more than a few GB of disk space and truly unlimited bandwidth costs a lot more than that per month.

Re:no

[Osgeld]

its though hostmonster, they claim right this second 4.95 a month, but to get that price you gotta pay forward for 2 years, I have been with them since ~2006, but yea
"unlimited" bandwidth
"unlimited" space
awesome speed ... for 2006, but still decent now, not had too much of an issue and you can get ssh access for doing routine things (not running programs)

but I am sure there are others out there who are running the same thing, the ONE time I had a 400GB run in a month more than makes up the years of trickle in data

What does THAT matter?

[erroneus]

Punish the technology because of how it is used? I thought we grew past that notion already.

At some point, one of the few remaining ways to get good information and news will be through these rougue channels and methods. Do we have to keep re-hashing the same ridiculous notions? How about we ban types of music based on the fact that thugs and criminals like it and glorify killing?

Re:What does THAT matter?

[cheekyjohnson]

I thought we grew past that notion already.

Where have you been?

Yes indeed!

[briancox2]

Video content mostly.

And secondly, lots of Linux distributions. These serve 2 purposes. They support free culture and minimize the amount of load on the servers of those open source developers that already giving so much. And also, because I enforce encryption on everything, even if my ISP did see what I was torrenting, they'd likely find it was something legitimate. I find nothing lacking really. BitTorrent just came out with an alpha file sync program that seems to be working really well. I'm happy.

Re:In a word? YES!

[Anonymous Coward]

You folks that torrent movies and stuff that is not in the public domain are crazy in my book.

I also drink caffeine after 4pm and watch TV shows that use the r-word. Crrrrrrazy!

Facebook Does

[terbeaux]

Facebook deploys its 4GB binary to its 500,000 servers using a torrent client that has rack and switch affinity. Each client goes for data chunks that are already locally on a rack or switch that it is connected to. That is a crap-ton of data.

rsync transfers changed PARTS of files

[raymorris]

To be fair, rsync + ssh is equally secure as scp. I actually think rsync uses scp in that situation (please correct me if I am wrong).

rsync and scp operate somewhat similarly if the file does't exist all on the destination. If an earlier version of the file exists, rsync transfers only the changes. Therefore it couldn't use scp - rsync does things that aren't possible with scp.

Since sometimes you have to use rsync because scp can't do it and I don't care to memorize redundant options, I normally use rsync even with scp would do.

Re:rsync transfers changed PARTS of files

[denmarkw00t]

rsync only does incremental copies when you tell it to, and, technically, the same thing could be achieved through ssh and scp (using ssh to check for the destination file first and then copying with scp if needed). Of course, rsync does make it easy; just because it's easier doesn't mean it's not technically possible with scp

The big advantage

[tdelaney]

The big advantage of using BitTorrent over many other protocols for moving large amounts of data (as opposed to distributing it) is reliability - or lack of it. When you're moving large datasets, you don't want it to crap out and not be able to resume 90% in.

Sure - rsync has the ability to resume, but it requires explicit command line options. It's a terrible feeling to realise you just restarted a 10GB+ transfer instead of resuming it.

SOLR at Etsy.com?

[rwhiffen]

The folks at Etsy do it to replicate SOLR:
http://codeascraft.etsy.com/2012/01/23/solr-bittorrent-index-replication/

Not sure if that's what you mean.

+1 Linux distros. Only for multiple recievers

[raymorris]

Same here, I download and seed Linux distros.

with just two end points, it wasn't better

For point-to-point transfer to large amounts of data, the protocol does't matter, as long as the protocol is sane. The time spent moving data bytes will be much higher than any protocol overhead. rsync is roughly optimal because it won't transfer portions of the file that the receiver already has. BitTorrent is for distributing data to many destinations.

Red vs Blue

[Tarlus]

Back in the pre-YouTube days, Rooster Teeth distributed their videos using Bit Torrent to relieve their own HTTP load. I think they gave BT users the incentive of downloading earlier to encourage its use.

Re:Patent Trolls?

[Nocturnal Deviant]

i tend to run through a Romanian based VPN almost exclusively for work(my own personal. I wouldn't trust an outside vps with corporate information) related activity, and i can absolutely confirm what he is saying. I've been on the phone with my vpn provider before(hes an old personal friend), and he says "OH BOY MAIL TIME, now i get to have more firewood from your american lawyers" as to which we both chuckle.

All data is legal

[zoloto]

somewhere in the world, whatever you're moving, it's legal.

no

[Osgeld]

I dont have anything worthy of having to mass source it, and my 3$ a month "unlimited bandwidth" website has taken 400 gigs in a month downloads before without a sweat.

About the only thing I could think of is a linux distro, but again the only thing I am bound to cobble together is a lightweight debian mix for maintance / repair / recovery when someone brings me their computer to be fixed and winders is all trashed.

Not really a fan of it.

[UltraZelda64]

It's a good idea for sharing, but I'm not a fan of the way it loves to hog the Internet connection, it starts connections out the ass, flooding the network and just slows everything down. I tend to use wget for pretty much everything; with a decent server, it gets by just fine with only one connection. I also like the idea of maintaining at least somewhat-accurate timestamp data whenever possible, and BitTorrent doesn't seem to have a concept of that. And also, I like to maintain my own checksum files, so BitTorrent doing that itself is extra functionality that I don't need. Not too crazy about the serious fragmentation it can cause, and many clients just do not pre-allocate disk space or even have an option. The bigger the file, the better BitTorrent works--but unfortunately that little fact kind of stops it in its tracks; a file of many gigabytes, downloaded in thousands of random pieces, is bound to end up fragmented to hell and back if its space is not pre-allocated.

However, if there is no FTP/HTTP link, I won't hesitate to use BitTorrent. I just have a tendency to download everything I get from that protocol to a different drive than its intended destination, so the final move will "defragment" it.

Re:Not really a fan of it.

[BitZtream]

If you're using a bittorrent client that doesnt' preallocation the file, then you should switch to a non-shitty one that preallocations, avoiding the entire fragmentation problem completely.

Or you could use a filesystem that didnt' suck.

Re:Not really a fan of it.

[wvmarle]

Defragging on any modern FS is easy: just copy the file to itself. That should do the trick to get a nicely defragmented file (assuming you have a long enough chunk of free space on your disk, of course).

Re:Not really a fan of it.

[magic maverick]

Regarding your first complaint, you do realize you can limit and throttle the connection?
E.g. where I am, my Internet connection is rubbish, so, during the day I throttle bittorrent to 10KB/s up & down. But, then, during the night, I let it run a little faster. It takes forever to actually download something large, but it works. And for some material, I'm one of only two or three seeders, so people downloading from me can damn well be grateful for what they get (and not complain about the horrendous speeds).

But then again, I'm probably just feeding a troll, seeing as you have a lot of similarly stupid complaints in your post.

Re:Not really a fan of it.

[Electricity Likes Me]

There are a lot of people out there using SOHO internet routers which are hard capped at 2500 NAT sessions. BitTorrent can easily take down the network by blocking outbound DNS until some of the sessions are dropped. It's an amazingly annoying problem, and it's not a well-advertised feature of routers (and getting over 10k is hard to find still for some reason).

Personally I find it staggering that a Linux box will handle 48k+ sessions no sweat on pretty much any type of hardware, but we're still well below the IPv4 port range in terms of allowable NAT sessions on most hardware.

Re:Not really a fan of it.

[magic maverick]

You know you can limit the number of peers you connect to as well?
E.g. I have it set to a maximum of ten torrents active at any one time. Each torrent can have no more than 60 peers. So, total, 600 peers maximum at any one time. Except, I also have it set so that I do not connect to more than 275 peers, ever. So that 600 number, irrelevant.

I.e. these are all problems that, well, aren't.

Re:Not really a fan of it.

[UltraZelda64]

While you're all saying "you know you can do this" and "you can do that too" to optimize your BitTorrent usage and make your network and Internet connection run decently , I can sit here downloading files at the maximum speed my connection will allow, with just one connection per download. Limit the rate of individual downloads, or better yet just queue them and they'll get done just fine. The problem with BitTorrent is NOT that clients come with sub-optimal settings (although they do) or that they need to be optimized for the local area network and Internet connection speed available (which is, again, true), but that they typically *require* you to do a decent amount to get good performance. Not just the seeding/downloading balance, but number of peers/connections (per torrent, total), speed limits (per file, total), etc.

Re:Not really a fan of it.

[UltraZelda64]

Linux, ext4, Transmission. I have found no compatible BitTorrent clients that I would want to use, and I wasn't aware that Transmission was "shitty" compared to its competition. I also was unaware that the ext4 file system "sucked."

Last I checked, it still produces ~100 fragments per file, though maybe this has been fixed since I last used it. It has been a while though. (Hey, if something else seems to work adequately or even better for my needs and preferences, why bother using it?)

Supposedly there is a mysterious "preallocation option 2" for the program that fully pre-allocates the space before writing anything according to a forum topic I just found, but I have no idea how to activate it: https://forum.transmissionbt.com/viewtopic.php?f=4&t=10013

That would at least solve the fragmentation problem (probably the biggest one), but the others would still remain. Lack of original timestamp would still be an annoyance, but the big problem would be network hogging--which, to be fair, can be somewhat alleviated with a lot of tweaking. But then, the lack of necessary tweaking is the entire reason I still use wget to download all files over FTP/HTTP in the first place.

[Pre-post discovery: The file ~/.config/transmission/settings.json houses that option. Nice way to hide it from view... without Google I would've never known it even existed. Pretty much everything else you might need is in the program's GUI/options screen. It looks like they pulled a GNOME here.]

Moving data with torrents

[woboyle]

I often download Linux DVD images and such using bittorrent. Bittorrent isn't "evil" - it is simple a means to share the load of moving large amounts of data. To make it illegal should be considered something akin to making automobiles and pickup trucks illegal - as they can move both large amounts of legitimate as well as other goods from point A to point B, and they can use many routes to get to B.

Only Legal torrents

[Vapula]

I use Torrent for many legal uses :
- Blizzard downloader (WoW, Starcraft2, Diablo 3)
- Humble Bundle
- Linux Distributions

But also, from time to time, some free tools, some F2P games that use BitTorrent for distribution, ...

Re:In a word? YES!

[bipbop]

Raffinate?

mimics my experiences

[erikscott]

I agree: torrent can't really saturate a 10GE... for that you should see something like bbcp, which will quite handily flood a 10gig ethernet and then some. :-)

NC State University uses torrent to let students download some commercial software so they don't have to hand out DVDs... they distribute SAS that way for certain, probably a few others.

ibiblio had someone who developed sort of a "perma-seed" to use torrent for some sort of archive-like thingie. I know Paul Jones is probably reading this, perhaps he would like to comment? :-)

Re:mimics my experiences

[swillden]

I agree: torrent can't really saturate a 10GE

Point-to-point torrents can't saturate a 10GE. Get enough nodes involved and they can.

Re:mimics my experiences

[wagnerrp]

That's stupid. Why use bittorrent when a standard http server would do just fine? It's not like they don't have nice servers sitting on the campus backbone ready and available to do precisely that.

Re:mimics my experiences

[adolf]

A better question is: Why not?

Some BT clients will seek out local peers, which helps people who are quite near eachother (in the same building, say) while reducing server and inter-building network load.

For free, more or less.

"Hey, boss! I might be able to reduce the load on our server-box by maybe 40% on install day, and keep things from grinding to a halt!"

"Oh, really -- using that BitTorrent crap? Forget it, lackey - we'll just use HTTP like everyone else. After all, the server is on the backbone, so it's not like it matters. The students can just wait if things get bogged down."

"But boss, with BitTorrent, things get faster as more people use it, and it can't grind to a halt."

"HTTP. We're done here."

Stupid? Yeah, what you said was pretty stupid.

Re:In a word? YES!

[sdreader]

You folks that torrent movies and stuff that is not in the public domain are crazy in my book.

Just out of curiousity, why?

Movies and TV programs.

[godel_56]

I've read that in Australia both the big, legitimate movie distributors and the major TV networks use (or have used) torrents to move programs around, between theaters (with digital projection facilities) and networked TV stations.

Yup, lots, pretty much always legal.

[seebs]

I pretty much always use torrents to download OS installers (Linux, etc.). I don't know whether I've ever downloaded anything that wasn't totally legal. Probably have at some point, but I mostly use torrents for things where I want to be contributing bandwidth back to the community.

Guns, drugs, ...

[Ungrounded Lightning]

Punish the technology because of how it is used? I thought we grew past that notion already.

If we had you'd be able to buy any weapon or drug you want without government interference or oversight.

The NRA could go back to its original functions of training and research, and the FDA and DEA to could be replaced by Underwriters' Laboratories and Consumers' Reports.

As you can see we have a long way to go.

Re:Guns, drugs, ...

[Ungrounded Lightning]

Seems to me the two issues are poster children for massive instutionalized policies of attacks on a technology rather than the misuse of it.

Massive to the point of billions, perhaps trillions, of dollars wasted and far more damage done that prevented.

Think you've seen massive overreaction to misuse of torrents? Compare it to "The War On Drugs" and you ain't seen nothing yet.

Re:In a word? YES!

[Anonymous Coward]

Man, you really just did that, didn't you?

Witchcraft!

[seanthegeek9306]

Look at the mobile app. An iOS UI on a Galaxy Nexus. What witchcraft is this?

Yes: Linux distros

[cybersquid]

Mostly SuSE. Some kubuntu, CentOS and Oracle OEL.

Wait...

[ericdano]

Wait, you can move illegal data using torrents? Who would have thunk!?!

Haven't used Torrents in years. Many years.

Re:+1 Linux distros. Only for multiple recievers

[thereitis]

That's not true: Some protocols are very "chatty" and may require, say, 1 round trip acknowledgement per file transferred. This is where rsync's streaming protocol shines.

Even at home I noticed that copying large files over wireless via SMB is much slower than copying them over an SSHFS mount (getting previously unseen transfer rates, actually). However, the SMB mount is more responsive when exploring files.

In my experience, protocol can matter a lot.

Yep.

[mindbooger]

Linux distros. After I'm done, I try to keep seeding for the next night or two to try to "give back" at least a little bit.

yes I do.

[davydagger]

many open source projects with large downloads like .ISOs for linux distribution use torrents because paying for bandwith would be really expensive for direct download.

bit torrent is great, because everyone with consumer broadband becomes and instant mirror when they download it.

Re:In a word? YES!

[jones_supa]

Actually, it's funny when you think about it. Legally freely distributable open source software is important for folks here, but the same people pirate copyrighted movies and stuff. But if you instead torrent culture that is in public domain, there is no legal problem and you can enjoy completely free digital lifestyle.

Re:Can you use it to move a 300 megabyte hosts fil

[jones_supa]

The $10,000 challenge!

Re:Yes, to distribute software on a closed network

[BitZtream]

'custom software' ... right because there aren't any libraries for multicast file distribution already or anything.

You fail at the Internet. Multicast file distribution is probably older than you are.

Re:+1 Linux distros. Only for multiple recievers

[wagnerrp]

I wonder how much of that is poor CIFS implementation or configuration. I can do 115-120MB/s either way between my (Win7) desktop and my SAMBA server in my basement.

Re:In a word? YES!

[wvmarle]

Very, very bad analogy as the concept of copyright is a good thing, the concept of racism is not.

Remember that only thanks to the existence of copyright the GPL and all those other FOSS licenses can exist. Without copyright it'd all be public domain.

It is the current implementation of copyright laws (primarily: too long terms) and all the other laws that surround it (e.g. DMCA blocking reverse engineering and circumvention of DRM) that is the problem.

Re:In a word? YES!

[Anonymous Coward]

Man, you really just did that, didn't you?

Uh, yes, he really just pointed out that GGP's implicit message that illegal==insane was fucking rubbish.

If GGP had a more nuanced point, such as that violating copyright is insane because copyright law is demonstrably just, maybe he should have made that point, and then nobody would have demonstrate the non-link between legality and sanity by pointing out examples of laws even more egregiously unjust than copyright.

Microsoft ISO's

[batwingTM]

We use bittorrent to move MS ISOs at work. We are an Educational institution and as such we are allowed to offer students access to MS products. (not the Office suite, but OS's .NET and the like) This is done via the Dreamspark site where students can get their license keys but from our internal network they can use bittorrent to get the ISOs

So we legally give access to MS OS via torrents (^_^)

Re:+1 Linux distros. Only for multiple recievers

[Drakonblayde]

BitTorrent is, conceptually, best used as a reimplementation of multicast. Multicast is probably far more efficient when it comes to the actual data distribution, but multicast (specifically, routing multicast) is one of those blackbox things that not a huge number of people understand. Last time I checked, I couldn't route a multicast source from a Comcast connection and have the data arrive on a receiver on the Cox network.

However, there was still a need for a protocol that could effectively do one to many distribution for the masses. Enter bittorrent. It fills the void nicely, and has the additional benefit that once a receiver has all the data from a given source, it then becomes a source itself, thereby increasing the aggregate bandwidth available for the feed, as well as making the feed resilient.

1 RTT per GB? For a Maildir mail spool, yes

[raymorris]

The GP referenced isos and "10s of terabytes of data". Unless those tens of TB are BILLIONS of tiny files, 1 RTT per file would be less than 0.01%. For lots of tiny files, like Maildir, yes SMB sucks. Of course SMB is Microsoft, so the fact that it sucks is assumed.

TFS

[thatkid_2002]

"I sometimes use torrents to move several GB of data, especially when pushing large bundles to multiple destinations."

Which, to my knowledge - Bittorrent achieves better than any in your list of 10. If I had to guess, the part about pushing to multiple destinations is the first part of the main reason Bittorrent was chosen. The second part is that once data is uploaded to another node, you have *two* links to supply uploads to further nodes - and then three, four, five... When moving data across a non-uniform network topography this is useful.

New FreeBSD Install ISOs

[utkonos]

Torrent has been the only way that I've downloaded FreeBSD ISOs when they're released for at least 5 years. It is lightening fast compared to even the fastest HTTP or FTP mirror.

Yes, low budget film

[ssam]

I was involved in making a low (GPB2000) budget film, which was released with a CC-BY-NC licence. For distribution we used youtube, vimeo, archive.org and bittorrent. Youtube was by far the most popular option for downloads (over 90%), but bittorrent was the only method that worked for the highest quality (~20GB 1080p file). we set up our own tracker, because we could not get clearbits to work. we seeded from a friends colo server, VPS and a few home broadbands.

Re:In a word? YES!

[number11]

Legally freely distributable open source software is important for folks here, but the same people pirate copyrighted movies and stuff. But if you instead torrent culture that is in public domain, there is no legal problem and you can enjoy completely free digital lifestyle.

You mean, culture that is more than 70 years out of date?

Podcasts?

[wagonlips]

Wouldn't bittorrent work well for podcasts? Seems like the perfect case but I've never heard of it being done.

Re:In a word? YES!

[Zontar The Mindless]

But if you instead torrent culture that is in public domain, there is no legal problem and you can enjoy completely free digital lifestyle for all of about 10 minutes before you realise that you've consigned yourself to a life of bad fanfic, worse poetry, and unimaginably bad amateur electronica courtesy GarageBand. And then kill yourself.

SYSTHLSOTET...

Re:In a word? YES!

[Zontar The Mindless]

Yes, he did, and I should like to offer him a cigar.

Re:BitTorrent is a silly way to move files

[gl4ss]

because he doesn't want to build his own broadcasting protocol.

it's a good way to move data from one target to multiple recipients unless you're in total control of the whole network flow.

Copyrighted material

[loufoque]

I move around a lot of copyrighted material.
This is allowed by fair use of course.

Re:Copyrighted material

[Fruit]

posting to undo moderation. sigh.

In France...

[Scotch42]

...the so called "freebox" (ADSL modem with lot of extra features) includes a bittorrent client. I use it to get and share the full gutenberg project.

Yep

[spxZA]

Bioinformatic data, Linux distros, F/OSS games, plus I wrote a libtorrent client to distribute multimedia for a digitical signage company.

1 file & multiple Internet connections

[xororand]

BitTorrent is great for transferring large files with multiple slow Internet connections.
Seed with each connection, leech with the destination machine.

Humble Bundle

[cgimusic]

The Humble Bundle HTTP downloads are a little on the unreliable side. I often get a "completed" download and then the MD5 doesn't match and I have to redownload. Now I use Bittorrent and get a perfect file every time.

Re:Humble Bundle

[apcullen]

Agreed! I always use bit torrent when I buy a humble bundle.

Re:BitTorrent is a silly way to move files

[serviscope_minor]

The only purpose it serves is to prevent leechers from taking and not giving back while moving your warez around.

Yeah and you're a drunk driving pedo terrorist.

You see if you can make up random facts about people then I can too!

[looks at torrent list]

CentOS-5.9-x86_64-bin-DVD

Ah yes, CentOS, that well known piece of warez.

Re:I really want to use it for software deployment

[Electricity Likes Me]

BitTorrent really needs some better client-side tools to make use cases like this more common.

Though that does get me wondering if you couldn't modify say, Nautilus, to automatically export torrent files (or generate on the fly torrent files) of files available on computer shares and keep a track of them so it would automatically use the torrent to download them from a local machine.

Almost

[DrYak]

To be fair, rsync + ssh is equally secure as scp. I actually think rsync uses scp in that situation (please correct me if I am wrong).

Almost. Technically, ssh (for shells), scp (for files), sftp (files too), and rsync+ssh, all run over a secured openSSL connection.
What change is what is running on the other side.
By default, the remote SSH just runs a shell and pipe its I/O to you local terminal, through a secure connection.

The other combinations all launch a specific program at the remote end (either by asking for it, or in the case of sftp, as specified in the server's configuration) which handles the remote part, while communicating over the same kind of secure openSSL connection as ssh does in shell mode.

So rsync uses rsync, not scp, but it does it over an encrypted ssh connection anyway.

scp vs. rsync

[DrYak]

...and if the connection breaks or anything happens, you'll have to restart copying everything from the beginning.

meanwhile rsync is able to do only partial sends.
That means rsync will only send the parts which were not transferred the first time.

that's also useful for updates, for incremental backups, etc.

Re:scp vs. rsync

[denmarkw00t]

...and if the connection breaks or anything happens, you'll have to restart copying everything from the beginning.

Interestingly, rsync can even resume a transfer that was started by scp but then interrupted - neat!

Re:Patent Trolls?

[Opportunist]

I think you mean copyright trolls.

Believe me, if you could somehow find a patent that could impede the use of P2P networks, the RIAA would already have paid $whateveryouwant dollars to get their hands on it, to make sure it will only be used in ways that ensure they can easily and cheaply spread content but nobody else can.

Speaking of scientific application

[DrYak]

Tranche is another peer-2-peer distribution software which has become the almost official standard for the huge masses of data found in life science (which, although aren't LHC-grade abominably over-massive, still can clog networks and would be a pain to distribute using classical server methods).

Re:Can you use it to move a 300 megabyte hosts fil

[Anonymous Coward]

I don't get it.

Re:In a word? YES!

[tehcyder]

You folks that torrent movies and stuff that is not in the public domain are crazy in my book.

"You black folks that ignore corrupt laws and sit at the front of the bus are crazy in my book!"

I think you've hit a new low in the old slashdot game of "let's make a stupid fucking analogy".

Re:Music and Linux Distros

[tehcyder]

I download a whole lot of music, a whole lot, as there's an endless supply of free legal music available.

Yes, in the same way that there's an endless supply of free fanfic available, i.e. utter shite composed in someone's bedroom while they're half drunk and masturbating to Japanese child porn.

It would be used even more

[apcullen]

If there were a way to make it ad supported. A lot of android developers and minecraft modders seem to use ad supported lockers, which are a pain and way less reliable than a torrent.

Re:In a word? YES!

[bobbied]

The chances of getting caught by a patent troll firm is pretty high because it is extremely easy to find you by IP address. Of course, it depends on where you live. Here in the US it's pretty risky, other countries may not have a legal system that is as easy to use by patent trolls.

Of course you *could* go out of your way to side step their ability to find you by IP address, but when you do, you just bypassed the benefit of using a torrent to get the material and would likely get it faster by just transferring it direct and bypassing all the risks of a Torrent.

So I don't engage in distribution of stuff that doesn't come with a license that allows me to make copies and distribute. First, it's too legally risky and more importantly I don't think it is morally and ethically right....

Que loud discussion about the moral and ethical nature of patent and copyright laws by the "everything should be free but keep sending my paycheck to me" crowd in three, two, ......

Re:Patent Trolls?

[poofmeisterp]

Does this seem like fishing expeditions by patent trolls?

No need to go fishing here.. Just find the torrent tracker and connect... Volia, you have a list of everybody who is distributing the material by IP address. It's all about tracing down the IP's and sending out the collection letters and cashing the checks from there, assuming you actually have the permission of the copyright holder to do so...

...But you don't have the psychological "why" or data to back up your court cases. Data like "We have 1200+ answers to the question 'what do you use torrent for other than pirating' and got nothing other than 'a lot of pirating' or 'I download Linux distros'.

Yes, it's possibly a fishing expedition. Gove 'em some good info!

Re:Patent Trolls?

[poofmeisterp]

Does this seem like fishing expeditions by patent trolls?

That was my first thought, and I'm responding appropriately to what is either a.) a data gathering exercise or b.) an indication that the person asking doesn't know how to get out of the bed unless a manual tells them which side to roll over on each morning.

Re:In a word? YES!

[poofmeisterp]

Raffinate?

(r)egistered AWESOME!

Re:Can you use it to move a 300 megabyte hosts fil

[poofmeisterp]

inquiring minds want to know.

Let's test it to find out. Just make sure the hosts file name has .iso at the end.

Re:In a word? YES!

[SiChemist]

I think you are confusing/conflating the terms patent and copyright. As for being targeted via IP address, just use a VPN. It's not rocket surgery.

Re:In a word? YES!

[sdreader]

The chances of getting caught by a patent troll firm is pretty high because it is extremely easy to find you by IP address. Of course, it depends on where you live. Here in the US it's pretty risky, other countries may not have a legal system that is as easy to use by patent trolls.

I live in Australia and although some ISPs have passed on warnings, my ISP (Internode), which is one of the largest in the country, does not. Honestly it's counter-productive for them to do so - if people were being threatened for downloading copyrighted material, they're probably end up moving to a small download quota plan since a large portion of what they'd use it for is no longer safe to do so. Lower quotas = cheaper plans = less money for ISP.

I don't think it is morally and ethically right....

To be honest, it a lot of cases I agree with you. Despite the fact that copying content is trivial, that does NOT make it worthless in a monetary sense and I believe in a fair exchange of money for product model, so long as its reasonable. But when some (old, obscure) content is not even available legally anymore, or not available in your country because no-one can be bothered to sell it to you... that's when the lines of morality and ethics start to blur.

Re:In a word? YES!

[sdreader]

It's not rocket surgery.

Shit, I'd LOVE to have some rocket surgery!

Humble Bundles

[deek]

Amongst the other uses already mentioned, I use bittorrent to download Humble Bundle games. Faster for me, less of a load on the host server. We all benefit.

PARTS! PARTS of files

[raymorris]

scp has a flag to transfer only the 1KB part that changed within a 100 MB log file? And FYI you're simply wrong about rsync. In fact, the default, when copying files over the network, is to not even copy modified files, only the modified parts. You have to use --whole-file to change that.

Useful for distributing scientific data

[one-egg]

I use BT to distribute large files from the SNIA IOTTA Trace Repository (http://iotta.snia.org/). Although there are typically no swarms, BT is still useful for a number of reasons, including in particular the ability to manage large collections of related files and the ability to deal with intermittent connections.

Unfortunately, many of my users work at sites that block BT, forcing them to revert to a horrible HTTP option.

And no, rsync isn't a solution for our situation.

As to what is needed, the primary thing is better tracker and seeder daemons. I use opentracker, which is OK but hardly perfect. I seed with deluge because it's one of the few seeders that can be run as a daemon (almost all BT clients expect you to dedicate a GUI window to them or they stop running--imagine what running a Web service would be like if you had to have a GUI for every instance of Apache).