Ask Slashdot: Do You Move Legal Data With Torrents?

An anonymous reader writes "We've recently seen a number of interesting projects come from bittorrent.com, including Sync and SoShare. I sometimes use torrents to move several GB of data, especially when pushing large bundles to multiple destinations. It's mostly a hodgepodge of open source tools, though. Apart from anecdotes and info from bittorrent.com, details are thin on the ground (e.g. the Blizzard Downloader). I have two questions for the Slashdot community. 1) Do you use BitTorrent to move data? If so, how? i.e. What kind of data and what's the implementation? 2) If you've looked at torrent clients/tools, what's missing in the open source ecosystem that would make it more useful for moving around large blobs of data?"

Comments

I use it for linux distributions

[Qzukk]

The entire point of swarm topology is to move data to a lot of places at the same time. If you just need to get data from A to B without sharing it with anyone else, rsync it.

Re:I use it for linux distributions

[bobbied]

rsync? I'd use SCP myself, it's a whole lot more secure and the data goes encrypted. That way you won't accidentally share it with anybody while it's in transit.

Re:I use it for linux distributions

[TooMuchToDo]

While working at Fermilab on the LHC CMS data taking team, I used bittorrent to speed up re-installs of thousands of worker nodes. I was able to saturate 10Gb Ethernet links this way, and could reinstall ~5500 Linux boxes within 10-15 minutes (with only two initial OS source servers).

Yes, Bittorrent is not just for piracy.

Re:I use it for linux distributions

[corychristison]

To be fair, rsync + ssh is equally secure as scp. I actually think rsync uses scp in that situation (please correct me if I am wrong).

The advantage I see rsync having is it is useful for automated (backups) of a large collection of files vs gzipping and copying via scp.

Although git also does a great job of that with concurrent revisioning built in.

It all boils back down to using the right tool for the job.

Re:I use it for linux distributions

[Anonymous Coward]

So does rsync, it uses ssh for remote xfers

Re:I use it for linux distributions

[Anonymous Coward]

Likewise.

I've gotten odd looks from my co-workers when I mention that I use bit torrent to download ISOs because it's faster. In most people's minds, bit torrent is synonymous with piracy.

Re:I use it for linux distributions

[Anonymous Coward]

That's because they're imbeciles. Not a big surprise, though, is it?

Re:I use it for linux distributions

[Anonymous Coward]

I nearly jizzed in my pants reading this. Nerdgasm.

Re:I use it for linux distributions

[Anonymous Coward]

I want to get data from A to B as fast as possible ... but B can not accept it at the rate I can send it ... so I send it to C,D,E,F,G ... and then they sync/send to B when B can catch up and A dropped offline.

Re:I use it for linux distributions

[bobbied]

It all boils back down to using the right tool for the job.

The wise keep as many tools in their tool box as possible. You seem to be among the wise.

Re:I use it for linux distributions

[Sloppy]

rsync? I'd use SCP myself, it's a whole lot more secure and the data goes encrypted.

Speaking as someone who, at any moment, only has to hit up arrow a couple times to see an "rsync .. -e ssh ..." line, I'm telling ya: it's all the same.

Re:I use it for linux distributions

[Tarlus]

Not by default, though. You have to use a flag.

Re:I use it for linux distributions

[Anonymous Coward]

Broadcast / multicast would have been a better solution.

It wouldnt have saturated your pipe, and only taken 10 seconds or so (4GB image * 8 b/B / 10Gb/s * 3 for overhead).

Re:I use it for linux distributions

[Anonymous Coward]

I was one of the leechers, and yes, it worked great!
Oops, was I not supposed to say that? :) :) :) :)

But, because of the distributed nature of the torrent protocol,
all of the peers share as efficiently as they can with each other
because (eventually) all of the peers themselves become seeders.

Honestly, I can't easily see a way of doing something like that
with any of the other tools out there - rsync, scp, tar w/ ssh, etc.

Re:I use it for linux distributions

[cheater512]

rsync will use SSH so same protections.

It also will only copy changed data which is much better for backups or archives.
scp will recopy everything no matter what.

Re:I use it for linux distributions

[TheCarp]

$ env |grep RSYNC
RSYNC_RSH=ssh

Worth putting right in /etc/profile so anyone who doesn't want it can disable it if they want.
It is an entirely sane default.

Re:I use it for linux distributions

[Anonymous Coward]

you're forgetting sync issues - what happens if one of the nodes looses a packet, do you stop the send and go back to that point? do you forget about that node and just keep transmitting?

Re:I use it for linux distributions

[PopeRatzo]

I was able to saturate 10Gb Ethernet links this way, and could reinstall ~5500 Linux boxes within 10-15 minutes (with only two initial OS source servers).

I love it when you talk dirty.

Could you just speak a little slower?

Re:I use it for linux distributions

[Anonymous Coward]

The node can just re-request a specific packet with point to point communication.

Re:I use it for linux distributions

[DragonTHC]

That's not the entire point.

If you want to save bandwidth and still distribute your data, then crowdsource your downloads with bit torrent.

Linux, MMO games, game mods, etc. All excellent uses of bit torrent.

Re:I use it for linux distributions

[Anonymous Coward]

I did something similar - Linux live environment, torrent is created against block devices partitions (or whole disk) and then automatically served from the node that created the torrent. Copying the image to a central server was as simple as starting a torrent client & pulling the data down from the master torrent source. And the clients that needed the image just PXE boot the live environment, load the torrent & torrent client and a while later it's done. Every node receiving the image is also an image source. I do this with both Windows, Linux & dualboot. Largest torrent is 110G.

Don't confuse this with public torrents / trackers - this was all privately hosted.

And for anyone that says "use multicast for this", the network I do this in (500+ desktop nodes get the image), doesn't support multicast. And besides, once I had this all up and running, it has it's benefits. Torrent client hashes the block device first and only requests the blocks that don't checksum correctly. The resumability of rsync (kind of) with the scalability of a torrent network.

One thing I didn't exploit, but that it would also fit well with - doing this when there is a bottleneck in the datapath, and having a localised preference for leechers, so the block only has to come across the WAN once. I don't think rtorrent supported it when I put this together (about a year ago) and I didn't need it anyhow.

Block level imaging may not be preferred these days (it has it's downsides), but if you have to do it, torrents are a very effective & scalable method.

cgb

Re:I use it for linux distributions

[Anonymous Coward]

export RSYNC_RSH=ssh

and you're good to go. Secure xfer, and rsync will only xfer the changed parts of a file. Way better than SCP if you're doing large #s of files in multiple directories, etc.

Re:I use it for linux distributions

[nabsltd]

The entire point of swarm topology is to move data to a lot of places at the same time. If you just need to get data from A to B without sharing it with anyone else, rsync it.

One huge advantage of bittorrent is that error checking/correcting is built in.

Although you would still need to re-download the block with the error, it's a very small amount of data (usually a few KB). This solves the problem of errors that happen after any data transport verification, as most bittorrent clients can be configured to do a final re-check after the torrent is complete. In addition, the data transport can be encrypted if you need it to be. Although it's not the strongest of encryptions, it would stop most casual snoopers.

Re:I use it for linux distributions

[Anonymous Coward]

I do not think scp and rsync are even in the same category lol
Here ya go: rsync -plarvUs -e ssh /tmp/file server:/tmp/file
It rsyncs over the ssh protocol.
Scp would simply copy over the ssh protocol.

Re:I use it for linux distributions

[Anonymous Coward]

How would you use broadcast or multicast to distribute an OS? Call me ignorant, but how would you do that in practice?

Re:I use it for linux distributions

[Isaac Remuant]

Probably the fastest way to get your latest linux distro (and Libre Office).

Re:I use it for linux distributions

[i.r.id10t]

Yup, it is one of the few ways that darn near any user out there can contribute back to Open Source.

I can code, some, but not very experienced with c/c++ - just haven't had the need to do it. I hate writing documentation. I do file bug reports, but the stuff I use is pretty darn stable.

So, to give back, I seed iso images for 24 hours or 100gb in upload of any Linux distro release notice I see here on slashdot, even if I don't use that distro.

Re:I use it for linux distributions

[wagnerrp]

How would you use bittorrent to distribute the OS? Hint... it's something a bit more complex than your typical system BIOS.

Re:I use it for linux distributions

[Anonymous Coward]

"Nearly"? Hand in your geek card.

(But wipe it off first, please. I ain't takin' no chances.)

Re:I use it for linux distributions

[fluffy99]

How would you use broadcast or multicast to distribute an OS? Call me ignorant, but how would you do that in practice?

Pretty easy to setup using Windows Deployment Services
http://www.windows-noob.com/forums/index.php?/topic/452-how-can-i-multicast-an-image-in-windows-deployment-services-windows-server-2008/

Or if Linux is your preference
http://www.udpcast.linux.lu/

Or Clonezilla has a multicast restore function
http://clonezilla.org/clonezilla-SE/use_clonezilla_live_in_drbl.php

Re:I use it for linux distributions

[wagnerrp]

Use multicast for this.

Stop using shitty networking gear that squashes multicast traffic. Use erasure or fountain coding to transmit your datagrams so clients can compute ones they missed. Have clients individually request any remaining lost datagrams over unicast.

Torrents are a very specific solution to a very limited problem in networking; specifically, people being too cheap to invest in proper co-located servers for on-demand traffic. Stop trying to apply it everywhere else.

Re:I use it for linux distributions

[kcbnac]

Here's how Norton did it, back in the late 1990's: http://www.symantec.com/business/support/index?page=content&id=TECH106806

I recall using this in college, in 2003, to reimage our 'learning' workstations. (After we'd break them, like discovering that Windows 98 SE would let you format the OS volume, and not crash.)

Re:I use it for linux distributions

[Anonymous Coward]

The entire point of swarm topology is to move data to a lot of places at the same time. If you just need to get data from A to B without sharing it with anyone else, rsync it.

One huge advantage of bittorrent is that error checking/correcting is built in.

Except that rsync does that, too -- it's not an advantage if both options have it!

In addition, the data transport can be encrypted if you need it to be. Although it's not the strongest of encryptions, it would stop most casual snoopers.

Except that rsync DOES THAT TOO, in the ubiquitous usage pattern with RSYNC_RSH=ssh in your environment.

So, no, neither of those are advantages for bittorrent.

Re:I use it for linux distributions

[TooMuchToDo]

You don't multicast images to Linux machines when you're using configuration management tools. PXEBoot->Bare Image Install->Puppet system configuration upon first boot based on machine grouping/criteria.

Disclaimer: I'm OP.

Re:I use it for linux distributions

[grcumb]

$ env |grep RSYNC
RSYNC_RSH=ssh

Worth putting right in /etc/profile so anyone who doesn't want it can disable it if they want. It is an entirely sane default.

I don't think that's required any more - not on Linux or Mac OS X, anyway. I use rsync several times a day and each time it just reads my ~/.ssh/config file for the options, sets up the connection and performs the transfer without any fuss or bother.

I haven't set the RSYNC_RSH env parameter since about 2002.

Re:I use it for linux distributions

[ArsonSmith]

This seems very interesting. I'm doing a much smaller setup now with NFS and running the OS in RAM. Even with only about 40-50 servers I can get boot storms on the pxe boot server. I may have to look into this as an option. If I can get a bittorent client into the initrd...hmm time to start playing.

Re:I use it for linux distributions

[tlhIngan]

There was an article on how Facebook did their software updates. First they compiled their PHP into C, then compiled that to produce a binary that's several gigabytes in size (with all the resources and stuff).

That binary is then sent to their datacenter via Bittorrent so it can replicate itself across all the servers efficiently. The final stage is to bring down the node, swap the binaries and bring it back up.

http://agilewarrior.wordpress.com/2011/05/28/how-facebook-pushes-new-code-live/

There was another article on it from Facebook, but I can't seem to find it.

Re:I use it for linux distributions

[Anonymous Coward]

He practices the pinch technique.

Re:I use it for linux distributions

[ArsonSmith]

If you download pirated files then the torrents win!!

Re:I use it for linux distributions

[gullevek]

Yes, via the -e command. I wouldn't use rsync anyway different between two hosts.

Re:I use it for linux distributions

[delt0r]

On all my linux installs rsync uses ssh by default. rsh hasn't been installed on anything in years. Is it even included in distributions anymore?

Re:I use it for linux distributions

[Anonymous Coward]

Various legal torrent repository sites, video game downloads, FOSS distros, public domain films. You name it. Got one public domain film up to a seeding ratio of 100.

Re:I use it for linux distributions

[dargaud]

vs gzipping and copying via scp

"scp -C" will compress as it transfer, no need to gzip.

Re:I use it for linux distributions

[DrXym]

Although git also does a great job of that with concurrent revisioning built in.

Git is a fine way to manage source code or perhaps semi-infrequently changing files like Word documents. It would be absolutely disastrous to use it to manage large binary blobs or log files or anything changes frequently. The reason for this is that a git project holds every version of the file ever and not necessarily as deltas either. Aside from that, the files have to be added and committed to the server before they're visible for pull which increases footprint size.

And the client side has to clone / pull every blob too. A clone can specify a depth to restrict the history of the clone, but it would still build up a history of cruft when perhaps you only want the latest version.

So while I use git in a lot of scenarios myself (mainly source control) I think it is not necessarily a substitute for either scp or rsync except for relatively sedentary directories where the corresponding git project isn't going to bloat out of control over time.

Re:I use it for linux distributions

[eyegor]

We're currently using the ROCKS cluster distro to run our cluster, but are finding that it's beginning to limit our ability to patch and otherwise maintain our cluster infrastructure. We've adopted cobbler and puppet for some of our HPC assets and will likely switch from ROCKS to more of a home-grown approach to manage our nodes. One thing I dearly love about ROCKS is the Avalanche Installer which uses bittorrent to distribute the image to the nodes when they do their initial build. I've

Are you using that or a similar package to do your node builds?

Re:I use it for linux distributions

[Anonymous Coward]

With a 10Gb network, it's probably faster to retransmit than compute missing pieces

Re:I use it for linux distributions

[Danathar]

Yea Rocks clusters has been using a modified version of bittorrent to re-install nodes in a cluster for years.

Re:I use it for linux distributions

[Anonymous Coward]

rsync + ssh is equally secure as scp? That's like saying http + TLS is equally secure as https!

Re:I use it for linux distributions

[Opportunist]

Hey, I practice safe sex. I have a firewall and every porn file gets scanned first before use.

Re:I use it for linux distributions

[Opportunist]

And how typically nerd. The discussion whether what you did was time efficient already takes WAY longer than the 10-15 minutes it took you to install it.

That's why we can't have nice things. They never get finished, we already fuck up everything in the planning phase with endless discussions and e-peen waving.

Re:I use it for linux distributions

[tehcyder]

Likewise.

I've gotten odd looks from my co-workers when I mention that I use bit torrent to download ISOs because it's faster. In most people's minds, bit torrent is synonymous with piracy.

That's because bit torrent is mostly used for piracy.

The percentage of the population that downloads Linux ISOs is tiny.

Re:I use it for linux distributions

[Anonymous Coward]

To be fair, rsync + ssh is equally secure as scp. I actually think rsync uses scp in that situation (please correct me if I am wrong).

The advantage I see rsync having is it is useful for automated (backups) of a large collection of files vs gzipping and copying via scp.

Rsync also makes re-starting an interrupted transfer easier that straight scp/tar/etc.

Re:I use it for linux distributions

[DuckDodgers]

It's common for the person in charge of distributing the operating system images to be very different from the person in charge of the network. Maybe the parent AC has no control over the networking gear.

But further, wouldn't multicast work best only when you're transmitting images to multiple machines at once? If the image has to be retransmitted after each system restart, wouldn't torrents work better? The machine that restarts can just grab everything it needs from all of the other machines running the same image on the same subnet.

Re:I use it for linux distributions

[wagnerrp]

If you're individually restarting machines, then it would be better for those handful of machines to pull an image from the image server, and not disrupt the other compute nodes. If you're restarting a whole bunch of machines, it's better to push the images out to all the nodes simultaneously using multicast, rather than bog down the network with all kinds of redundant unicast traffic.

Re:I use it for linux distributions

[Anonymous Coward]

There are half a dozen distros in my BT upload constantly, as well as some stuff I created myself, and FLACs from musician friends who want it shared. BT is a great tool for sharing!

Re:I use it for linux distributions

[kyrsjo]

Why should he/she come up with a new solution (and the one you propose honestly sounds more complex), which requires different HW (migth be problematic if all the HW isn't in his control), if he/she already have a working solution?

Re:I use it for linux distributions

[TooMuchToDo]

That's exactly what we used for configuration management and distributed/unattended installations :)

Re:I use it for linux distributions

[TooMuchToDo]

Something to think about:

If you're seeing boot storms, get a network switch that is managed and support vlans. Either programatically switch VLAN access on the switch, or use trunking support in Linux, so you have a production VLAN and an installation VLAN. This should segment your network to the point where boot storms are no longer an issue.

Re:I use it for linux distributions

[pnutjam]

Here's a question: To know which bits have changed, doesn't it need to compare the two files. How does this result in bandwidth savings?

Re:I use it for linux distributions

[TooMuchToDo]

We used ROCKS, and started running into the same problems you did. We were mid-process of moving to our own custom built solution when I left to work somewhere else.

Re:I use it for linux distributions

[3dr]

It doesn't need to compare the file contents bytewise. Each end of the rsync connection generates hashes of blocks of the file, and the hashes that do not match indicate underlying file differences. If a 1MB chunk in the middle of a 100MB database file generates different hashes, then only that 1MB chunk is transferred. That's how bandwidth savings are guaranteed.

Re:I use it for linux distributions

[denmarkw00t]

The wise keep as many tools in their tool box as possible. You seem to be among the wise.

The wise build workspaces with proper space arrangements and storage for their tools - only a fool puts all of their tools in one box.

Re:I use it for linux distributions

[ArsonSmith]

The OS is actually redeployed on every startup and only ever stored in system ram ramdisk. The image is very small and it hasn't yet caused an issue. I can just see an issue coming when we go from the current 32 nodes to the planed 3-400 by next year. The NFS infrastructure can handled what we have now with little impact. OS and apps are 1.2G, although I've been trying to trim that down a bit too.

Re:I use it for linux distributions

[toddestan]

I've gotten odd looks from my co-workers when I mention that I use bit torrent to download ISOs because it's faster. In most people's minds, bit torrent is
synonymous with piracy.

By ISO, I assume you mean pirated copies of Windows, right?

Re:I use it for linux distributions

[KingBenny]

lots of stuff, mostly demo's, games and linux isos yea, i usually let it run until i need the space back or it's format c: time, lots of games use it, since these days 10gb for a single mmo or similar is not a rarity anymore (something local ISP's dont seem to get since i sometimes get warnings about too much traffic on this 'unlimited downloading' pack subscription when i check out some stuff on steam. It's probably most useful since the load is distributed and that probably save money serverside. Also the clientside doesnt need to worry about a broken http download so yea, i see lots of uses

Re: I use it for linux distributions

[Anonymous Coward]

Hey, I work for BitTorrent and would love to know more about what the LHC is doing with the protocol, so great you're using it. Hit me up - Matt at BitTorrent dot com

Re:I use it for linux distributions

[Anonymous Coward]

(cgb here)

DuckDodger has read it right - the network team doesn't support multicast for these kind of applications, not that the equipment can't do it (it's a University, Cisco is used everywhere).

And it's 100Mbit to the desktops, so to not perform any kind of checksum means a 100+GB image has to be retransmitted from start to finish every time you do a reimage (whether it's unicast or multicast), close to 3 hours at 100Mbit.

I'm not aware of another decentralised, checksumming, verifying, transfer resuming image-level solution to this problem (rsync provides some of this with it's block device patches, but not the decentralisation). And it takes load off of the "proper co-located servers" when localised leeching nodes (or even a seeder) exist closer to the access edges.

Multicast is clearly the most efficient solution to minimise bytes traversing your network to a set of nodes that all want the same data though. wagnerrp: I am curious about your suggestion. Can you provide references to an image-level solution that implements "erasure or fountain coding to transmit your datagrams so clients can compute ones they missed" ?

Re: I use it for linux distributions

[MattMason]

This is awesome! I work at BitTorrent and would be great to hear more about this. Hit me on email - Matt at BitTorrent dot com

Re:I use it for linux distributions

[Man Eating Duck]

Here's a question:
To know which bits have changed, doesn't it need to compare the two files. How does this result in bandwidth savings?

It only needs to compare hashes. Rsync goes a step further and uses the ingenious concept of rolling checksums to determine if only parts of a file has changed.

Re: I use it for linux distributions

[TooMuchToDo]

Email away!

Re:I use it for linux distributions

[WhiteDragon]

Here's a question:
To know which bits have changed, doesn't it need to compare the two files. How does this result in bandwidth savings?

No, because it only sends a hash of the data. The other side computes a hash of it's data, and if the hashes match, the transfer is complete. If The hashes don't match, there is a rolling hash that can verify a partial match, and send only the changed data. There's more info here

Re:I use it for linux distributions

[WhiteDragon]

How would you use broadcast or multicast to distribute an OS? Call me ignorant, but how would you do that in practice?

I used to work in a computer lab at the university I went to. We used Ghost to do exactly that. All the computers in the lab (a few hundred PCs) were booted up off a special floppy (or later, cd) that started up Ghost in listen mode. Then, the central server used multicast to send the OS image to all the clients at once. It took less than an hour, and that was with 100Mb/s Ethernet.

Re:I use it for linux distributions

[TheCarp]

Well I will be damned, the man page agrees with you, this would tend to imply the -e ssh is obsolete too.

For remote transfers, a modern rsync uses ssh for its communications, but it may have been configured to use a different remote shell by default, such as rsh or remsh.

Absolutely

[Anonymous Coward]

All of my Torrents are legal data. What else would I use Bittorrent for besides Linux distros and Humble Bundle games?

Re:Absolutely

[kamapuaa]

Amateur electronica, generated in Garageband?

Re:Absolutely

[Anonymous Coward]

aren't the integrated publishing options easier?

Re:Absolutely

[Hentes]

Eclipse, of course. Although I'm unsure whether the submitter meant "moving data" as using bittorrent for file transfer, not downloading.

Re:Absolutely

[Nadaka]

Distributing home made erotica of course!

Re:Absolutely

[UltraZelda64]

Porn?

Re:Absolutely

[Anonymous Coward]

Not all home made erotica is quite legal.

Re:Absolutely

[Opportunist]

C'mon, that's cheating. "Porn" is a valid answer for all questions that somehow deal with the internet.

Re:Absolutely

[poofmeisterp]

Amateur electronica, generated in Garageband?

Nah nah, Amateur Garageband, generated in electronica. Way more transfers.

No - Resources

[Synerg1y]

Moving large data requires resources. In the case, of bittorent most things don't qualify because it's a distributed network, if 10 people in the office have the file and all know how t seed / use bittorent, you'd still be throttled by your bandwidth. Bittorent has however time and time again shown that it's distributed architecture can get something out to the masses very effectively.

Re:No - Resources

[Grishnakh]

I don't see why Bittorrent wouldn't work for an office of 10 people; its strength is copying data over distributed networks. For an office with 10 people, assuming they're on wired Ethernet links rather than WiFi, all 10 of those links will be connecting to one or more switches, which are able to handle full 100Mb or 1Gb speeds, duplex, between the computer and the switch, simultaneously. Using BT to copy the data might be a little slower than just using scp if it were only one PC, but the total transfer time for all 10 will be less.

Re:No - Resources

[sourcerror]

If the 10 people are in the same collision domain, torrenting won't make things faster. Actually it can slow things down with unneccesary collisions.

Re:No - Resources

[RuaisLampSilog]

Yes and No. Moving data from A to B using scp is ok if your link is stable. At my work, to download the 12 Gbytes of that other crappy OS, there is an SMB share. I have tried many times to copy 4 Gbytes dvds over it, most of the times with no luck. Even when there is a single source and a single destination torrent provides error recovery, and consistency not matched by other protocols. Other nice feature is that any client can become source of new files. Let's face it: rsync is beautifull but setting up an rsync server is not for newbies. On the other hand, any newbie can create a torrent file and let others download from him, with a lot of benefits. And the last is that most torrent clients have bandwidth limiting options. Even when you can say rsync how much you want it to use, to be able to change it on the fly sometimes is priceless. So yes, I think torrent in some scenarios, even with single source and client, is the way to go.

Re:No - Resources

[darkain]

Serious question... How long have you been using a network HUB instead of a network SWITCH?

Re:No - Resources

[MightyYar]

Collisions? Are switches that expensive?

Re:No - Resources

[StuartHankins]

cwRsyncServer (I'm not affiliated with them) makes setting up an rsync server in Windows straightforward. It's only 5 minutes or so in Linux. Bandwidth limiting is a --bwlimit switch. I do notice that in general rsync in Windows is slower than on Linux or OSX.

Re:No - Resources

[Drakonblayde]

You're confusing broadcast domain with collision domain.

On a switch, a collision domain is limited to the link itself, ie, collisions aren't possible. However, all ports on the switch (or all ports in the same VLAN, if the switch is manageable) share the same broadcast domain, which merely allows for all ports to contribute to saturation, but there still won't be any collisions

Re:No - Resources

[dead_user]

I wasn't even aware they still SOLD hubs. I haven't seen one in years.

Re:No - Resources

[bruce_the_loon]

Probably the same guys that never upgrade from their 486DX120 because it's fast enough to run Lynx.

Re:No - Resources

[henrypijames]

But if only I have a file and want to distribute it to those 10 people in my office, using BitTorrent is much faster than copying it to them one after another -- precisely because of the bandwidth throttle on my end.

Re:No - Resources

[Whiteox]

On the other hand, any newbie can create a torrent file and let others download from him, with a lot of benefits.

I can't.
I've tried numerous times:
1. Create torrent (easy)
2. Wait (easy)
3. No-one downloads torrent (fail).

Apart from P4P, I've given up on uploading torrents as NO-ONE IS INTERESTED.

Yours,
Failed Newbie

Re:No - Resources

[Synerg1y]

You're correct there, it would take a little longer for anybody to get it as the file seeds, but the 10th person would get it a ways faster and at around the same time as the others. However, traditionally this problem is handled via a file share server, but of course not everybody has / wants one.

Re:No - Resources

[poofmeisterp]

If the 10 people are in the same collision domain, torrenting won't make things faster. Actually it can slow things down with unneccesary collisions.

Is this 1994?

Re:No - Resources

[poofmeisterp]

Probably the same guys that never upgrade from their 486DX120 because it's fast enough to run Lynx.

bah dump bump, tsssing!

Patent Trolls?

[Anonymous Coward]

Does this seem like fishing expeditions by patent trolls?

Re:Patent Trolls?

[bobbied]

Does this seem like fishing expeditions by patent trolls?

No need to go fishing here.. Just find the torrent tracker and connect... Volia, you have a list of everybody who is distributing the material by IP address. It's all about tracing down the IP's and sending out the collection letters and cashing the checks from there, assuming you actually have the permission of the copyright holder to do so...

Re:Patent Trolls?

[larry bagina]

It's a honeypot to trap people that can't tell the difference between copyrights and patents.

Re:Patent Trolls?

[Anonymous Coward]

Luckily, when the IP address you're using is a Romanian one, these collection letters will be totally ignored.

Re:Patent Trolls?

[Nocturnal Deviant]

i tend to run through a Romanian based VPN almost exclusively for work(my own personal. I wouldn't trust an outside vps with corporate information) related activity, and i can absolutely confirm what he is saying. I've been on the phone with my vpn provider before(hes an old personal friend), and he says "OH BOY MAIL TIME, now i get to have more firewood from your american lawyers" as to which we both chuckle.

Re:Patent Trolls?

[Opportunist]

I think you mean copyright trolls.

Believe me, if you could somehow find a patent that could impede the use of P2P networks, the RIAA would already have paid $whateveryouwant dollars to get their hands on it, to make sure it will only be used in ways that ensure they can easily and cheaply spread content but nobody else can.

Re:Patent Trolls?

[poofmeisterp]

Does this seem like fishing expeditions by patent trolls?

No need to go fishing here.. Just find the torrent tracker and connect... Volia, you have a list of everybody who is distributing the material by IP address. It's all about tracing down the IP's and sending out the collection letters and cashing the checks from there, assuming you actually have the permission of the copyright holder to do so...

...But you don't have the psychological "why" or data to back up your court cases. Data like "We have 1200+ answers to the question 'what do you use torrent for other than pirating' and got nothing other than 'a lot of pirating' or 'I download Linux distros'.

Yes, it's possibly a fishing expedition. Gove 'em some good info!

Re:Patent Trolls?

[poofmeisterp]

Does this seem like fishing expeditions by patent trolls?

That was my first thought, and I'm responding appropriately to what is either a.) a data gathering exercise or b.) an indication that the person asking doesn't know how to get out of the bed unless a manual tells them which side to roll over on each morning.

Nope.

[Rinikusu]

I mean, other than the Blizzard stuff, no, I don't use bittorrent at all unless I'm downloading movies (usually) or software (sometimes).

Rapidshare/megaupload/etc work much better for my one-off transfer needs, while I leave media distribution to the masses via Youtube, Vimeo, Bandcamp, and media collaboration to Dropbox and sneaker-usbdrive-net (especially for big projects).

Yes, I have

[EkriirkE]

Many times a person is searching for a program to do something by keyword instead of software title, and for free. Torrent sites are a common place to go for something free. I just generate a .torrent for my software(s) and upload it to a few big trackers and the others seem to pilfer it from there. Just make sure the filenames and titles are relevant. It's like SEO, but TTO: Torrent Tracker Optimization.

Re:Yes, I have

[Anonymous Coward]

It's still SEO. Trackers do not have search functionality. Taking TPB as an example, the website is a search engine. That address also happens to have a tracker (probably even on the same host).

Comparison: using a website to search through an FTP server. FTP doesn't have search functionality, you're using an external tool to search it.

In a word? YES!

[bobbied]

I specifically do not torrent anything that has copyright issues but I do seed a number of Linux distributions and development tools which do not prevent distribution in their licenses. Downloading anything using torrent is effectively distribution of the material too, so you had better KNOW that the license allows you to make copies and give them away.

You folks that torrent movies and stuff that is not in the public domain are crazy in my book.

Move Legal Data With Torrents?

[MarkvW]

Only a complete fucking moron would move legal data with torrents.

A lawyer is obligated to preserve his clients' confidences. When you store your information on somebody elses server or servers you are giving up custody and control over some of those confidences. In that situation you are entirely dependent upon the strength of your encryption.

That encryption might be good today or tomorrow, but how good will it be five years from now or ten years from now when quantum computing or the next best thing becomes available for codebreakers.

Don't risk a lawsuit from a pissed client!

Re:Move Legal Data With Torrents?

[godrik]

In this context, I believe legal meant "not illegal". Or "data I own" or "data that nobody will sue my ass off for moving them around"

Re:Move Legal Data With Torrents?

[treajones]

I don't think that is the Legal data they are talking about.........

Re:Move Legal Data With Torrents?

[Theaetetus]

Only a complete fucking moron would move legal data with torrents.

A lawyer is obligated to preserve his clients' confidences. When you store your information on somebody elses server or servers you are giving up custody and control over some of those confidences. In that situation you are entirely dependent upon the strength of your encryption.

That encryption might be good today or tomorrow, but how good will it be five years from now or ten years from now when quantum computing or the next best thing becomes available for codebreakers.

Don't risk a lawsuit from a pissed client!

Respectfully, I believe the submitter meant "legal" as opposed to "illegal" or unlicensed data under someone else's copyright - i.e. the vast majority of torrents.

Re:Move Legal Data With Torrents?

[Anonymous Coward]

Plenty of things are not illegal but not lawful. To be precise it should probably be "Do you lawfully move data with torrents?"

(Your last suggestion has most merit were it not for SCO and similar scum)

Re:Move Legal Data With Torrents?

[Anonymous Coward]

Okay, I saw the joke, and it was good and all, but maybe shorten it a bit. Like, take out the encryption sentence. Being concise helps!

Re:Move Legal Data With Torrents?

[Anonymous Coward]

If you're really worried about quantum, either your IT guy isn't good enough -- or you aren't good enough.

Yeah, there is quantum that will crack some forms of symmetric crypto. There's also crypto that's been designed robustly against it.

There's also forms that are provably robust against quantum computation and even oracular systems.

Pick one and use it.

If you're worried about hypothetical attacks in the future -- don't be. Seriously, it's a totally utterly disproportionately minor risk. Sure, the threat is catastrophic, but you're going to have a catastrophic fault in your other aspects a hundred times before the crypto itself breaks down.

If there's ever a quantum machine capable of eating existing RSA keys, there's going to be massive password breaches *anyway*, and there will probably be a variety of nasty people on your network before you get it patched. And quite possibly even afterwards given password reuse.

Bottom line -- the strength of encryption is nearly an irrelevant concern once you're using something strong enough. The correctness of the implementation, and your key and password management practices, you should be in terror of.

In terms of moving legal data over a torrent, I think you misunderstand the op meaning 'legal' as in 'not unlawful', vs 'of or pertaining to lawyers/courts/etc'.

But really, it's not a horrible idea assuming you know how to set up a private seed. It's a robust protocol good at piercing firewalls (not as good as skype or gotomeeting), it has rich error recovery, checksumming, the ability to prioritize specific pieces of content. As a way to collaborate remotely when dealing with large quantities of datafiles it wouldn't be bad in the face of disproportionately low bandwidth.

Of course, all this ignores that if you're storing the data anywhere at rest, it ought to have been encrypted anyway. Sooner or later the people engaged in electronic espionage are going to realize there's more profit to be made working for the private sector than the government. The targets won't be universities and comcast systems for botnets, or routine industrial espionage. It'll be the legal networks to figure out the litigation strategies and weaknesses. Courts trump economics, warfare trumps courts. If there isn't someone specializing in it now, it's just a matter of time.

Re:Move Legal Data With Torrents?

[Anonymous Coward]

#accidentaljoke

Re:Move Legal Data With Torrents?

[Anonymous Coward]

sure,

      and my offsite tapes backup are different ?

Re:Move Legal Data With Torrents?

[Sloppy]

Respectfully, I believe the submitter meant "legal" as opposed to "illegal"

Aren't you worried about getting the deaf penalty for using their network racehorses to trade TV shows containing sax and violins?

Oh.

Never mind.

Re:Move Legal Data With Torrents?

[UltraZelda64]

What are they gonna do? Fire a machine gun by your ears a few minutes until you can't hear any more as a sentence? Yeah, that would have the potential to make you go deaf...

Re:Move Legal Data With Torrents?

[Anonymous Coward]

in this context, i believe "woosh!" is in order.

Re:Move Legal Data With Torrents?

[Zontar The Mindless]

Dude. Take a Valium.

Re:Move Legal Data With Torrents?

[Anonymous Coward]

You seem to have left a stick in your anus. You might sit more comfortably if you removed it.

Re:Move Legal Data With Torrents?

[xyzyxx]

I think you are misinterpreting the meaning of the word "legal".

Re:Move Legal Data With Torrents?

[Anonymous Coward]

Either way, there is enough WHOOSH to all the replies to the GP to run my wind turbine for a year.

Re:Move Legal Data With Torrents?

[poofmeisterp]

In this context, I believe legal meant "not illegal". Or "data I own" or "data that nobody will sue my ass off for moving them around"

Oh, so THAT'S why Slashdot has a moderation unit called "Funny". NOW I get it!

Re:Move Legal Data With Torrents?

[poofmeisterp]

Dude. Take a Valium.

Are you torrenting any?

*snerk*

Re:Move Legal Data With Torrents?

[Anonymous Coward]

Poe's law? Or perceptions in skew planes?

Can you use it to move a 300 megabyte hosts file?

[Anonymous Coward]

inquiring minds want to know.

Linux ISO's mostly

[dlapine]

At work I need to install several different types/versions of linux OS's for testing. I always torrent the ISO as a way of "paying" for the image that I'm using.

A few years back, we did some experimenting with torrents over the Teragrid 10GBe backbone, to see how well that worked over the long haul between IL and CA. With just 2 endpoints, even on GBe, it wasn't better than a simple rsync. We did some small scale test with less than 10 cluster nodes on one side, but still not as useful as a Wide Area filesystem we were testing against. Bittorrent protocols just aren't optimized for a few nodes with a fat pipe between them.

I am interested in looking at the new Bitorrent Sync client to see how thanks for our setup. We have many users with 10's of TB's of data to push around on a weekly basis.

Yep- Linux

[markdavis]

Just about the only time I use torrents is when downloading Linux distributions- Mageia, Fedora, CentOS, etc. Occasionally iso's for grub magic, ultimate boot CD, and such. All of that legal. And I usually leave it up at least long enough that my share ratio is 100% (1.0).

Re:Yep- Linux

[nabsltd]

Occasionally iso's for grub magic, ultimate boot CD, and such. All of that legal. And I usually leave it up at least long enough that my share ratio is 100% (1.0).

For such tiny things (latest UBCD is 486MB), I pretty much seed forever, as it doesn't really cost me anything. I'm at 67:1 on the latest UBCD, and over 400:1 on some other much smaller torrents.

I leave them running because I don't artificially limit my upload rate on a per-torrent basis, only as a total for all torrents (and that limit is quite high, as I have Verizon FiOS). It really bugs me when I try to download an older torrent and the only seeds are uploading at a few KB/sec. Even on a 100MB torrent, it can take 5-6 hours to download.

Re:Yep- Linux

[StuartHankins]

Thank you for seeding :) It's underappreciated.

Re:Yep- Linux

[jones_supa]

Just about the only time I use torrents is when downloading Linux distributions- Mageia, Fedora, CentOS, etc. Occasionally iso's for grub magic, ultimate boot CD, and such. All of that legal. And I usually leave it up at least long enough that my share ratio is 100% (1.0).

Even better is if you aim for a share ratio of 2.0. With a share ratio of 1.0, you only "give back what you take", so the swarm stays as strong as it was. That's good. But sending back an "extra copy" gives your contribution to make the swarm stronger.

occasionally

[BenSchuarmer]

mostly for music that's under creative commons license and the occasional Linux download.

Not anymore...

[Anonymous Coward]

Because AT&T throttles my connection any time I try.

I've never been much of a pirate. I don't remember downloading any mp3, DVD or blu ray rips. Not even in the days of edonkey, lime wire and napster.

I have always had a Linux addiction, though. I get bored and visit distro watch, and decide to burn a few discs and install them on whatever computer I have lying around not getting used.

My choice is to take the load off the distro's servers by torrenting, which downloads in a day or more. Or to download via http/FTP and get it in less than an hour.

Yes, with Sync

[Anonymous Coward]

Yes. Sync is being used to distribute various sized files between a large amount of machines within the private network.

World of Warcraft

[steamraven]

If I remember correctly, some online games use P2P to distribute updates legally (though they might not use bittorrent).

Re:World of Warcraft

[Black Parrot]

If I remember correctly, some online games use P2P to distribute updates legally (though they might not use bittorrent).

Before I moved to Steam, torrents were the *only* way I could get updates for Company of Heros.

Re:World of Warcraft

[Luuseens]

Indeed, World of Warcraft used to (still does?) provide the updates and installs as a thin torrent client. You could even extract the torrent file from the binary and use a different torrent client to download the update files if you wanted to.

Nope.

[Anonymous Coward]

The main problem with file swarm protocols is the lack of standardization.

Like if I'm creating a webcomic and I want anyone to be able to read it provided they are also sharing it with others, there's no way to simply have a "bittorrent web browser", no what we instead have are a hodgepodge set of either pile of files in a zip/rar/cbr/7z file or all the files are separate but the segments don't correlate with the files. eg If I want to read just the first 8 pages to see if it's good either I have to download the entire zip file, or I have to download those 8 files plus likely another 8 files shared in their chunk if the chunk sizes are large. Either way it's not efficient, rendering the entire use of bittorrent for comics as more of a interesting test case.

The above scenario:
a) I can not progressively read the comic due to out-of-order segmentation or archive-packing
b) I can not get the latest update to the webcomic due to the torrent hash having to change for every update
c) I can not view the content in the way it was intended since the torrent program is not a web browser.

Now replace comic with "video"
B no longer applies unless subtitles or separate audio streams become available at a later date. A and C still apply. Video unlike image files may be partially viewable if enough of the beginning data is available, but won't be seekable, and is generally not useable.
Where this can be improved is creating multiple data streams (eg files) for videos that progressively get larger, eg 160x120->320x240->640x480->1280x720->1920x1080 and only downloading the stream segments you need. This is partly done now by just going directly to the stream size they want, but what I'm talking about is downloading the tiny stream first so that you have useable data to seek the file.

Now replace comic/video with "game"
This is where the torrent protocol should shine, as games are themselves progressively loaded.
A game could send a "fileswarm request" for all the files it needs as they load, and when the game is restarted, check again that the files haven't changed. Thus removing the never-ending patch cycle of most MMORPG's and various single player games. If there's no network, then use all local files until network is restored. If the file is critical, prompt the player to go online.

Some

[GameboyRMH]

Linux distros, free movies, free games...

I tried to switch to Deluge but it couldn't handle a file with a Japanese character in its name...other than that, only things that I think many torrent clients could use is the ability to accept magnet downloads through a drop folder somehow, and searching & better sorting/filtering options for downloaded torrents.

Re:Some

[David_W]

I tried to switch to Deluge but it couldn't handle a file with a Japanese character in its name...

FYI, IIRC, it only has issues adding a .torrent with Japanese in the filename; if you rename the file and add it, it can download the files "within" the torrent just fine even if they have Japanese in them. And yes, this is odd.

Re:Some

[GameboyRMH]

Huh thanks I'll have to try that, that may be the problem and it was the only holdup that kept me from switching. IIRC it added the torrent successfully but then couldn't find the files that were already downloaded.

Yes:VMs

[Anonymous Coward]

I used torrents to move VMs (20gb) across a 1.5 mbit T1 line, after it failed to do a direct transfer the night before.

Unlike the direct file transfer, which failed during some unknown disruption, the direct transfer was simple to configure (more so than an FTP), and was done by the next day.

Democracy Now

[Anonymous Coward]

is a daily TV news show distributed by torrents:

http://ewheel.democracynow.org/

I download it every day.

We deploy VM images to our developers over BT

[carlhaagen]

We use the OpenBitTorrent tracker and the Transmission client to deploy and acquire virtual machine hard drive images among our developers. The obvious reason is that it's much faster for our developers to help eachother out with shoveling the data around rather than the developers having to get all the data from one and the same link (read: the main server). Compare it to a person reading a novel, once, out loud, to a group of people, instead of reading it in private over and over to each and every attendee.

Re:We deploy VM images to our developers over BT

[wagnerrp]

That's a horribly inaccurate analogy. Bittorrent is like a restaurant. If you have two people holding a conversation in a restaurant, everything is fine. Even if you have several pairs talking, it's really not a problem. When you have a full restaurant and everyone is holding their own independent discussions, no one can hear a damn thing. If your network is using a hub, you would be stupid, but that's what you would get. Assuming you're using switches like a rational person, chances are not everyone is on the same switch. If they were, you would have sufficiently few developers that they would have no problem just sequentially pulling the new images off a shared server. So, further assuming you have developers networked through a branching topology, bittorrent is not topology aware. It will not attempt to favor those peers close to it on the network, and thus will quickly clog your trunk lines between switches.

Your analogy is an accurate description of broadcast or multicast networking. Why aren't you using multicast networking? Have your developers run a client that joins a certain multicast group. At a certain, scheduled time of the day, the server pushes new images onto that group, and to any client listening on that group. The client then reconstructs the image, and stores it on the hard drive of each developer. One person talks; everyone receives. No physical link ever has to transmit more than that one original copy of the file. Anyone whose machine is not available for the update can then download it manually from the central server at their leisure.

Re:We deploy VM images to our developers over BT

[Anonymous Coward]

the few people who put up with socializing with you get tired of you really fast, don't they?

Re:Can you use it to move a 300 megabyte hosts fil

[kid_wonder]

now that is humor! well done, sir

gittorrent

[lkcl]

the one thing that would help enormously would be to have git be *truly* peer-to-peer distributed. not "yeah shure mate you can always git pull and git push, that's distributed, and you're a peer, right, so... so... git is peer-to-peer and distributed, so what are you talking about you moron??" but "at the network level, git pull and git push have a URL type that is **TRULY** peer-to-peer distributed. to illustrate what i mean, i would like to be able to do the following - with all that it implies:

git clone magnet://abcdefg0123456789/gittorrent.git

if you're familiar with magnet links, you'll know that there is *no* central location: a DHT lookup is used to find the peers.

now, what wasn't clear to the people on the git mailing list when i last looked at this, was that it is possible to use bittorrent to do git pack objects, by creating a file named after the pack object itself. and what wasn't clear to sam (the last person who tried to put git over bittorrent) was that you *MUST NOT* make use of bittorrent's "multiple file in a torrent" feature, because bittorrent divides up its data into equal-sized blocks that *do not* line up with the files that are in them, which is why when you download one file in a torrent you almost always end up with the end of its preceding file and the start of the one after it, as well.

the idea i came up with is that you create *multiple* torrents - one per git object (or git pack object). if you want to pull a tree, you create a torrent containing *one file* which is the list of objects in that tree; gittorrent would then know to map each of those objects onto yet *another* torrent (one per object), repeat until all downloading happily. gittorrent objects are of course named after the hash, so you can pretty much guarantee they'll be unique.

and, adding in a DHT (a la magnet links), you are now no longer critically dependent on something like e.g. github, or in fact any server at all.

to answer your question in a non-technical way, mr anonymous, i think you can see that i feel it would be much more useful to have development tools that use bittorrent-like protocols to share files-as-revision-controlled-data (and, if you've seen what joey hess is doing with bittorrent you'll know that that's a hell of a lot - including storing home directories in git and doing automatic distributed backups)

Re:gittorrent

[Anonymous Coward]

You lost me when you mentioned Git during a BitTorrent discussion. Is there some relevance to your little rant, or were you just aching to hork it up somewhere and figured "the Bit in BitTorrent MUST be a corruption of Git!"?

Re:gittorrent

[Qzukk]

Intriguing idea, but I tried subscribing to your newsletter and I keep getting the Sept. 24, 1998 edition over and over. The problem with using bittorrent for this is distributing NEW data. If the protocol could cope with a seed appending data to the torrent without having to create a whole new .torrent file, then this could be awesome. As it is, you're just changing the problem from "how do I send out new versions of files when I commit something" to "how do I send out new versions of the .torrent files every time I commit something"

Re:gittorrent

[jones_supa]

I tried subscribing to your newsletter and I keep getting the Sept. 24, 1998 edition over and over.

A new meme is born!

Re:gittorrent

[akanouras]

I see your flamebait already fooled some mods. However, you're supposed to attack the idea, not the man. And you failed spectacularly at that.

The key component in what he's proposing would be the DHT; not even necessarily the same DHT as used by standard BitTorrent clients. The use (or not) of the BitTorrent specification and its intricacies would be just an implementation detail.

You're free to be one of the "can't be done" people; it would be wise not to advertise it too much around here however; as it's not become Reddit yet.

"how do I send out new versions of the .torrent files every time I commit something"

I think I've had enough. Off to BT101 with you.

Re:gittorrent

[Qzukk]

However, you're supposed to attack the idea, not the man.

Care to point out where I insulted the man? All I did was torrent his newsletter, but every time I connect to the swarm, I never get any new data, I always get the same data again. This has been the way bittorrent worked since its inception (so I was off by 3 years). After all, if anything changes then the hashes in the .torrent file will tell the client it's corrupt and the client will try to change it back.

The use (or not) of the BitTorrent specification

If you "(or not)", then you're making it something other than what he's proposing.

Off to BT101 [wikipedia.org] with you

OK, I read it. I'm not seeing how DHT will solve the fact that the first time you commit a change to your code, the bittorrent client will detect it as corruption and replace the files with the original version from the swarm since the modified files won't match the hashes in the .torrent. Using a magnet link just hides the .torrent from you, the first commit will still need to somehow add new hashes (god forbid you do a rebase and change ALL the hashes).

The least invasive way I can think of to do this with some semblance of security would be some sort of public/private key arrangement that would identify the author of the .torrent file and allow that person to distribute replacement .torrent files through the swarm. When I commit a new version of foo.c, my signed update goes out and everyone updates their .torrent file with the new foo.c hash, and starts getting the new foo.c file. Anyone joining the swarm with an old .torrent gets notified about the updates and new hashes. Meanwhile, someone attempting to add evil.c to the torrent would not be able to get anyone else to recognize it as a valid block.

Re:gittorrent

[magic maverick]

I might switch to Git (from Bzr) if this sort of thing existed. I would love for a way to anonymously (pseudo-anonymously) host and distribute my software, including changes as I make them. I'm not interested in signing up for Github or similar though, and so I'll continue to use Bzr because it's what I use...

Re:gittorrent

[benob]

I think you've just found how to save rss.

Re:gittorrent

[Anonymous Coward]

Better distributed git suggestion would be XMPP URIs (Jabber resources) like git clone xmpp:coolperson@example.com/git/somecodeproject. Bonus is it includes Presence information out-of-the-box and you can Pidgin your way to knowing if a repository is currently available.

Re:gittorrent

[fulldecent]

So the magnet url changes with each commit?

Re:gittorrent

[akanouras]

First of all, I apologise for the tone of my post... lkcl's posts frequently attract trolls and, posting at a late hour again, my hasty judgement got the better of me. :-(

When you give your bittorrent client a magnet link with a "btih" (BitTorrent Info-Hash) component, it starts looking up the DHT (asking other peers it's already made contact with) for peers sharing that infohash's torrent. As soon as it finds at least one, it connects to that peer ("joins the swarm") using the BT protocol and asks for the torrent's metadata (the.torrent file's info section). It is at this point where it will pop up a window asking you where to save the torrent and/or which files to download from it.

Given the above, one way that "gittorrent" could work would be the following:
1. Each Git object (file/file tree/commit) would be a separate "torrent", identified by its hash. Information about which peers have which hashes (i.e. objects) would be stored in the DHT. Nothing new for DHT so far. With this in place, you could checkout any individual git file/tree/commit and its history (each git commit references its parent).
2. Here we introduce a DHT protocol extension (let's call it "get_hash_addenda") using which you could get information stored in the DHT (this is the major difference with .torrent files) about newer commits. With this in place you would also find other users' "forks". This would also be useful for ordinary torrents by the way (get subtitles/new episodes for this torrent).
3. With these in place, you now have a (very slow) "gittorrent" implementation. Additional extensions/local DHT data caching/assumptions/whatever) would be used to speed the whole thing up. Existing BT trackers (fast peer lookup for a given infohash) would work as is. Existing git daemons would also work, and could also be extended to speak DHT and the BT protocol. All these things map well to existing BT concepts.

The protocol(s) used for transferring objects between peers could be the BT protocol (provided we treat each git object as a separate torrent, as lkcl recommended), and/or any protocol already supported by Git (as an analogue of "web seeds").

Example bare bones magnet links:

magnet:?xt=urn:git:<object id>
magnet:?xt=urn:git:<long hex number representing a user>/<repo>[/<branch|tag name>]

or even magnet:?xt=urn:git:<user>/<repo>/<branch|tag name>
if a way of having non-spoofable and globally unique nicknames is found (probably in an FCFS fashion).

A nice research project on extending BT (which already somewhat implements the "get_hash_addenda" functionality) is Tribler.

The earlier "gittorrent" effort that lkcl criticised can be found here. A cursory glance reveals an emphasis on trackers and almost no mention of DHT, probably due to it being written in 2008.

Disclaimer: All this is from a layman's point of view and horribly inaccurate. The DHT protocol extension especially could probably be avoided using some convention. I've been reading up on how BitTorrent and DHT works lately and, honestly, I can't blame you: apart from the BEPs it's basically UTSL.

Re:gittorrent

[Anonymous Coward]

You can violate 1-file-per-torrent safely if you do what Azureus/Vuze and a few others can do, use hidden null files to align the files with the piece boundaries.

Re:gittorrent

[lkcl]

I'm not seeing how DHT will solve the fact that the first time you commit a change to your code, the bittorrent client will detect it as corruption and replace the files with the original version from the swarm since the modified files won't match the hashes in the .torrent.

ok. there's a couple of solutions here. one is to query a number of peers for the same object, obtain its MD5 (or other) checksum and validate them aand the other iiiis...

The least invasive way I can think of to do this with some semblance of security would be some sort of public/private key arrangement that would identify the author of the .torrent file and allow that person to distribute replacement .torrent files through the swarm.

yes. exactly. git allows you to GPG sign tags. the GPG-signed tags would be the key point around which you would verify that you (ultimately) got the right objects. and it's the GPG-signed tags that would allow you to decide to fork an entire project, or upgrade an entire gnu/linux distro, by simply setting a new target to pull and verify against.

the question which i have yet to resolve is: what the hell do you do about all the intermediate commits, intermediate objects etc. etc.? enough idiots trying to corrupt the system would result in quite a lot of bandwidth wasted before you got to the point where the git tagged branch could be verified by MD5 summing.

what i don't quite understand, though, is why git over http (or any other network protocol) doesn't have the same issue. or, is it the fact that there is only one central control typically for a particular branch (or tag) that makes this moot? so you know that you're only going to ever be pulling one git pack-object from that one server, and having done so you're now up-to-date so can do the checksum, bam, done.

I use the bittorrent protocol sync my computers

[runner_one]

I have three computers. Home desktop, Work Desktop, and, a laptop.
I use this newly released piece of software to keep them all in sync. I added a server to the mix as a backup, and now all my data is on four computers. The peace of mind given by having my data automatically mirrored in four locations and the resulting lowered chances of loosing all my data enables me to sleep better at night.

Damn right

[EmagGeek]

Linux ISOs
VM Images
Backup Images
Home Movies
etc...

Re:Damn right

[tehcyder]

"Home Movies". Nudge nudge. Wink wink. Know what I mean? Say no more, say no more. "Home Movies" eh? Are you keen on videos? Eh? Eh? Got any videos of your wife naked in your "Home Movies"? Like to buy some?

Re:Damn right

[poofmeisterp]

"Home Movies". Nudge nudge. Wink wink. Know what I mean? Say no more, say no more. "Home Movies" eh? Are you keen on videos? Eh? Eh? Got any videos of your wife naked in your "Home Movies"? Like to buy some?

Oh, come on that's totally inappropriate.

You torrent that stuff at the office.

Game download

[Spy Handler]

I got a game called War Thunder via bit torrent. What you do is download a small installer program from the publisher's website and run that. The installer automatically connects to BT seeds and peers and downloads the actual game itself.

There is no other way to get War Thunder. I suppose since they're a small publisher, their web server can't handle distributing the 13 GB game file to tens of thousands of users.

Re:Game download

[damnbunni]

I hate it when developers do this. Offer a torrent as your default download, sure, but put a direct download buried somewhere in your support pages.

Some people (like me) have net connections that just crap out on torrents. I can download a large file reasonably well, but the same file in a torrent will take weeks.

And some people (a majority, in some nations) have caps, and a torrent-based downloader eats into that quite a bit.

Re:Game download

[Anonymous Coward]

Get a better ISP. If you don't vote with your wallet they're just going to keep ramming you and content providers (like this game studio) harder.

Multi site sync

[silas_moeckel]

I have a fairly large and growing (3.7TB) dataset that needs to be replicated nightly to a bunch of different sites. By the nature of the dataset nothing is ever removed or changed just new files added. It needs to be copied out to a half dozen locations that have as much outbound bandwidth as the primary. So a cron job sets up the torrent every night and all the remote sites pull data from the primary and reshuffle it between themselves.

Re:Multi site sync

[wvmarle]

I may miss something, but isn't this the ultimate job for rsync, to copy only the new bits? Or do you take a diff and distribute only that?

Re:Multi site sync

[keshet]

Let me guess.. bitcoin blockchain?

Re:Multi site sync

[hab136]

Rsync would also work, but BitTorrent will be faster.

> a half dozen locations that have as much outbound bandwidth as the primary

Assume all the sites have 10 MB/s inbound and outbound. To transfer 3.7 TB (3,879,731 MB) would take 107 hours per site, 646 hours total.

Using BitTorrent, all 6 sites will take somewhere between 107 and 646 hours - probably around 300 hours, because after one site receives a block, it can send it to the other sites, without using the primary's bandwidth.

The daily diffs will of course be smaller - but the same principle applies. BitTorrent will also verify the existing data and then just transfer the new bits, same as rsync.

Re:Multi site sync

[silas_moeckel]

Since there are no deletes or changes to existing files it just remakes a torrent with all the new stuff. The far end sees the existing bits checks to make sure it matches the source and skips it.

Rsync would require the same data to be sent 6 times from the primary or a convoluted process to syncing from P to A then P to B, while A to C and finally P to D A to E B to F or similar. In any case it's still 3 times as long as a torrent. At 40 ish million files rsync takes 10 minutes or so just verify that 2 sites match over a local gigabit connection (the hardware is meh e3 with 4x sata 3tb in raid 5).

Re:Multi site sync

[wvmarle]

Your comment would be true if they were really pumping around 3.7 TB every day. However, they don't (or at least: they shouldn't have to):

By the nature of the dataset nothing is ever removed or changed just new files added.

This is why I was thinking of rsync. Only transfer the changed bits. Now indeed one could use a combination of a diff tool (check what's new) plus torrent to distribute the updates... whether that extra effort is worth it depends of course on the total amount of data to transfer.

These count

[Anonymous Coward]

http://ocremix.org

Any linux distro I download

Bibles, Officer. Lots and lots of Bibles.

[Anonymous Coward]

Also, U.S. flag images, anthems, church prayers and sermons, videos and transcripts of presidential speeches, 4th of July military marches videos... And, lots and lots of Bibles...

Move my legal data?

[Jeremiah Cornelius]

I let my Lawyers handle that. It's what they're paid for, isn't it?

NO

[ADRA]

Bit torrent is a good data -distribution- tool, not a data -mover-, and it would be lousy to play that role. There are at least a dozen possible open solutions for moving data from point to point, but I have no idea why you'd use a protocol/tool stack that are designed for broadcast/graph distribution to do so.

An off the top list:
      1. NFS
      2. SMB
      3. FTP
      4. SFTP/SCP/rsync
      5. HTTP/HTTPS
      6. sz/rz
      7. iscsi
      8. DFS
      9. AFS
      10. UFTP/XFTP

The question should really be what exactly do you see ad being deficient about all these protocols that deems it necessary to re-invent the wheel yet again?

Re:NO

[Anonymous Coward]

i have 4 backup services "in the cloud".
which one of your examples will work with my remote sites at the same time ?

Re:no

[nabsltd]

I dont have anything worthy of having to mass source it, and my 3$ a month "unlimited bandwidth" website has taken 400 gigs in a month downloads before without a sweat.

That's an awesome price, even if the overall speed isn't that great.

Everything I have found that gives you more than a few GB of disk space and truly unlimited bandwidth costs a lot more than that per month.

Re:no

[Osgeld]

its though hostmonster, they claim right this second 4.95 a month, but to get that price you gotta pay forward for 2 years, I have been with them since ~2006, but yea
"unlimited" bandwidth
"unlimited" space
awesome speed ... for 2006, but still decent now, not had too much of an issue and you can get ssh access for doing routine things (not running programs)

but I am sure there are others out there who are running the same thing, the ONE time I had a 400GB run in a month more than makes up the years of trickle in data

What does THAT matter?

[erroneus]

Punish the technology because of how it is used? I thought we grew past that notion already.

At some point, one of the few remaining ways to get good information and news will be through these rougue channels and methods. Do we have to keep re-hashing the same ridiculous notions? How about we ban types of music based on the fact that thugs and criminals like it and glorify killing?

Re:What does THAT matter?

[cheekyjohnson]

I thought we grew past that notion already.

Where have you been?

Yes indeed!

[briancox2]

Video content mostly.

And secondly, lots of Linux distributions. These serve 2 purposes. They support free culture and minimize the amount of load on the servers of those open source developers that already giving so much. And also, because I enforce encryption on everything, even if my ISP did see what I was torrenting, they'd likely find it was something legitimate. I find nothing lacking really. BitTorrent just came out with an alpha file sync program that seems to be working really well. I'm happy.

Re:In a word? YES!

[Anonymous Coward]

You folks that torrent movies and stuff that is not in the public domain are crazy in my book.

I also drink caffeine after 4pm and watch TV shows that use the r-word. Crrrrrrazy!

Facebook Does

[terbeaux]

Facebook deploys its 4GB binary to its 500,000 servers using a torrent client that has rack and switch affinity. Each client goes for data chunks that are already locally on a rack or switch that it is connected to. That is a crap-ton of data.

rsync transfers changed PARTS of files

[raymorris]

To be fair, rsync + ssh is equally secure as scp. I actually think rsync uses scp in that situation (please correct me if I am wrong).

rsync and scp operate somewhat similarly if the file does't exist all on the destination. If an earlier version of the file exists, rsync transfers only the changes. Therefore it couldn't use scp - rsync does things that aren't possible with scp.

Since sometimes you have to use rsync because scp can't do it and I don't care to memorize redundant options, I normally use rsync even with scp would do.

Re:rsync transfers changed PARTS of files

[Anonymous Coward]

I believe he means that rsync uses the ssh encryption, and it does if you're transfering over that protocol (I don't know about an rsync server daemon.)

sftp, ssh, scp, rsync to user@somehost:/some/directory, all use the exact same encryption.

Re:rsync transfers changed PARTS of files

[denmarkw00t]

rsync only does incremental copies when you tell it to, and, technically, the same thing could be achieved through ssh and scp (using ssh to check for the destination file first and then copying with scp if needed). Of course, rsync does make it easy; just because it's easier doesn't mean it's not technically possible with scp

The big advantage

[tdelaney]

The big advantage of using BitTorrent over many other protocols for moving large amounts of data (as opposed to distributing it) is reliability - or lack of it. When you're moving large datasets, you don't want it to crap out and not be able to resume 90% in.

Sure - rsync has the ability to resume, but it requires explicit command line options. It's a terrible feeling to realise you just restarted a 10GB+ transfer instead of resuming it.

Re:The big advantage

[Anonymous Coward]

By default, rsync will resume transfers, but only as soon as the sender can figure out what still has to be sent.

It does this by asking the receiver for a list of files and their metadata, then asking to receiver to send rolling checksums for files with mismatching metadata. If any of the rolling checksums fails, then the block for that checksum will be sent. Yes, rolling checksums consume some bandwidth and disk I/O; but it's still much faster than having to actually restart the entire transfer all over. It also makes doubly sure that data hasn't changed underfoot since the last transfer attempt.

SOLR at Etsy.com?

[rwhiffen]

The folks at Etsy do it to replicate SOLR:
http://codeascraft.etsy.com/2012/01/23/solr-bittorrent-index-replication/

Not sure if that's what you mean.

Install Disks

[Anonymous Coward]

I use it for Linux Install Disks and Other Free Software, faster than downloads from many mirrors which have limits

+1 Linux distros. Only for multiple recievers

[raymorris]

Same here, I download and seed Linux distros.

with just two end points, it wasn't better

For point-to-point transfer to large amounts of data, the protocol does't matter, as long as the protocol is sane. The time spent moving data bytes will be much higher than any protocol overhead. rsync is roughly optimal because it won't transfer portions of the file that the receiver already has. BitTorrent is for distributing data to many destinations.

Red vs Blue

[Tarlus]

Back in the pre-YouTube days, Rooster Teeth distributed their videos using Bit Torrent to relieve their own HTTP load. I think they gave BT users the incentive of downloading earlier to encourage its use.

All data is legal

[zoloto]

somewhere in the world, whatever you're moving, it's legal.

no

[Osgeld]

I dont have anything worthy of having to mass source it, and my 3$ a month "unlimited bandwidth" website has taken 400 gigs in a month downloads before without a sweat.

About the only thing I could think of is a linux distro, but again the only thing I am bound to cobble together is a lightweight debian mix for maintance / repair / recovery when someone brings me their computer to be fixed and winders is all trashed.

You guys are hillarious.

[Anonymous Coward]

I see so many posts about "I only use torrents for legal means because I am a upstanding law abiding citizen and would never sink as low as a terrorist or a rapist in using torrents for illegal means. I never do anything wrong ever, even be accident." blah blah blah.

Not really a fan of it.

[UltraZelda64]

It's a good idea for sharing, but I'm not a fan of the way it loves to hog the Internet connection, it starts connections out the ass, flooding the network and just slows everything down. I tend to use wget for pretty much everything; with a decent server, it gets by just fine with only one connection. I also like the idea of maintaining at least somewhat-accurate timestamp data whenever possible, and BitTorrent doesn't seem to have a concept of that. And also, I like to maintain my own checksum files, so BitTorrent doing that itself is extra functionality that I don't need. Not too crazy about the serious fragmentation it can cause, and many clients just do not pre-allocate disk space or even have an option. The bigger the file, the better BitTorrent works--but unfortunately that little fact kind of stops it in its tracks; a file of many gigabytes, downloaded in thousands of random pieces, is bound to end up fragmented to hell and back if its space is not pre-allocated.

However, if there is no FTP/HTTP link, I won't hesitate to use BitTorrent. I just have a tendency to download everything I get from that protocol to a different drive than its intended destination, so the final move will "defragment" it.

Re:Not really a fan of it.

[BitZtream]

If you're using a bittorrent client that doesnt' preallocation the file, then you should switch to a non-shitty one that preallocations, avoiding the entire fragmentation problem completely.

Or you could use a filesystem that didnt' suck.

Re:Not really a fan of it.

[wvmarle]

Defragging on any modern FS is easy: just copy the file to itself. That should do the trick to get a nicely defragmented file (assuming you have a long enough chunk of free space on your disk, of course).

Re:Not really a fan of it.

[magic maverick]

Regarding your first complaint, you do realize you can limit and throttle the connection?
E.g. where I am, my Internet connection is rubbish, so, during the day I throttle bittorrent to 10KB/s up & down. But, then, during the night, I let it run a little faster. It takes forever to actually download something large, but it works. And for some material, I'm one of only two or three seeders, so people downloading from me can damn well be grateful for what they get (and not complain about the horrendous speeds).

But then again, I'm probably just feeding a troll, seeing as you have a lot of similarly stupid complaints in your post.

Re:Not really a fan of it.

[Electricity Likes Me]

There are a lot of people out there using SOHO internet routers which are hard capped at 2500 NAT sessions. BitTorrent can easily take down the network by blocking outbound DNS until some of the sessions are dropped. It's an amazingly annoying problem, and it's not a well-advertised feature of routers (and getting over 10k is hard to find still for some reason).

Personally I find it staggering that a Linux box will handle 48k+ sessions no sweat on pretty much any type of hardware, but we're still well below the IPv4 port range in terms of allowable NAT sessions on most hardware.

Re:Not really a fan of it.

[magic maverick]

You know you can limit the number of peers you connect to as well?
E.g. I have it set to a maximum of ten torrents active at any one time. Each torrent can have no more than 60 peers. So, total, 600 peers maximum at any one time. Except, I also have it set so that I do not connect to more than 275 peers, ever. So that 600 number, irrelevant.

I.e. these are all problems that, well, aren't.

Re:Not really a fan of it.

[UltraZelda64]

While you're all saying "you know you can do this" and "you can do that too" to optimize your BitTorrent usage and make your network and Internet connection run decently , I can sit here downloading files at the maximum speed my connection will allow, with just one connection per download. Limit the rate of individual downloads, or better yet just queue them and they'll get done just fine. The problem with BitTorrent is NOT that clients come with sub-optimal settings (although they do) or that they need to be optimized for the local area network and Internet connection speed available (which is, again, true), but that they typically *require* you to do a decent amount to get good performance. Not just the seeding/downloading balance, but number of peers/connections (per torrent, total), speed limits (per file, total), etc.

Re:Not really a fan of it.

[UltraZelda64]

Linux, ext4, Transmission. I have found no compatible BitTorrent clients that I would want to use, and I wasn't aware that Transmission was "shitty" compared to its competition. I also was unaware that the ext4 file system "sucked."

Last I checked, it still produces ~100 fragments per file, though maybe this has been fixed since I last used it. It has been a while though. (Hey, if something else seems to work adequately or even better for my needs and preferences, why bother using it?)

Supposedly there is a mysterious "preallocation option 2" for the program that fully pre-allocates the space before writing anything according to a forum topic I just found, but I have no idea how to activate it: https://forum.transmissionbt.com/viewtopic.php?f=4&t=10013

That would at least solve the fragmentation problem (probably the biggest one), but the others would still remain. Lack of original timestamp would still be an annoyance, but the big problem would be network hogging--which, to be fair, can be somewhat alleviated with a lot of tweaking. But then, the lack of necessary tweaking is the entire reason I still use wget to download all files over FTP/HTTP in the first place.

[Pre-post discovery: The file ~/.config/transmission/settings.json houses that option. Nice way to hide it from view... without Google I would've never known it even existed. Pretty much everything else you might need is in the program's GUI/options screen. It looks like they pulled a GNOME here.]

Moving data with torrents

[woboyle]

I often download Linux DVD images and such using bittorrent. Bittorrent isn't "evil" - it is simple a means to share the load of moving large amounts of data. To make it illegal should be considered something akin to making automobiles and pickup trucks illegal - as they can move both large amounts of legitimate as well as other goods from point A to point B, and they can use many routes to get to B.

Only Legal torrents

[Vapula]

I use Torrent for many legal uses :
- Blizzard downloader (WoW, Starcraft2, Diablo 3)
- Humble Bundle
- Linux Distributions

But also, from time to time, some free tools, some F2P games that use BitTorrent for distribution, ...

Re:I use the bittorrent protocol sync my computers

[Anonymous Coward]

Ironically the site is blocked on my work PC due to it being involved with "Peer to peer" software.

Re:In a word? YES!

[bipbop]

Raffinate?

Of course: free DVDs and OS distributions

[Anonymous Coward]

I've downloaded DVDs released by the artists themselves onto bittorrent (and seeded it for weeks afterwards -- like the artists also wanted), and I've downloaded many Linux distributions that way. I seem to recall also downloading some huge Unreal Tournament 3 patches using bittorrent as well.

mimics my experiences

[erikscott]

I agree: torrent can't really saturate a 10GE... for that you should see something like bbcp, which will quite handily flood a 10gig ethernet and then some. :-)

NC State University uses torrent to let students download some commercial software so they don't have to hand out DVDs... they distribute SAS that way for certain, probably a few others.

ibiblio had someone who developed sort of a "perma-seed" to use torrent for some sort of archive-like thingie. I know Paul Jones is probably reading this, perhaps he would like to comment? :-)

Re:mimics my experiences

[swillden]

I agree: torrent can't really saturate a 10GE

Point-to-point torrents can't saturate a 10GE. Get enough nodes involved and they can.

Re:mimics my experiences

[wagnerrp]

That's stupid. Why use bittorrent when a standard http server would do just fine? It's not like they don't have nice servers sitting on the campus backbone ready and available to do precisely that.

Re:mimics my experiences

[adolf]

A better question is: Why not?

Some BT clients will seek out local peers, which helps people who are quite near eachother (in the same building, say) while reducing server and inter-building network load.

For free, more or less.

"Hey, boss! I might be able to reduce the load on our server-box by maybe 40% on install day, and keep things from grinding to a halt!"

"Oh, really -- using that BitTorrent crap? Forget it, lackey - we'll just use HTTP like everyone else. After all, the server is on the backbone, so it's not like it matters. The students can just wait if things get bogged down."

"But boss, with BitTorrent, things get faster as more people use it, and it can't grind to a halt."

"HTTP. We're done here."

Stupid? Yeah, what you said was pretty stupid.

Re:In a word? YES!

[sdreader]

You folks that torrent movies and stuff that is not in the public domain are crazy in my book.

Just out of curiousity, why?

Movies and TV programs.

[godel_56]

I've read that in Australia both the big, legitimate movie distributors and the major TV networks use (or have used) torrents to move programs around, between theaters (with digital projection facilities) and networked TV stations.

TA Spring maps

[Anonymous Coward]

The open source real-time strategy game TA Spring client used bittorrent to transfer the next map from the players who have it to the players who don't.

Too bad my university dorm blocks torrent traffic...

Yup, lots, pretty much always legal.

[seebs]

I pretty much always use torrents to download OS installers (Linux, etc.). I don't know whether I've ever downloaded anything that wasn't totally legal. Probably have at some point, but I mostly use torrents for things where I want to be contributing bandwidth back to the community.

Re:In a word? YES!

[VortexCortex]

You folks that torrent movies and stuff that is not in the public domain are crazy in my book.

"You black folks that ignore corrupt laws and sit at the front of the bus are crazy in my book!"

Guns, drugs, ...

[Ungrounded Lightning]

Punish the technology because of how it is used? I thought we grew past that notion already.

If we had you'd be able to buy any weapon or drug you want without government interference or oversight.

The NRA could go back to its original functions of training and research, and the FDA and DEA to could be replaced by Underwriters' Laboratories and Consumers' Reports.

As you can see we have a long way to go.

Re:Guns, drugs, ...

[Anonymous Coward]

People who are drug and gun fanboys can always find a way of squeezing in the topic of drugs or guns. Whereas the majority of us want nothing to do with either. They are the last resort of the inadeauate paranoid, and therefore perennially popular on slashdot.

Re:Guns, drugs, ...

[Ungrounded Lightning]

Seems to me the two issues are poster children for massive instutionalized policies of attacks on a technology rather than the misuse of it.

Massive to the point of billions, perhaps trillions, of dollars wasted and far more damage done that prevented.

Think you've seen massive overreaction to misuse of torrents? Compare it to "The War On Drugs" and you ain't seen nothing yet.

Honorary Troll Golden Bridge Award of 2013

[Anonymous Coward]

Your post has been chosen as a contender in the 2013 Honorary Troll Golden Bridge Award on the following merits:
- Exemplary super high quality trolling.
- Deadpan mastership at legendary level.
- 100% factual content.
- Use of appropriate verbal abuse.
- Entirely misplaced and off topic.
- Seamless application of "apparently genuine good advice".
- Successful high-level or maximum mining of likes, pluses, thumbs' up, points, or karma or similar localized accolades (maximum "+5 Funny" in your case).
- Avoided proper classification as trolling except by Troll connoisseurs.

You are thus eligible to win the 2013 HTGBA the results of which will be announced medio March 2014 by the HTGBA Mighty Three Election Jury. The HTGBA Award Ceremony is scheduled to take place in Oslo, Norway on the 1st of April 2014.

Sincerely,
Rudolf Reinsdyr HTGBASC HTGBAMTEJ HTGBAFT (PRO)
rudreinsdyr@kongehuset.no
Oslo, Norway, 25th of April 2013

HTGBA, Monolittbroen 1a, N-0266 Oslo, Norway
Main visitors entry along Kirkeveien 1 to 30.
Directions: follow the tourists to the main gate or ask a tourist guide for directions to "the bridge in the Vigeland sculpture park", approach the bridge from the main entrance but do not walk over it, instead right before the start of the bridge take to the left and you will see a small curving paved road going down along the bridge, follow it down to the waterfall, jump over the bushes towards the waterfall and walk through the waterfall and you've arrived! Please show appropriate caution against currents and drowning when entering the waterfall and searching for our entrance door, HTGBA will not accept responsibility for deaths or injuries.

P.S. HTGBA website will be up shortly, watch out for our press release!

Music and Linux Distros

[Anonymous Coward]

I primarily use a torrent client when I'm after legal Music and Linux Distros.

I'm a distro hopper, and have a bunch of computers around here, so I'm always trying out something 'new', and I download a whole lot of music, a whole lot, as there's an endless supply of free legal music available.

Re:Music and Linux Distros

[tehcyder]

I download a whole lot of music, a whole lot, as there's an endless supply of free legal music available.

Yes, in the same way that there's an endless supply of free fanfic available, i.e. utter shite composed in someone's bedroom while they're half drunk and masturbating to Japanese child porn.

http://gameupdates.org/

[Anonymous Coward]

http://gameupdates.org/

Re:In a word? YES!

[Anonymous Coward]

Man, you really just did that, didn't you?

Witchcraft!

[seanthegeek9306]

Look at the mobile app. An iOS UI on a Galaxy Nexus. What witchcraft is this?

I really want to use it for software deployment

[Anonymous Coward]

I really want to use torrents for software deployment on a LAN.

I work as a sysadmin for a school division (primarily using Mac computers), and if 60 computers at one site need 3 GB of (legally licensed) GarageBand loops in the middle of the year, it seems ridiculous to copy them one at a time -- I'd love to be able to tell all the clients that need the files, 'Hey, your data is over here. Grab the torrent and install it.'

Even better than that would be if I could image the student workstations using BitTorrent. Yeah, I've tried multicasting; with the software I've had available to me, by the time you've tuned the speed so that it works right at one site, you could've unicasted it to each client and moved on. Here is a feature request I've made to that end: http://www.deploystudio.com/Forums/viewtopic.php?id=1935 . I did try adapting 'Twitter Murder' to do this, but never really made it work.

I'd be thrilled to be able to copy large files on a LAN easily using a torrent, and even more so if I could write to raw disk devices and clone filesystems. (I'd also love a tool that would do a multicast copy an adapt the speed of the stream based on what the clients are able to handle.)

Re:I really want to use it for software deployment

[Electricity Likes Me]

BitTorrent really needs some better client-side tools to make use cases like this more common.

Though that does get me wondering if you couldn't modify say, Nautilus, to automatically export torrent files (or generate on the fly torrent files) of files available on computer shares and keep a track of them so it would automatically use the torrent to download them from a local machine.

Quite a lot

[Anonymous Coward]

I use it a lot, and lately, more and more of the data has been for fully legal items, certainly for the last 5 months or so there were no more illegal downloads through bittorrent.

Things I get with it:
Linux iso
Certain games like black mesa and goldenEye source
Often other bigger opensource projects, for example openoffice (although I stopped using it, having google docs and latex now).
The blender movie projects also released through bittorrent.

In the past, I also got wow patches through bittorrent
and currently I use the new Synch also, its great to keep bigger files on multiple platforms and will be wonderful at small lanparties to ensure everybody has the game we are going to play quickly enough.

For anything that is both quite big and has at least one moment of being wanted, bittorrent is wonderful. Linux isos on release day, black mesa on release day. All of these would have been lots harder if not impossible for smaller independent groups if not for bittorrent.

Even for illegal downloads, the reason I use bittorrent is because it usually is a better service than whatever else available. Its quick, its easy to find the link, it usually allows you to get the content in whatever way you wish for whenever you wish. Especially with non-steam games I will often download a bittorrent version simply because I know it will always work.

Yes, to distribute software on a closed network

[Anonymous Coward]

For the product I work on, we use torrents when distributing updated software to all of the devices on the network (~300 nodes). It is a closed network with redundant control units that serve the data, and all others load via update, then kick off the install. It makes the loading process rather resilient, at a slight expense of efficiency (we could just broadcast the data to everyone, which would mean custom software to accomplish that vs. a torrent library).

Re:Yes, to distribute software on a closed network

[BitZtream]

'custom software' ... right because there aren't any libraries for multicast file distribution already or anything.

You fail at the Internet. Multicast file distribution is probably older than you are.

Yes: Linux distros

[cybersquid]

Mostly SuSE. Some kubuntu, CentOS and Oracle OEL.

Wait...

[ericdano]

Wait, you can move illegal data using torrents? Who would have thunk!?!

Haven't used Torrents in years. Many years.

Re:+1 Linux distros. Only for multiple recievers

[thereitis]

That's not true: Some protocols are very "chatty" and may require, say, 1 round trip acknowledgement per file transferred. This is where rsync's streaming protocol shines.

Even at home I noticed that copying large files over wireless via SMB is much slower than copying them over an SSHFS mount (getting previously unseen transfer rates, actually). However, the SMB mount is more responsive when exploring files.

In my experience, protocol can matter a lot.

Yep.

[mindbooger]

Linux distros. After I'm done, I try to keep seeding for the next night or two to try to "give back" at least a little bit.

yes I do.

[davydagger]

many open source projects with large downloads like .ISOs for linux distribution use torrents because paying for bandwith would be really expensive for direct download.

bit torrent is great, because everyone with consumer broadband becomes and instant mirror when they download it.

Re:In a word? YES!

[Anonymous Coward]

I hate torrents. Administrators and users of torrents are thieves with elaborate excuses.

Rare in /.

[Anonymous Coward]

I don't use BitTorrent much, because before I download something I make sure it is free, in the sense that the people who created the content want it to be distributed without being compensated). Early on there was a decent amount of that content (America's Army (not a great game, but free), Creative Commons, etc. ) but it soon got to the point that I had to do so much research to see if I was ripping someone off that I stopped installing a bit torrent client on rebuilt computers.

Re:Rare in /.

[Anonymous Coward]

Also rare on /., I've never really found a need or use for it. Bandwidth has been large enough for long enough that any large data transfers complete in a reasonable amount of time and never need to bother.

Re:In a word? YES!

[jones_supa]

Actually, it's funny when you think about it. Legally freely distributable open source software is important for folks here, but the same people pirate copyrighted movies and stuff. But if you instead torrent culture that is in public domain, there is no legal problem and you can enjoy completely free digital lifestyle.

Re:Can you use it to move a 300 megabyte hosts fil

[jones_supa]

The $10,000 challenge!

Re:In a word? YES!

[Anonymous Coward]

You folks that torrent movies and stuff that is not in the public domain are crazy in my book.

Private Tracker + Block List + Non-Logging VPN == Not so crazy

Re:+1 Linux distros. Only for multiple recievers

[wagnerrp]

I wonder how much of that is poor CIFS implementation or configuration. I can do 115-120MB/s either way between my (Win7) desktop and my SAMBA server in my basement.

I use it for physical distribution

[Anonymous Coward]

Or people too cheap to buy physical copies through the oldest of physical multicast networks....the store.

Re:In a word? YES!

[wvmarle]

Very, very bad analogy as the concept of copyright is a good thing, the concept of racism is not.

Remember that only thanks to the existence of copyright the GPL and all those other FOSS licenses can exist. Without copyright it'd all be public domain.

It is the current implementation of copyright laws (primarily: too long terms) and all the other laws that surround it (e.g. DMCA blocking reverse engineering and circumvention of DRM) that is the problem.

Re:In a word? YES!

[Anonymous Coward]

Man, you really just did that, didn't you?

Uh, yes, he really just pointed out that GGP's implicit message that illegal==insane was fucking rubbish.

If GGP had a more nuanced point, such as that violating copyright is insane because copyright law is demonstrably just, maybe he should have made that point, and then nobody would have demonstrate the non-link between legality and sanity by pointing out examples of laws even more egregiously unjust than copyright.

Microsoft ISO's

[batwingTM]

We use bittorrent to move MS ISOs at work. We are an Educational institution and as such we are allowed to offer students access to MS products. (not the Office suite, but OS's .NET and the like) This is done via the Dreamspark site where students can get their license keys but from our internal network they can use bittorrent to get the ISOs

So we legally give access to MS OS via torrents (^_^)

Re:+1 Linux distros. Only for multiple recievers

[Drakonblayde]

BitTorrent is, conceptually, best used as a reimplementation of multicast. Multicast is probably far more efficient when it comes to the actual data distribution, but multicast (specifically, routing multicast) is one of those blackbox things that not a huge number of people understand. Last time I checked, I couldn't route a multicast source from a Comcast connection and have the data arrive on a receiver on the Cox network.

However, there was still a need for a protocol that could effectively do one to many distribution for the masses. Enter bittorrent. It fills the void nicely, and has the additional benefit that once a receiver has all the data from a given source, it then becomes a source itself, thereby increasing the aggregate bandwidth available for the feed, as well as making the feed resilient.

1 RTT per GB? For a Maildir mail spool, yes

[raymorris]

The GP referenced isos and "10s of terabytes of data". Unless those tens of TB are BILLIONS of tiny files, 1 RTT per file would be less than 0.01%. For lots of tiny files, like Maildir, yes SMB sucks. Of course SMB is Microsoft, so the fact that it sucks is assumed.

TFS

[thatkid_2002]

"I sometimes use torrents to move several GB of data, especially when pushing large bundles to multiple destinations."

Which, to my knowledge - Bittorrent achieves better than any in your list of 10. If I had to guess, the part about pushing to multiple destinations is the first part of the main reason Bittorrent was chosen. The second part is that once data is uploaded to another node, you have *two* links to supply uploads to further nodes - and then three, four, five... When moving data across a non-uniform network topography this is useful.

New FreeBSD Install ISOs

[utkonos]

Torrent has been the only way that I've downloaded FreeBSD ISOs when they're released for at least 5 years. It is lightening fast compared to even the fastest HTTP or FTP mirror.

Yes, low budget film

[ssam]

I was involved in making a low (GPB2000) budget film, which was released with a CC-BY-NC licence. For distribution we used youtube, vimeo, archive.org and bittorrent. Youtube was by far the most popular option for downloads (over 90%), but bittorrent was the only method that worked for the highest quality (~20GB 1080p file). we set up our own tracker, because we could not get clearbits to work. we seeded from a friends colo server, VPS and a few home broadbands.

Re:In a word? YES!

[number11]

Legally freely distributable open source software is important for folks here, but the same people pirate copyrighted movies and stuff. But if you instead torrent culture that is in public domain, there is no legal problem and you can enjoy completely free digital lifestyle.

You mean, culture that is more than 70 years out of date?

Podcasts?

[wagonlips]

Wouldn't bittorrent work well for podcasts? Seems like the perfect case but I've never heard of it being done.

Re: In a word? YES!

[Anonymous Coward]

Regardless of the soundness of his argument, he could have used a metaphor that was easier to parse.

Re:In a word? YES!

[Zontar The Mindless]

But if you instead torrent culture that is in public domain, there is no legal problem and you can enjoy completely free digital lifestyle for all of about 10 minutes before you realise that you've consigned yourself to a life of bad fanfic, worse poetry, and unimaginably bad amateur electronica courtesy GarageBand. And then kill yourself.

SYSTHLSOTET...

Re:In a word? YES!

[Zontar The Mindless]

Yes, he did, and I should like to offer him a cigar.

Re:BitTorrent is a silly way to move files

[gl4ss]

because he doesn't want to build his own broadcasting protocol.

it's a good way to move data from one target to multiple recipients unless you're in total control of the whole network flow.

Copyrighted material

[loufoque]

I move around a lot of copyrighted material.
This is allowed by fair use of course.

Re:Copyrighted material

[Fruit]

posting to undo moderation. sigh.

In France...

[Scotch42]

...the so called "freebox" (ADSL modem with lot of extra features) includes a bittorrent client. I use it to get and share the full gutenberg project.

Yep

[spxZA]

Bioinformatic data, Linux distros, F/OSS games, plus I wrote a libtorrent client to distribute multimedia for a digitical signage company.

1 file & multiple Internet connections

[xororand]

BitTorrent is great for transferring large files with multiple slow Internet connections.
Seed with each connection, leech with the destination machine.

Humble Bundle

[cgimusic]

The Humble Bundle HTTP downloads are a little on the unreliable side. I often get a "completed" download and then the MD5 doesn't match and I have to redownload. Now I use Bittorrent and get a perfect file every time.

Re:Humble Bundle

[apcullen]

Agreed! I always use bit torrent when I buy a humble bundle.

Re:BitTorrent is a silly way to move files

[serviscope_minor]

The only purpose it serves is to prevent leechers from taking and not giving back while moving your warez around.

Yeah and you're a drunk driving pedo terrorist.

You see if you can make up random facts about people then I can too!

[looks at torrent list]

CentOS-5.9-x86_64-bin-DVD

Ah yes, CentOS, that well known piece of warez.

Almost

[DrYak]

To be fair, rsync + ssh is equally secure as scp. I actually think rsync uses scp in that situation (please correct me if I am wrong).

Almost. Technically, ssh (for shells), scp (for files), sftp (files too), and rsync+ssh, all run over a secured openSSL connection.
What change is what is running on the other side.
By default, the remote SSH just runs a shell and pipe its I/O to you local terminal, through a secure connection.

The other combinations all launch a specific program at the remote end (either by asking for it, or in the case of sftp, as specified in the server's configuration) which handles the remote part, while communicating over the same kind of secure openSSL connection as ssh does in shell mode.

So rsync uses rsync, not scp, but it does it over an encrypted ssh connection anyway.

scp vs. rsync

[DrYak]

...and if the connection breaks or anything happens, you'll have to restart copying everything from the beginning.

meanwhile rsync is able to do only partial sends.
That means rsync will only send the parts which were not transferred the first time.

that's also useful for updates, for incremental backups, etc.

Re:scp vs. rsync

[denmarkw00t]

...and if the connection breaks or anything happens, you'll have to restart copying everything from the beginning.

Interestingly, rsync can even resume a transfer that was started by scp but then interrupted - neat!

Re:In a word? YES!

[Anonymous Coward]

public tracker + block list + not living in america + only downloading films that are more than a year old == no problems for the last ten years

Did you for get to turn the encryption?

[Anonymous Coward]

Encrypted data is still safe to transfer.

Speaking of scientific application

[DrYak]

Tranche is another peer-2-peer distribution software which has become the almost official standard for the huge masses of data found in life science (which, although aren't LHC-grade abominably over-massive, still can clog networks and would be a pain to distribute using classical server methods).

Re:Can you use it to move a 300 megabyte hosts fil

[Anonymous Coward]

I don't get it.

Re:In a word? YES!

[tehcyder]

You folks that torrent movies and stuff that is not in the public domain are crazy in my book.

"You black folks that ignore corrupt laws and sit at the front of the bus are crazy in my book!"

I think you've hit a new low in the old slashdot game of "let's make a stupid fucking analogy".

Some of my vendors use BitTorrent to distribute..

[Anonymous Coward]

.. new versions of their products, but the University where I work (~2Mbps) throttles BT traffic, so I usually do it from home (where I have ~82/32Mbps fiber) and then copy it to my work machine over VPN, or put it on a USB key or DVD. Annoying.

It would be used even more

[apcullen]

If there were a way to make it ad supported. A lot of android developers and minecraft modders seem to use ad supported lockers, which are a pain and way less reliable than a torrent.

lots of software

[Anonymous Coward]

I bittorrent a lot of pirated software. However, it's really frustrating when just the torrent you need isn't being seeded. I think there should be a required period of time after you download a torrent that you should be seeding for

Re:In a word? YES!

[bobbied]

The chances of getting caught by a patent troll firm is pretty high because it is extremely easy to find you by IP address. Of course, it depends on where you live. Here in the US it's pretty risky, other countries may not have a legal system that is as easy to use by patent trolls.

Of course you *could* go out of your way to side step their ability to find you by IP address, but when you do, you just bypassed the benefit of using a torrent to get the material and would likely get it faster by just transferring it direct and bypassing all the risks of a Torrent.

So I don't engage in distribution of stuff that doesn't come with a license that allows me to make copies and distribute. First, it's too legally risky and more importantly I don't think it is morally and ethically right....

Que loud discussion about the moral and ethical nature of patent and copyright laws by the "everything should be free but keep sending my paycheck to me" crowd in three, two, ......

Re:Can you use it to move a 300 megabyte hosts fil

[Anonymous Coward]

Naw. Wouldn't work effectively. Every time you change the file, the hash changes, which means you now need to go to each and every single computer to tell it to download the new hash. It would not be that much different than having the modified HOSTS file on your USB and copying it into each computer.

We need to design a new system that distributes these names to all the computers. I propose that this new system be called Distributed Name System, or DNS for short. :D

define legal

[Anonymous Coward]

"conforming to or permitted by law or established rules"?

Not often... Arch Linux only distributes the install media by torrent I believe.

The real question is: "Do you use it to download things that SHOULD be illegal?"
My answer is no.

Games updates?

[Anonymous Coward]

League of Legends, World of Tanks and many other games use torrent to allow faster updates while reducing usage of their own bandwidth.

Re:In a word? YES!

[poofmeisterp]

Raffinate?

(r)egistered AWESOME!

Re:Can you use it to move a 300 megabyte hosts fil

[poofmeisterp]

inquiring minds want to know.

Let's test it to find out. Just make sure the hosts file name has .iso at the end.

Re:In a word? YES!

[SiChemist]

I think you are confusing/conflating the terms patent and copyright. As for being targeted via IP address, just use a VPN. It's not rocket surgery.

Re:Can you use it to move a 300 megabyte hosts fil

[Anonymous Coward]

try browsing comments at -1

Re:In a word? YES!

[sdreader]

The chances of getting caught by a patent troll firm is pretty high because it is extremely easy to find you by IP address. Of course, it depends on where you live. Here in the US it's pretty risky, other countries may not have a legal system that is as easy to use by patent trolls.

I live in Australia and although some ISPs have passed on warnings, my ISP (Internode), which is one of the largest in the country, does not. Honestly it's counter-productive for them to do so - if people were being threatened for downloading copyrighted material, they're probably end up moving to a small download quota plan since a large portion of what they'd use it for is no longer safe to do so. Lower quotas = cheaper plans = less money for ISP.

I don't think it is morally and ethically right....

To be honest, it a lot of cases I agree with you. Despite the fact that copying content is trivial, that does NOT make it worthless in a monetary sense and I believe in a fair exchange of money for product model, so long as its reasonable. But when some (old, obscure) content is not even available legally anymore, or not available in your country because no-one can be bothered to sell it to you... that's when the lines of morality and ethics start to blur.

Re:In a word? YES!

[sdreader]

It's not rocket surgery.

Shit, I'd LOVE to have some rocket surgery!

Humble Bundles

[deek]

Amongst the other uses already mentioned, I use bittorrent to download Humble Bundle games. Faster for me, less of a load on the host server. We all benefit.

PARTS! PARTS of files

[raymorris]

scp has a flag to transfer only the 1KB part that changed within a 100 MB log file? And FYI you're simply wrong about rsync. In fact, the default, when copying files over the network, is to not even copy modified files, only the modified parts. You have to use --whole-file to change that.

Useful for distributing scientific data

[one-egg]

I use BT to distribute large files from the SNIA IOTTA Trace Repository (http://iotta.snia.org/). Although there are typically no swarms, BT is still useful for a number of reasons, including in particular the ability to manage large collections of related files and the ability to deal with intermittent connections.

Unfortunately, many of my users work at sites that block BT, forcing them to revert to a horrible HTTP option.

And no, rsync isn't a solution for our situation.

As to what is needed, the primary thing is better tracker and seeder daemons. I use opentracker, which is OK but hardly perfect. I seed with deluge because it's one of the few seeders that can be run as a daemon (almost all BT clients expect you to dedicate a GUI window to them or they stop running--imagine what running a Web service would be like if you had to have a GUI for every instance of Apache).

PlanetLab as a good example

[Anonymous Coward]

Some PlanetLab components (it's a distributed platform for network research) use the BitTorrent protocol to distribute software upgrades to all of its 1000+ nodes.

Re:Nope.

[Anonymous Coward]

You're doing it wrong. This will become a long reply.

1. Use no .zips or .rars or anything like that. Why?
1.a. Torrents compress your data for transmission and solely for transmission efficiency (different from storage like .zip and .rar because packet sizes matter) which is better in this case than anything else.
1.b. People who torrent .zips and .rars make seeding harder for the rest because many will extract and then delete the torrent file as soon as they get it.

It's stupid so don't do it that way!

If you have to force structure you can do it differently and in the case of a comic of static pictures you should consider implementing a straightforward naming scheme for your "page images"/strips/equivalent and seed them individually.

E.g.:
"Title-uniqueID-pageID.format"
"MySuperComic-#01-00.png"

Or similar as long as it only increments in a straightforward manner. Don't use names like *-00frontpage.*, keep it simple and numerical.

You could do this for .swf. or .html. or anything really.

Result: tons of small torrents that are fast to complete and which gives the leechers/seeders instant access and trial (tell them to start at the beginning i.e. *-00.*).

If people want it in a folder they can make the folder themselves and move the torrents there without interrupting seeding.
Alternatively you can put it in a folder but only make a seed of the entire folder (in addition to the rest) when your comic or chapter or whatnot is done.

There won't be any shared chunks when each page is an individual torrent. Archive issues are not applicable when you don't (mis-)use archives.

2. Updating/pushing/pulling is not for torrents to handle, a lot of people use rss feeds for this and it's a good solution so set up a rss feed solely for your specific comic and point people in the right direction. Huge amounts of content is distributed this way *nudge* *nudge*.

3. Content must always be used in its appropriate application no matter what. For comics all free/open source comic readers I've seen and used handle reading consecutively named files without having to do anything at all (it's a default that would have to be turned off). As long as it's in the same folder they handle at least .jpeg/.jpg, .gif, .png and often .tiff in addition to various archive formats (which you don't want to use anyway). Default f/oss image viewers also do this.

Consecutive series of .html and .swf could require a skeleton page for loading it all into a web browser but it's not impossible. In the .html case it can be as simple as making every .html page it's own torrent and in the .html. include appropriately used <a>-tags.

Like this:
<a href="NameOfNextPage.html">Next page</a>;

Using the naming scheme above you already know what the name will be even if you haven't made it yet. Do the same for going back pages and you're set. N.B.: make sure you use relative paths (if any).

If you want it all on one page you need to use a NameOfComic-Index.html file with a simple javascript snippet that loads all local instances of NameOfComic-Number.html files in the same folder (using relative paths if needed) from 00 to 99 or similar (simple while or for loop from 00 to 99).

Either way they only have to click the torrented .html file to open it in the browser, there's nothing simpler than that.

4. Video? Have you heard of anime? At least a petabyte of ongoing series are transmitted daily using torrents, episode by episode as well as completed series or seasons and old stuff.

You have unreasonable demands, bandwidth is not an issue for torrents, seeders are. But if you're legit then seeding is not a prob