Ask Slashdot: Do You Move Legal Data With Torrents?

An anonymous reader writes "We've recently seen a number of interesting projects come from bittorrent.com, including Sync and SoShare. I sometimes use torrents to move several GB of data, especially when pushing large bundles to multiple destinations. It's mostly a hodgepodge of open source tools, though. Apart from anecdotes and info from bittorrent.com, details are thin on the ground (e.g. the Blizzard Downloader). I have two questions for the Slashdot community. 1) Do you use BitTorrent to move data? If so, how? i.e. What kind of data and what's the implementation? 2) If you've looked at torrent clients/tools, what's missing in the open source ecosystem that would make it more useful for moving around large blobs of data?"

Comments

I use it for linux distributions

[Qzukk]

The entire point of swarm topology is to move data to a lot of places at the same time. If you just need to get data from A to B without sharing it with anyone else, rsync it.

Re:I use it for linux distributions

[TooMuchToDo]

While working at Fermilab on the LHC CMS data taking team, I used bittorrent to speed up re-installs of thousands of worker nodes. I was able to saturate 10Gb Ethernet links this way, and could reinstall ~5500 Linux boxes within 10-15 minutes (with only two initial OS source servers).

Yes, Bittorrent is not just for piracy.

Re:I use it for linux distributions

[corychristison]

To be fair, rsync + ssh is equally secure as scp. I actually think rsync uses scp in that situation (please correct me if I am wrong).

The advantage I see rsync having is it is useful for automated (backups) of a large collection of files vs gzipping and copying via scp.

Although git also does a great job of that with concurrent revisioning built in.

It all boils back down to using the right tool for the job.

Re:I use it for linux distributions

[i.r.id10t]

Yup, it is one of the few ways that darn near any user out there can contribute back to Open Source.

I can code, some, but not very experienced with c/c++ - just haven't had the need to do it. I hate writing documentation. I do file bug reports, but the stuff I use is pretty darn stable.

So, to give back, I seed iso images for 24 hours or 100gb in upload of any Linux distro release notice I see here on slashdot, even if I don't use that distro.

Yes, I have

[EkriirkE]

Many times a person is searching for a program to do something by keyword instead of software title, and for free. Torrent sites are a common place to go for something free. I just generate a .torrent for my software(s) and upload it to a few big trackers and the others seem to pilfer it from there. Just make sure the filenames and titles are relevant. It's like SEO, but TTO: Torrent Tracker Optimization.

gittorrent

[lkcl]

the one thing that would help enormously would be to have git be *truly* peer-to-peer distributed. not "yeah shure mate you can always git pull and git push, that's distributed, and you're a peer, right, so... so... git is peer-to-peer and distributed, so what are you talking about you moron??" but "at the network level, git pull and git push have a URL type that is **TRULY** peer-to-peer distributed. to illustrate what i mean, i would like to be able to do the following - with all that it implies:

git clone magnet://abcdefg0123456789/gittorrent.git

if you're familiar with magnet links, you'll know that there is *no* central location: a DHT lookup is used to find the peers.

now, what wasn't clear to the people on the git mailing list when i last looked at this, was that it is possible to use bittorrent to do git pack objects, by creating a file named after the pack object itself. and what wasn't clear to sam (the last person who tried to put git over bittorrent) was that you *MUST NOT* make use of bittorrent's "multiple file in a torrent" feature, because bittorrent divides up its data into equal-sized blocks that *do not* line up with the files that are in them, which is why when you download one file in a torrent you almost always end up with the end of its preceding file and the start of the one after it, as well.

the idea i came up with is that you create *multiple* torrents - one per git object (or git pack object). if you want to pull a tree, you create a torrent containing *one file* which is the list of objects in that tree; gittorrent would then know to map each of those objects onto yet *another* torrent (one per object), repeat until all downloading happily. gittorrent objects are of course named after the hash, so you can pretty much guarantee they'll be unique.

and, adding in a DHT (a la magnet links), you are now no longer critically dependent on something like e.g. github, or in fact any server at all.

to answer your question in a non-technical way, mr anonymous, i think you can see that i feel it would be much more useful to have development tools that use bittorrent-like protocols to share files-as-revision-controlled-data (and, if you've seen what joey hess is doing with bittorrent you'll know that that's a hell of a lot - including storing home directories in git and doing automatic distributed backups)

Facebook Does

[terbeaux]

Facebook deploys its 4GB binary to its 500,000 servers using a torrent client that has rack and switch affinity. Each client goes for data chunks that are already locally on a rack or switch that it is connected to. That is a crap-ton of data.

SOLR at Etsy.com?

[rwhiffen]

The folks at Etsy do it to replicate SOLR:
http://codeascraft.etsy.com/2012/01/23/solr-bittorrent-index-replication/

Not sure if that's what you mean.