BitTorrent For Enterprise File Distribution?

HotTuna writes "I'm responsible for a closed, private network of retail stores connected to our corporate office (and to each other) with IPsec over DSL, and no access to the public internet. We have about 4GB of disaster recovery files that need to be replicated at each site, and updated monthly. The challenge is that all the enterprise file replication tools out there seem to be client/server and not peer-to-peer. This crushes our bandwidth at the corporate office and leaves hundreds of 7Mb DSL connections (at the stores) virtually idle. I am dreaming of a tool which can 'seed' different parts of a file to different peers, and then have those peers exchange those parts, rapidly replicating the file across the entire network. Sounds like BitTorrent you say? Sure, except I would need to 'push' the files out, and not rely on users to click a torrent file at each site. I could imagine a homebrew tracker, with uTorrent and an RSS feed at each site, but that sounds a little too patchwork to fly by the CIO. What do you think? Is BitTorrent an appropriate protocol for file distribution in the business sector? If not, why not? If so, how would you implement it?"

Comments

Re:Different torrent client ?

[Anonymous Coward]

rtorrent watching a directory for .torrent would be the way to go. And then use unison to keep the .torrent directory in-sync.

How I would do it...

[LuckyStarr]

...is quite straight forward in fact.

1. Create a "Master" GnuPG/PGP Key for yourself. This key is used to sign all your data as well as your RSS feed (see below).
2. Set up an RSS feed to announce your new files. Sign every entry in it using your "Master-Key".
   • All the stores check the validity of your RSS feed via your public key.
   • All the stores have one (or the same) GnuPG/PGP key to decrypt your files. The beauty of GnuPG/PGP is that given many destinations you can encrypt your data so that every recipient (each with their own key) can decrypt them. Nice, eh?
3. Set up a standard BitTorrent server to distribute your files.
4. Announce all your new files via your RSS feed.

This has many advantages:

The beauty of this system is that it relies heavily on existing technology (BitTorrent, RSS, GnuPG, etc), so you can just throw together a bunch of libraries in your favourite programming language (I would use Python for myself), and you are done. Saves you time, money and a lot of work!

Furthermore you do not need to have a VPN set up to every destination as your files are already encrypted and properly signed.

Another advantage is: As this is a custom-built system for your use-case it should be easy to integrate it into your already existing one.

it's called dsync

[slashdotmsiriv]

and you can find documentation for it here:
http://www.cs.cmu.edu/~dga/papers/dsync-usenix2008-abstract.html

It is rsync on steroids that uses a BitTorrent-like P2P protocol that is even more efficient because it exploits file similarity.

You may have to contact the author of the paper to get the latest version of dsync, but I am sure they would be more than happy to help you with that.

Re:Cisco already makes a product to do this - WAAS

[Bert64]

Bittorrent will transfer the differences too, if you make a new file overwrite an old one, it will replace any chunks which are different.

Re:load balancing?

[abigsmurf]

It's not usually the bandwidth that kills networks for BT (although it does if you're not careful), It's the hundreds of temporary connections and half open connections it does. Lots of routers weren't designed for this and give up the ghost. You can configure them to lessen this (port forwarding, limiting it in the client) but some routers just can take it. There are also lots of routers which degrade over time with heavy BT usage and need occasional reboots.