Have a look at bup (https://github.com/apenwarr/bup). Though still very new and missing some features, it's pretty stable, fast (at deduplication, restore is another story) and very effective. Better ratio than lessfs, sdfs and zfs. I'm rather impressed how it sucks in full partition images, several hundred gigabytes, each day at my place. It's meant for virtual machine images. File based backup is also included, but still missing meta data AFAIK.
Also, have a look at Btrfs. Btrfs does not include deduplication per se yet, but you can use volume snapshots and rsync --inplace so it only does block level changes within updated files, if filenames don't change (watch/var/log).
Wow this was disappointing. I was hoping for an actual hardware cache that can be put between HDD and mobo, with an integrated SATA host. This is a software only solution which can be applied to any SSD.
Setup a cheap file server in a datacenter, hook it up into your VPN network and store all backups there. Use rsync - very fast, uses SSH nowadays for auth and encryption. Encrypt the whole backup partition (dmcrypt, truecrypt, etc.) and keep the key private. Manual mount and key entry after rebooting. That way datacenter operators can't (easily) gain access to the files. Or transfer already encrypted files, which will destroy rsync performance, though.
Set SSH and all the other services to listen on the VPN IP only, making the machine invisible to the common internet.
Not as fancy as Peer-To-Peer distribution, but very reliable and fast. Also you get less administrative headaches, I think.
On a side note, GuildWars has an artificial limit of 1.000.000 Gold in chest (aka "bank"), which is "worth" only 60 bucks. It took me > 2400 hours of casual gaming to achieve this amount of Gold.
Slashdot Mirror
Have a look at bup (https://github.com/apenwarr/bup). Though still very new and missing some features, it's pretty stable, fast (at deduplication, restore is another story) and very effective. Better ratio than lessfs, sdfs and zfs. I'm rather impressed how it sucks in full partition images, several hundred gigabytes, each day at my place. It's meant for virtual machine images. File based backup is also included, but still missing meta data AFAIK.
Also, have a look at Btrfs. Btrfs does not include deduplication per se yet, but you can use volume snapshots and rsync --inplace so it only does block level changes within updated files, if filenames don't change (watch /var/log).
Wow this was disappointing. I was hoping for an actual hardware cache that can be put between HDD and mobo, with an integrated SATA host. This is a software only solution which can be applied to any SSD.
Setup a cheap file server in a datacenter, hook it up into your VPN network and store all backups there. Use rsync - very fast, uses SSH nowadays for auth and encryption. Encrypt the whole backup partition (dmcrypt, truecrypt, etc.) and keep the key private. Manual mount and key entry after rebooting. That way datacenter operators can't (easily) gain access to the files. Or transfer already encrypted files, which will destroy rsync performance, though.
Set SSH and all the other services to listen on the VPN IP only, making the machine invisible to the common internet.
Not as fancy as Peer-To-Peer distribution, but very reliable and fast. Also you get less administrative headaches, I think.
On a side note, GuildWars has an artificial limit of 1.000.000 Gold in chest (aka "bank"), which is "worth" only 60 bucks. It took me > 2400 hours of casual gaming to achieve this amount of Gold.