Slashdot Mirror

and don't forget the excellent EnhanceIO, https://github.com/stec-inc/EnhanceIO. This allows you to add cache to any device on the fly without and pre-formatting. For the storage tiering inclined people, similar performance gains can be made with btier, http://www.lessfs.com/. Mark from lessfs also has a great deduplication project as well called...well... lessfs.

There's a FUSE-based file system called LessFS capable of performing block-level deduplication. The project is actively maintained and looks like worth a shot. For more information, check its webpage at http://www.lessfs.com

I was doing similar research a few days ago.

Some of these are already mentioned...

• Lessfs - v1 is stable, v2 is pre-alpha/alpha. http://www.lessfs.com/
• Blackhole - http://www.vanheusden.com/java/BlackHole/ - requires Java, which seems like a bad idea to me for a block level device, but I haven't tested it yet.
• SDFS from OpenDedup - http://code.google.com/p/opendedup/ - http://www.opendedup.org/ - looks very promising, but may have stalled
• Dedupfs for Ext3 - http://pages.cs.wisc.edu/~kosmatka/dedupfs/
• ZFS. You know about that.
• DragonFly/Hammer - http://www.dragonflybsd.org/hammer/ includes dedup. Competitor to ZFS and Btrfs, also using Btree. Includes block level dedup, but I'm not sure if it's fixed block or not. Suspect it is fixed.
• Btrfs - there's a patch. Not sure if it's in mainlined yet. But without fsck btfs is not trustworthy enough. That's coming soon, but has been for a while. In case you read this as being negative about btrfs, it's not; it's an awesome file system, combining modern ideas and an excellent implementation, but it's still at testing stage for critical data.

Other stuff:

• Dext2 - an idea. No code. http://code.google.com/p/binarywarriors/
• BackupPC, the next version may have block level dedup, it's been suggested/requested. Numerous people pointed out the hard linking scheme it uses. I'm backing up VM images, which is what started me on this block-level dedup search, and when you have a small change in a 60BG file, it's a new file. (Yes, I have thought of schemes to split them.)
• Bacula have been experimenting with block level dedup, fixed and sliding. May be in future versions.
• Bup - https://github.com/apenwarr/bup has many of the ideas. It's not a file system, but could be reconstructed, I think. Based on Git store. I recommend reading http://apenwarr.ca/log/ which has more, and is entertaining. I think this is an excellent approach. Read back in his blog for details on bup ideas.
• SquashFS - for static data.
• Epitome - http://www.peereboom.us/epitome/man/ - for static data too, I think. Not fully investigated.
• I know I saw at least one Google Summer of Code submission about dedup. Haven't followed it up yet, and couldn't find the tab in my browser.
• Interesting conversation - http://news.ycombinator.com/item?id=2932335

By fixed block I mean that the file system does not search out shared data when the blocks are not on block boundaries. So if you add one byte to the beginning of a 10 GB file, and that has the unfortunate consequence of rippling up through all the blocks that make the file, then there will be no block level sharing with the original file. Of course that's a pathological case, but you get the idea.

Original poster, perhaps you could keep us informed of your findings? There's at least me who is also interested.

Active developer and Open Source:
http://www.lessfs.com/wordpress/

That's more or less what LessFS does.
It does compression and also does De-duplication, so it is equal or many many times more efficient and space saving than simple compression.

LOL at the anti linux comments below. I see your transparent compression and raise you the deb package management with debian multi arch repos. *rakes in the chips* Nice playing with you pals.

Are there any open-source filesystems that offer deduplication?

I've been meaning to look at http://www.lessfs.com/wordpress/, a Linux FUSE based dedup system.

But ZFS is open source, you know. If you don't like the way OpenSolaris has gone, try it with FreeBSD.

There's a few. I've read there's a patchset for ZFS on FUSE that can do deduplication; there's also opendedup and lessfs. The problem is that none of these has been around long enough to be considered bulletproof yet, and for a filesystem whose job is to play fast and loose with file contents in the name of space savings, that's kinda worrisome.

Considering that no other open source file system offers this feature, it seems like a stupid criticism.

There are a couple in development although I wouldn't use in a production environment just yet:
1) SDFS - http://code.google.com/p/opendedup/
2) LessFS - http://www.lessfs.com/

I think they are both based on FUSE but I'm not entirely sure, I haven't delved that deep into them I was just poking around one random night.

The LessFS project also deserves mention: http://www.lessfs.com/ . Just think of the effect of combining a deduplication system with an iSCSI shared virtual tape library like http://sites.google.com/site/linuxvtl2/

Using something like lessfs ( http://www.lessfs.com/ ) but across the *entire* network?

Is it illegal to download a list of instructions on which chunks to use ( and in what order ) to create a copyrighted work from your family photos? :)

"putting aside some of the enterprise features that most of us don't need" - Oh you need them, you just very likely don't ever see them. I work for the largest storage vendor on the planet and your bank account lives on enterprise storage as do your medical records, insurance, etc.


I really look forward to seeing btrfs and other ideas like lessfs move forward and hope that people will be amazed at what they're actually doing because it is really, really hard to get this stuff right so that you get your data back. Deduplication introduces brain-hurting complexities and challenges.

Here you go.

I really don't see why ZFS deserves a front page article every time it assimilates another piece of old news. I guess it's a lesson to any software project looking for publicity - build everything into a monolithic ball of mud and win free slashvertisements.

Take a look to lessfs.
It's still experimental though.

http://www.lessfs.com/wordpress/

If they make good stuff and sell it cheap, do we not get cool new stuff for cheap? I fail to see the problem.

What happens when an economic contraction bottoms out is that the smart people who squirrelled away their cash in good times get to buy up neat stuff at fire sale prices. I think that's all that's happening here. It's a sign that we've turned the corner and the wise guys are buying up the stuff that's oversold.

Please don't read into the above that I approve of the purchase price for DataDomain. A proprietary implementation of lessfs is not worth two billion dollars. I could write that code myself and so could many of you. Whichever company gets it is going to gut it for the customer list and that's even more dumb because after you've killed their incumbent product, they don't want to buy from you. I can't wait 'till the a FOSS alternatives to that mature.