Slashdot Mirror
Data Deduplication Comparative Review
snydeq writes "InfoWorld's Keith Schultz provides an in-depth comparative review of four data deduplication appliances to vet how well the technology stacks up against the rising glut of information in today's datacenters. 'Data deduplication is the process of analyzing blocks or segments of data on a storage medium and finding duplicate patterns. By removing the duplicate patterns and replacing them with much smaller placeholders, overall storage needs can be greatly reduced. This becomes very important when IT has to plan for backup and disaster recovery needs or when simply determining online storage requirements for the coming year,' Schultz writes. 'If admins can increase storage usage 20, 40, or 60 percent by removing duplicate data, that allows current storage investments to go that much further.' Under review are dedupe boxes from FalconStor, NetApp, and SpectraLogic."
Same as the first.
Filesystems should be doing this.
Give me Classic Slashdot or give me death!
The shiny new NetApp appliance that my PHB decided to blow the last of our budget on saves around 30% by using de-dupe, however we could have had 3 times conventional storage for the same cost.
NetApp is neat and all but horribly overpriced.
For all intensive porpoises your a bunch of rediculous loosers
Odd that if they reviewed this class of products they didn't review the most common deduping NAS/SAN applicance - the EMC NS-series (particularly NS20).
I can't wait until the Dilbert strip hits where the PHB does this across all their backups and deduplicates them all away, thinking he's just saved a ton of money on backup media.
Redundancy can be a very good thing!
Oh, say does that Star-Spangled Banner entwine / The myrtle of Venus with Bacchus's vine?
Diffs are fine until you lose the root file upon which they are based. Then you lose everything you've never changed. You need to do periodic full backups.
Oh, say does that Star-Spangled Banner entwine / The myrtle of Venus with Bacchus's vine?
Filesystems should be doing this.
The one on your desktop machine, or the primary NAS storage that you access shared data from, or the backup server that ends up getting it all anyway? You see, this is a shared database problem. If your local filesystem does this, then it has to 'share' knowledge of all the unique blocklets with every other server/filesystem that wishes to share in this compressed file space. De-duplication is a means of compression that works across many filesystems - or at least it can be, if it is properly implemented.
"Every time I see an adult on a bicycle, I no longer despair for the future of the human race." - H. G. Wells
Are there any open-source filesystems that offer deduplication?
It seems that the FS du-jour changes faster than any of the promised 'optional' features ever materialize.
Instead of working full-bore on The Next Great FS, it would be really nice to have compression, encryption, deduplication, shadow copies, and idle optimization running in EXT4.
Maybe I'm just jaded, but I've been a Linux user for 12 years now. Sometimes it feels like the names of the technologies are changing, but nothing ever gets 'finished'. Maybe the NTFS/BSD model (good core design, long intervals with only minor changes) would be wise in Linux filesystem development.
"Sometimes, I think Trent just needs a cup of hot chocolate and a blankie." -Tori Amos on Nine Inch Nails
No need to give it a fancy name.
It's much easier for sales if you give it a fancy name, and preferably one that doesn't trigger comparisons with other solutions.
Of course, as deduplication is mainly a solution for enterprises that have been tricked into buying obscenely expensive storage, and who lack any coherent data storage policy and tiering strategy, the fancy name might be superfluous; they're spread wide and lubed up already.
However they have a ton of features including extremely high performance and reliability. For example they monitor your unit and if a drive fails, they'll send one you next day air. Sometimes the first you know of the failure is a disk shows up at your office.
Don't get me wrong, they aren't the only way to go, we have a much cheaper storage solution for less critical data, but the people who think dropping a bunch of disks in a Linux server gives you the same thing as a NetApp for less cost are fooling themselves.
It is exceedingly high end stuff, which is why it costs so much.
ZFS offers dedupe, and is even available in prepackaged NAS distributions such as Nexenta and OpenNAS. You too can have these great features, for much less than NetApp and friends.
Didn't Plan 9's filesystem combine journaling and block-level de-duplication years ago?
Proud member of the Weirdo-American community.
However, the article seems centered on primary storage, and not the marriage of backup/replication/physical tape, which is Quantum's focus.
Personally, I'd be _terrified_ of using dedup for primary storage. What this does is exactly the opposite of RAID - it squeezes every last bit of redundancy out of your data, and makes everything dependent upon the integrity of your blockpool database. Loose a single blocklet and you stand to lose _all_ of your data.
Compressing common data across many filesystems for things like backups makes a lot more sense, and seems more cost effective.
"Every time I see an adult on a bicycle, I no longer despair for the future of the human race." - H. G. Wells
No, it's not.
Differential backups are taking a single filesystem, seeing what changed (either at the file level (whole changed/updated/new files) or block level (changed blocks within files).
Block level deduplication is noticing that the storage appliance on which you back up 100 desktops and 10 servers has 50 copies of the same version of each data block in each Microsoft OS file from XP, 25 from Win 7, and 35 from Fedora, and only storing 1 copy of each of those blocks rather than 100 separate ones. It's returning those blocks to the usable storage pool and remapping without having to "compress" anything, not having to rewrite the backup data images, etc. It's just saying "This is block 3 of the binary for Internet Explorer 8, and I already have a copy of that", for each and every common block out there.
You still have to upload the blocks, and the system still needs to scan them to notice the duplication, but it's a lot more than "oh, compression".
http://www.opendedup.org/
AFAIK this is pretty much how every compression algorithm works. No need to give it a fancy name.
The reason it has a different name is to distinguish this from a compressed file system. The blocks of data are not compressed in these systems. Imagine that you have a file system that stores lots of vmware images. In this system, there are lots of files that store the same information because the underlying data is OS system files and applications. Even if you compress each image, you will still have lots of blocks that have duplicate values.
Deduplication says that the file system recognizes and eliminates duplicate blocks across the entire file system. If a given block has redundant data within it, that redundancy is not removed because the blocks themselves are not actually compressed. This is the difference between a compressed file system and a deduplicated file system. In fact, there is no reason that you could not combine both of these methods into a single system.
Although there is nothing to say compression of data might not also happen. I don't believe compression and de-duplication are mutually exclusive.
This is actually a good argument for de-duplication to run on the device. It can surf thru files more or less at leisure looking for duplicate blocks all over the file system, without tying up the server's bus/controller.
That could be done independent of File System compression, which generally, as you pointed out, works best on large blocks of repetitive bytes within a single file.
Sig Battery depleted. Reverting to safe mode.
Except NexentaStor (3.0.3) has an OpenSolaris upstream (which has gone away, by the way) kernel bug that hanged our Nexenta test box. Not a real good first impression.
they dont have to do this at the file level.
they do it at the block level.
so in your example, since the only change would be the signature on the bottom of each email, the email blocks themselves would be deduped, and the signatures would be retained.
think of backing up a whole bunch of similar desktops in an enterprise situation where the majority of the OS files are going to be the same or have only slight variations.
even if the files have slight variations, only the actual bits that are different would be stored and the rest would be deduped and only one copy kept.
personally i know a fairly large company using avamar for this, however they do it on their backups only. And iirc, they still keep different sets of backups, just dedupe within the backup itself which saves them quite a bit of space per backup.
I think what you're talking about is single instance storage in your mail server. But as you mentioned, it only works well on identical emails and attachments.
No dedupe system that I'm aware of does what you'd need to do to dedupe forwarded emails. It's technically possible by recognizing similar messages and doing diff's on them to find identical sections. But, it's computationally difficult and there's not much payback -- better to go after the lowhanging fruit and dedupe all of the identical gif's and mp3s that people have downloaded off the internet.
When we deduped our corporate fileserver, we got around 40% of our space back.
Something you start to appreciate when you are called on to do a really high availability, high reliability system is to have features like this. For one thing it reduces the time it takes to get a replacement. Unless a drive fails late at night, you get one the next day. You don't have to rely on someone to notice the alert, place the order, etc. It just happens. Also, like most high end support companies, their shipping time is fairly late so even late in the day it is next day service. What arrives is the drive you need, in its caddy, ready to go.
Then there's just the fact of having someone else help monitor things. It's easy to say "Oh ya I'll watch everything important and deal with it right away," but harder to do it. I've known more than a few people who are not nearly as good at monitoring their critical system as they ought to be. A backup is not a bad thing.
You have to remember that the kind of stuff you are talking about for things like NetApps is when no downtime is ok, when no data loss is ok. You can't say "Ya a disk died and before we got a new on in another died so sorry, stuff is gone."
Not saying that your situation needs it, but there are those that do. They offer other features along those lines like redundant units, so if one fails the other continues no problem.
Basically they are for when data (and performance) is very important and you are willing to spend money for that. You put aside the tech-tough guy attitude of "I can manage it all myself," and accept that the data is that important.
Yeah! To fight dupes I compute CRC checksum for each file and store it (and only it) on my back up drive. That method removes dupes almost automatically and there is a side effect of a huge compression ratio too. I have been downloading the high def videos from Internet for quite a while now and with my compression method I have used less than 10 percent of 1GB flash drive! I strongly recommend this method to everyone!
In theory there is no difference between theory and practice. In practice there is. - Yogi Berra
Did it cost less than buying 40% more disks? Heck, did it cost less than building another fileserver with 100% more disk and then syncing between them?
After an analysis of a 1TB drive, I noticed that roughly 95% were 0's with only 5% being 1's.
I was then able to compress this dramatically. I just record that there are 950M 0's and 50M 1's. The space taken up drops to around 37 bits. Throw in a few checksum bits, and I am still under eight bytes.
I am not sure what is so hard about this disaster recovery planning. Heck, I figure I am up for a promotion after I implement this.
See my journal for slashdot ID's by year. Mine created in 2005. http://slashdot.org/journal/289875/slashdot-ids-by-year
I found a ton of stuff I didn't really care for with Nexenta. They've put some good effort into it, and it'd be a fine way to go if you wanted commercial support, but overall it doesn't really seem to fit our needs here. ZFS itself is a resource pig, but on the other hand, resources have become relatively cheap. It's not unthinkable to jam gigs of RAM in a storage server ... today. Five years ago, though, that would have been much more likely to be a deal-breaker.
That's still a particular type of compression, isn't it? I mean, I can buy giving it a new name, since it has a bunch of infrastructure around it, but it's a perfectly general kind of data-compression algorithm as well, even if not the world's most efficient: break the data into fixed-size blocks, then, for any blocks that appear more than once, replace all occurrences after the first with a pointer to the first. Block-based RLE compression is basically a simpler version of that (where you only deduplicate the blocks when they appear in sequence).
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
aside from the mentioned 'to reduce duplicate data to increase available storage space' are there any other benefits to de-duplicating your storage?
An intelligent caching layer will only store the deduped data once, allowing it to cache more data, get more cache its, reduce physical disk IO and improving response times.
I just bookmark the movies on movie2k and huku and play them when I want to watch them. That saves even more space.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
If the market leader isn't included in the review, I am wondering how worthy this report is.
Of course, as deduplication is mainly a solution for enterprises that have been tricked into buying obscenely expensive storage, [...]
If we were "tricked", what is the cheaper, but equally capable alternative ?
That's still a particular type of compression, isn't it?
Not really. Compression is taking a chunk of data and replacing it with a different, smaller chunk of data plus instructions (albeit in abbreviated form) about how to turn it back into the original chunk of data. Dedupe is taking a chunk of data and replacing it with a pointer to a "remote", identical chunk of data.
Compression is nearly always applied on a per-file basis, whereas dedupe is applied on a per-volume basis. Conceptually similar to the difference between compressing every individual file in a volume vs tar.gz-ing the whole volume.
Compression is generally much more limited over the "window" of data it can work with at any time (eg: if you have two similar bits of data at the start and finish of a file, but they're separated from each other by 10 gigabytes of other data, then they'll probably be compressed independently).
Consider also the difference in processing necessary to decompress data vs dereference a pointer on disk. Some compression schemes (eg: h.264) are quite CPU intensive to decompress, while the overhead for reading deduped data is basically zero.
Since the dedupe license came for "free" with my filer, yes, that 40% improvement cost less than buying 40% more disks.
And yes, it's much cheaper than building another fileserver with 100% more disk and syncing between them. How much do you think it costs to build a fileserver with 150TB of disk space, and how would you recommend that I sync the 75TB of data between them? I don't think this is a job for rsync.
I do actually replicate between two identical (nearly identical) arrays, but I use my array vendor's software to do this -- they can replicate only modified blocks, much faster that a tool that only has visibility at the filesystem level.
EMC's software is the most buggy, unintuitive, poorly documented and abysmally supported piece of shit I've ever had the displeasure to use. It is simply revolting. Considering how much it costs, it's mind boggling.
(Well, what could I expect from an appliance that runs on Windows 95?)
I have to use that shit and it's obviously designed by complete morons. Seriously, I have to work with a spanking new SAN worth hundreds of thousands and it's so full of bug I can't imagine why people buy that crap. Oh wait I do know but I can't tell.
Yup, I totally agree with you. There is a *lot* of stupidity on display today.
Tell you what, why don't you image a 100GB disk into a single file, then run zip over it and come back and tell us how it worked out for you when you're done...
Here is a hint: the buffer is a constant-size because it has been tuned for a particular type of file where the granularity of repetition is very fine/small. Trying to do the same on much larger files where the granularity of repetition is much coarser requires a much bigger buffer. And it does not increase linearly...
Slashdot: where don knuth is an idiot because he cant grasp the awesome power of php
Well, if that 100GB disk is full of the usual mix of programs and data it'll deflate to about 50% of its size. Maybe even better; whole hard drives are a special case with lots of "slack" and unallocated space.
Software implementations of compression have some issues - because of the way the compressed data is stored it's not tolerant of errors and a problem in the driver that implements the compression can render the entire disk unreadable. Doing it in hardware can improve the reliability, but it's not an invention. I suppose that next we'll hear about how a patent has been issued and when that gets discussed the trolls will come out for another party.
Product announcements like this one wouldn't happen and discussions like this would be much shorter if the people involved knew what they were talking about. But I suppose that's too much to expect these days...
That's still a particular type of compression, isn't it?
Yes, it's a fairly simple kind of compression too. Basically, your FS is a mapping from names to sets of numbers, then from numbers to blocks of data. With deduplication, you are just tweaking the second map so that identical blocks are only stored once. This is compression - you are storing data in less space - but it is a very specific kind of compression, which is why it gets the name.
Compression is a very generic term, but in the context of filesystems usually refers to per-file or per-block compression. With something like ZFS, you can combine the two, so individual blocks are LZJB-compressed and duplicated blocks are not stored twice.
Pretty much any compression algorithm works on a window. With something stream-based, like gzip, it builds the compression tables as it scans the file, discarding data when they get too big. With something like bzip2, it scans a fixed size (up to 900KB, as I recall), calculates redundancy within that, and then stores the result. If you had 900KB of random data, repeated over and over in a file, these algorithms would do quite badly.
The reason for deduplication is that the coarse granularity can catch compression opportunities that conventional compression algorithms miss. It would not be feasible, for example, to build a Huffman tree for the contents of a 250GB hard disk. If you did, you'd almost certainly get a lot of compression, but you'd need a huge amount of RAM, and you'd need to rebuild the tree periodically when you modified the FS. In contrast, deduplication can run quite quickly. ZFS computes checksums for every block anyway - it just needs to store a table of these hashes to find duplicates. This takes quite a bit of RAM, but not an unfeasibly huge amount, and the CPU load is relatively small (just one hash table lookup per store - the bottleneck is almost certainly the disk, rather than the CPU).
I am TheRaven on Soylent News
Well done, you've missed the point. That 32KB buffer is not so useful if you have the same 32KB of data repeated - the deflate algorithm will miss it entirely because there's little redundancy within the 32KB blocks, but a lot of redundancy among them. Now what happens when you scale up the window to the 250GB of a cheap hard disk? First, your RAM usage goes to several TB. Second, your CPU usage spikes so high that it takes a few days to write a single 512-byte block. Not so great.
The point of deduplication is to find identical blocks on the disk and replace them with copy-on-write references to the same block. This is very cheap, in terms of CPU time, and relatively cheap in terms of RAM usage.
This is not a substitute for block-level compression, it's an orthogonal approach. A filesystem with dedup can also use block-level compression. ZFS, for example, uses LZJB for compressing individual blocks, then uses deduplication to avoid storing redundant blocks (it also implements copy-on-write for snapshots and clones, so if you copy a filesystem you don't actually allocate any space until you modify either version).
I am TheRaven on Soylent News
Which "usual" mix of programs and data is that then?
Or are you taking your experience on zipping smaller things that they "usually" decrease in size by about 50%. Because I've already explained to you why that experience doesn't translate very well into spotting duplicate blocks at much larger sizes...
The test is not exactly hard: take a file that doesn't compress very well under zip (i.e almost any media file) and build a fake file hierarchy out of lots of copies of it. Then make a disk image and try and zip it. What you've described will fail, for the reasons that I've described to you. But de-duplication would definitely work on that disk image.
Slashdot: where don knuth is an idiot because he cant grasp the awesome power of php
In realm of big IT where I have about 13PB of backup data on deduplicated disk we didn't even look at these products. Data Domain's DD690 and 880 are overall excellent but can't compress Oracle data if their lives depended on it. At least not the ways that the DBAs like to back up Oracle. IBM's Diligent product is a fantastic piece of technology for both Open and Mainframe systems, but is VTL based and does not come cheap.
Optimized replication between sites is one of the best parts of dedupe, even over storage. If something can actually get 10x compression then that 1GbE link I have between locations functionally acts more like a 10GbE for no more cost. A huge boost on the WAN for DR.
I know this is functionally different, but I have a Windows Home Server backing up my PCs at home. WHS apparently uses a "Single Instance Store" model to store backup sets. If the same file is detected on multiple computers, it is stored only once saving storage space. I'm backing up the C: drive of three Windows 7 Home Premium PCs to the WHS. Each PC uses between 40 and 60GB of space, yet the backup sets on the WHS total only 80GB. I'm sure that could partly be attributed to compression, but still, this seems to be pretty cool.
My mom always said, "Jim, you're 1 in a million." Given the current population, there are 7000 of me. God help us all!
I wrote a simple python program that does file based disk de-duplication. It will work anywhere python runs as long as the filesystem supports hard links.
It is available under gpl at: http://jdeifik.com/
With the hardware solutions in the article, prices start at $25k or so, and you are beholden to the hardware vendor. Doing it in software will work with any reasonable OS, any reasonable filesystem, and any hardware. Sure, block based de-duplication is more efficient, but it is filesystem specific, and right now ZFS is the only somewhat reasonable filesystem that supports it.
It's the decompression time that will kill you (and cause the heat death of the universe).
coffee | nose > keyboard
The ZFS-FUSE setup is fantastic. For most things you are very much limited by platter speed; I've found the performance to be quite acceptable.
As far as stability goes, the 0.6.9 release, which has been out for around 3 or 4 months, has been exceptional. I did extensive stress testing of it over the last 9 months or so, and all the issues I found were resolved (quickly) by the ZFS-FUSE folks.
I currently have a 16TB backup system running with something around 2,000 snapshots and 80% space used, and it works just great. I also have a personal storage system that I have been running ZFS-FUSE on for around 3 years now, and it also has been great. I was originally running 0.5.0 on it but upgraded to the 0.6.9 after my above stress testing. It also has been great.
I used Nexenta in the past, it was ok but I think there were definite ZFS issues in it at that time, maybe 3 years ago. The systems I had would reboot every 30 to 60 days. Then I upgraded it to the latest Nexenta a year or 18 months later and had all sorts of data loss issues while trying to do the zfs send/recv from the old systems. An annoyance with Nexenta was the hardware support; coming from Linux which supports just about anything to having to dig around to find compatible storage controllers... I recently (maybe 10 months ago now, before I started really hard down the ZFS-FUSE testing path) tried OpenSolaris and ran into some weirdness where I did the install, then did an update and it spent an hour downloading updates, then bombed out. So I started it again and it spent an hour downloading updates and bombed out.
There are two reasons I'm using ZFS-FUSE so heavily: 3 years ago there was no option for encrypting my ZFS storage system in Solaris, and I just am so much more comfortable with Linux than Solaris. My home storage system stores a lot of private data, that I want to have close at hand at home, but if someone steals it I don't want to worry about the scanned check images and bills we have saved there, etc... So crypto was a huge deal for me in that server.
Sean