Slashdot Mirror

Basically it is a NAS that breaks your data into sub-block sized chunks at ingestion, checksums it and single instances the chunks. Most users get 20:1 average compression. Obviously different data types de-dupe at different efficiencies. Our VM images typically hit 40:1, SQL Server DBs about 30:1, medical images 5:1. DDOS now includes some archive specific features, which it sounds like you might find useful. It also has a very robust replication feature. Since it was originally designed for backup, it is optimized for data integrity. DD is dead simple to configure and operate and the support is great. http://www.datadomain.com/products/appliances.html

Windows Storage Server 2003 (yes, yes I know its from Microsoft) shipped with this feature (that is called Single Instance Storage)
http://blogs.technet.com/josebda/archive/2008/01/02/the-basics-of-single-instance-storage-sis-in-wss-2003-r2-and-wudss-2003.a

It's not even close to the same thing.

We investigated this a while back, and it is basically a dirty, filthy hack on top of vanilla NTFS.

First of all, it doesn't compare blocks or byte-ranges, but entire files only. If two files are 99% identical, then they are different, and SIS won't merge them.

Second, it uses a reparse point to merge the files, which has significant overhead, at least 4KB for each file, if I remember correctly. That is, SIS won't save you any disk space for small files, which is actually quite common on file servers. The overhead erases much of the benefit even for larger files, to the level that SIS will skip files smaller than 32KB by default.

Third, it operates in the background, after files have been written. This means that files have to be written out in their entirety, read back in, compared byte-for-byte to another file, and then erased later. This is incredibly inefficient. On large file servers, the disk was thrashed like crazy.

Lastly, we found that the Copy-on-Write mechanism immediately copied out the entire file if it was changed even slightly. For small files, this is not noticable, but for large files this can be a massive performance hog. A 4kb write can be potentially translated into a multi-GB copy!

Proper single-instancing systems use in-memory hash tables that are often partitioned using "file similarity" heuristics to prevent cache thrashing. Even more advanced systems can maintain single-instancing during replication and backups, reducing bandwidth requirements enormously. Take a look at the features of the Data Domain filers for an idea of what the current state of the art is.

I can put over 28PB in one of my racks.

I hate having anything on tape. I prefer capacity optimized disk with offsite replication. Tape sucks.

For about $1/GB you can have something really cool. Check out Data Domain boxes. Capacity optimized storage (which is cool in itself) which looks like a backup device. http://www.datadomain.com/ So, stream your backup to disk, clone DR to tape, but you can keep months of backups on line. Admittedly, the Data Domain boxes are not for archival purposes but they should be a lot faster and more reliable than tape for the typical duties of a backup system (1 file, maybe a directory structure). They also support WAN replication so you can mirror your backups offsite. It's not cheap, but it isn't stuffing tapes in boxes either. (no, I don't work for them. I just think they're pretty cool)

This is entirely possible and they are not the only ones doing it, for example http://www.datadomain.com/ has been doing it for a while. The big storage vendors do it to some extent as well.
The idea is based on "de-duplication" of data and is only really practical for backups (where most data from backup to backup is identical) or central repositories of data for a large organization that has multiple similar data sets, for example, many installations of Windows that are often similar.
From my experience x25 is a bold claim for general data. I've seen small scale tests that showed x30 compression over backup sets but those implementations had performance issues.
From the description in their white-paper, despite their claims, it appears they are performing some kind of hash by definition (e.g. mapping a space to a smaller space).

A company called DataDomain makes a very similar product that they claim averages 20:1 compression for backups. It's real, has been shipping for some time, and generally works as advertised. The trick to getting such good compression is in the kind of data you're storing. If you run three backups in a week, the amount of actual changed data each time will be very small. Of course, if you just try to use a DataDomain box or similar as general-purpose storage for your MP3s, you're going to get very limited benefit out of it.

Check out Data Domain for a similar product. There are other people doing this stuff.

IF you want long term archiving, still need tape.

BUT IF you want weekly, monthly, quarterly, yearly backups then a virtual tape library (VTL) is a better option. For most servers, the change in the dataset is small and gradual so a VTL stores one full compressed back + diffs for incremental/differential/full backups. Also, VTLs look for redundant data across servers; 10 similar linux servers will have the almost identical binaries.

I am currently looking at http://www.datadomain.com/ VTL to replace a 72 slot dual drive LTO 1st gen library.

A VTL costs a bit less than a regular tape library + all the tapes you need but the increased throughput and no more tape handling is what makes it worth it.

0.02cents.

To be fair, I have a medium term solution inthe pipe and there is budget for it. Rather than wait for the DR datacenter project to mature, we will pursue tape elimination and replicate the backup over the wire. Basically we are going to go with a content addressable disk backup target. Something like Data Domain. It still has no value from a DR perspective, but it eliminates the HIPAA exposure and restore latency. It alsogetsw us out of the tape management business (yay!). Basically we replace tape with CAS and replicate the CAS box to a second one in another site. The second site does not have to be a full data center, only meet minimal standards. That will get me by until the DR project comes to fruition. Right now we are reviewing possible target.

There's an appliance that does this (targetted for back-up storage). http://www.datadomain.com/

There have been several good answers for the front end. Here's a good backend architecture.

make sure you virtualize. VMWareESX and Vmotion are very cool. They have tons of info on their site for using virtualization to increase uptime and it's all true. I thought it was a load of BS until I started using it. It's great for DR and multiple sites.

flat backend filestore....
ok, I have nothing to do with these people. I have no stock in the company, I jsut think they have a cool product. http://datadomain.com/

Check out Data Domain if you're going to use flat filesystem for the filestore. They use bitpattern matching to provide pseudo single instance store, and (they claim) 20x compression (though with this technology and something like mail, you could probably approach 8x easially.

Their products do remote replication so you can have your multpile sites with the same mailstores.

also figure storage.... 1M users, figure average message size is 15K (in a single instance store system, no SIS, figure 75K). Figure everyone is going to have 1000 messages in their inbox.

so that's 15-75TB if you could limit mailbox size reasonably, you could probably get away with the DataDomain DD460 without too much hassle. put one at each site, set up asynchronous replication, buy 2 extras for backups in different locations and offset their synchro schedules. if you need a message delete more than a few weeks ago, tell whoever wants it to go ask the corporate lawyers why you shouldn't keep email on tapes. If they really want to back it up to tape, email me and I'll build you an architecture on paper.

if you don't use a DD product for your backend, look at pillardata.com you could build a 20TB system for about $6/GB and when you fill it up, expand it for about $3.50/GB (in 4TB chunks)

I do storage management for a living. I have about 160TB of accessible storage spinning right now. Backend I can do off the top of my head. Front end is best left to others, but the backend is always the same.

Tapes are also slow, expensive, hard to verify, labor intensive, and did I mention slow?

Now, when a hard drive fails, what're you going to do?

Clever solutions spread the risk around multiple hard drives and allow you to recapture some of the idle disk space on desktop system. When a tape drive fails are you even going to know about it? I have had numerous tape drives fail silently, and the only time you know that you weren't getting a good backup was then the time came to restore the data and you found your tapeset to be useless. Disk drives are cheap, and with RAID and distributed-RAID solutions you can use error correction techniques to eliminate the problems of losing multiple drives across the system -- and when any component fails you know about it and can act to eliminate the potential for cascading failures that might actually lead to data loss.

The biggest win from dumping tapes is that you can restore data quickly and eliminate the IT burden by turning most data restore operations into a user self-help situation. Let the user restore their own data and give them the additional advantage of being able to do file versioning and online backup verification. Offline media (like tapes) will continue to have a small role as a solution for catestrophic failures like the building burning down, but when it comes to day to day disaster recovery (and the all-too-frequent "pilot error" events) tape backup's time has come and gone.

Disk-based backup is also much more efficient than tape, allowing you to do all sorts of fun tricks like single-instance storage (e.g. only one copy of word.exe needs to be backed up, the rest of the systems can just point to this copy) and using disk-based backup systems as an intermediate cache (a la Data Domain boxes) is the smarter solution if you can't quite give up your tape habit just yet.

Tapes are cheap and high density.

Oh really? I can buy 1 TB of IDE disk for 1K and drop it into a cheap Linux box that was previously acting as a doorstop. How much does your tape drive cost? And the tapes? And the server to front the tape drives? And the additional networking hardware to provide a fat pipe to this data funnel you have just put into place? Tape is no longer even the cheap solution, it is just the one you are used to....