Slashdot Mirror

is http://www.avamar.com/

The Backup server or cluster of servers store 20KB blocks keyed to the block's SHA-1 hash.

Smart agents on each backup client chunks each new file to be backed up into 20KB blocks and calculates SHA-1 hashes which it compares against the backup server.

If the block is new (not on the backup server) the block itself is transfered.
If the block is old, the backup server stores an extra reference to the block for the client/file.

The end result is..
a) a 1000 windows backup clients will result in only one copy of a windows dll being saved
b) every full backup is like an differential in terms of size/speed.
c) you have weeks (if not months) of daily backups of all backup clients stored in 110% of total backup size.
d) the backup agent on the client has a larger footprint and requires more CPU while running.

Put the backup servers at a remote site with a high speed link and you have disaster recovery as well.
If the high speed link isn't an option, there is support for remote replication; requires another backup server (or cluster) of same size.

This is the way all backups will be done in the future.

What these guys are doing is not compression. It's commonality factoring. No piece of data is ever stored more than once. Typically, this is done by a hashing algorithm that starts at a high level and indexes everything down to a discrete block size of a few K.

Each block gets an index checksum, then each file, each subdirectory, each parent directory, and so forth until the entire disk volume has a cumulative hash. Then, it's very easy to determine a) what has changed (and where), and b) what has been seen before.

When a backup starts, the client compares the volume hash signature to that on the backup system. If it matches - nothing has changed. Backup over. If it doesn't, then you walk the indexes to find out exactly what has changed, and then only prepare to send those dirs, files, or discrete blocks - whatever's the smalles object that expresses the delta. When those objects are queued to send to the repository, the client first generates a hash of the object and asks the repository if it's seen it before. If not, it sends the index and the data. If so, it sends nothing, since the repository's already got that particular chunk stored somewhere. There's some re-hashing and index reverification on the other end to make sure that all is consistent.

Therefore, each backup appears a "full" backup, not a file-level diff, since the entire image is comprised of a map of every object in the volume. In reality, each backup is a set of pointers into a hashed data store (commonly called a CAS, or Content Addressed Store) from which is is reconsitituted as needed.

Having tested and deployed one of these types of systems, I can say that a) it's great for desktops, where most of the data between boxes is identical - the OS, the core apps, etc, and only the user data and localization is different, and b) it's awful for pre-compressed data like streaming audio, video, JPEGs, PDFs, etc. Since compressed data is entropic data, there can be no commonality within the file or versions of it, unless the file itself is identical and present from multiple sources. Change one byte of a file and recompress it, and all the blocks are unique.

However, this is not new. Giggle Avamar Technologies and Arsenal Digital. BTW - this tech is pretty good for remote backups over low-bandwidth links, since it vastly reduces the amount of data that needs to traverse the wire.

I came across a company called Avamar (http://www.avamar.com/). They do something similar, except they deduplicate the data on the client side, before it ever traverses the network. Needless to say, I was a bit skeptical with their claims. I was able to con them into letting me eval the platform for 3 months. As it turns out - it works as advertised.

I was able to consolidate 2 large tape libraries (L700's) into 8 x Dell 2850's - all running Redhat Enterprise, all with 6 x 300GB SCSI drives in them. We are currently protecting 7TB of data (Note: The hardware is currently 58% utilized). We process 20,000 backup jobs a month. And as an example, pulled directly from last night's activity log, we performed a full backup of a Windows box with 100GB of data on it in less than 15 minutes. (Note: The science behind this is very practical.

This is the naive foolishness that leads people into believing ridiculous compression claims, buying into them again and again. This sort of "just magically see the repeats" nonsense has been debunked a trillion times, so I'm not going to point it out.

I've actually seen either Diligent's or someone-else's similar presentation before, while looking for Virtual Tape Libraries. (maybe Avamar? http://www.avamar.com/products4.asp )
I was actually quite impressed with the product. I think they use huge hash index tables to do the block comparisons quickly. Avamar's site mentions they use 12k chunks. It was just damn expensive.

In any case, the 25x number I'm sure is an average they've found. Keep in mind this is for backups. So, they are saying if you backup 1000 PCs with Windows, 98% of the OS never changes from machine-to-machine, and you only have to backup those blocks once.

This company Avamar is an archive to disk specialist.

Disclosure: I am a shareholder and former employee. I haven't worked there for 2 years so this info is a little out of date... but they've been improving it susbstantially in that time frame so use this as a baseline.

They are not a disk to disk staging server company like EMC. They focus on data archival on disk... have a system called RAIN (Redundant Array of Inexpensive Nodes.. or Independent Nodes, both are correct) which uses N-Level parity like a RAID array but across server nodes, 4 disk 1 or 2U servers in a rack... which allows for swapping out new nodes on the fly w/o data loss.

Additionally they use an advanced Content Addressed Storage algorithm which does CAS at the byte level instead of the file level tagging your data for date/time/location/backup etc. as well as providing a base for a technique called Commonality Factoring which factors out common data across files before it's ever transferred (happens in the background on the client during scheduled analysis windows) and for many applications, reduces storage and transfer amounts by up to 90%.... meaning you can backup a lot more data to a 1TB system than with standard compression. 3-5 TB of real world data is typical for a 1TB system, which includes all the incrementals you can stand... hourly, daily, weekly, monthly, with checkpoints at any point in time you flag or schedule and a policy management system for deleting incrementals and saving checkpoints permanently.

Their client server architecture allows for remote backups easily and for doing remote mirroring for disaster recovery scenarios. Remote mirroring costs extra, a software plugin called Replicator.

Intel uses Axion (their software) for archiving chipset blueprints. Goldman Sachs uses Axion for storing archived financial and legal data. Visa is using Axion for storing digital versions of checks. Nasa uses Axion for storing signal data from satellites and radio antennaes.

Base system comes in around 40k but call for a demo and sales pitch... they'll bring out a CAT (Commonality Assessment Test) to show you how much of your own data they can archive. The first run will take a day to get a level 0 snapshot, then subsequent backups will take between an hour and 3 hours to do what a tape system will take 12 hours or more to do every single time, even with software like Veritas or EMCs Legato acquired stuff....

They support all standard OS's used in servers and their clients support Windows, Linux, Oracle, MSSQL, Exchange and Solaris. Also they support VTL for use as a backup to your CLARION system which means the server will look like a tape system and can be managed as such....

Finally they do encryption on server, before transmission if required for remote backup. Allow for online access to data 24/7 with full search capabilities and network mapping for mounting a backup directly on your administrative client that looks like a normal filesytem... oh yeah and a web based filebrowser for doing individual or directory sized restores if you just want to grab an archived copy of something small... rather than having to do a full restore w/ all the incremental restores, etc. just to get at one file

Well I'm a little biased having worked for a disk-as-backup company for over 4 years... so...

here's some links:

Scroll down about half way

This will give you a good overview of TCO for tape vs. Disk

Do your own research... you'll see the facts for yourself...

or go here to see the future of backup and restore:

Avamar Technologies

Tape works, but not as well (i.e. robustly), as fast, *or as cheaply* as an intelligently designed software based archival system that uses geographically distributed hard disk drive storage.

Hey! Yeah, you in the back about to submit the inital barrage in a pro-tape flame war, at least read the rest of my post before pulling the trigger. That might keep the discussion more productive. heh.

Designing the software and algorithms to run such a system is hard, but once you solve the basic problems the advantages are pretty compelling, and a lot of really juicy bits sort of just 'fall out'. I may, of course, be biased because I spent quite a few years building just such a system.

For more information see the company at
Avamar Technologies
(course you have to wade through marketroid speak to actually understand what they do)

A customer reference from someone who switched over
Qualcomm press release

And, of course, there are some patents granted and pending for this class of algorithms. (one of my specific interests)
Press release for patent #6,704,730

In the interest of full disclosure, as I said, I worked there from the beginning for over three years until it turned into less of an R&D project and more of a money making machine.

I am, however, free to discuss most of the algorithmic aspects of the system in some detail. So specific questions are welcome as long as they come with a certain degree of technical sophistication. ;-)

Tape works, but not as well (i.e. robustly), as fast, *or as cheaply* as an intelligently designed software based archival system that uses geographically distributed hard disk drive storage.

Hey! Yeah, you in the back about to submit the inital barrage in a pro-tape flame war, at least read the rest of my post before pulling the trigger. That might keep the discussion more productive. heh.

Designing the software and algorithms to run such a system is hard, but once you solve the basic problems the advantages are pretty compelling, and a lot of really juicy bits sort of just 'fall out'. I may, of course, be biased because I spent quite a few years building just such a system.

For more information see the company at
Avamar Technologies
(course you have to wade through marketroid speak to actually understand what they do)

A customer reference from someone who switched over
Qualcomm press release

And, of course, there are some patents granted and pending for this class of algorithms. (one of my specific interests)
Press release for patent #6,704,730

In the interest of full disclosure, as I said, I worked there from the beginning for over three years until it turned into less of an R&D project and more of a money making machine.

I am, however, free to discuss most of the algorithmic aspects of the system in some detail. So specific questions are welcome as long as they come with a certain degree of technical sophistication. ;-)

Tape works, but not as well (i.e. robustly), as fast, *or as cheaply* as an intelligently designed software based archival system that uses geographically distributed hard disk drive storage.

Hey! Yeah, you in the back about to submit the inital barrage in a pro-tape flame war, at least read the rest of my post before pulling the trigger. That might keep the discussion more productive. heh.

Designing the software and algorithms to run such a system is hard, but once you solve the basic problems the advantages are pretty compelling, and a lot of really juicy bits sort of just 'fall out'. I may, of course, be biased because I spent quite a few years building just such a system.

For more information see the company at
Avamar Technologies
(course you have to wade through marketroid speak to actually understand what they do)

A customer reference from someone who switched over
Qualcomm press release

And, of course, there are some patents granted and pending for this class of algorithms. (one of my specific interests)
Press release for patent #6,704,730

In the interest of full disclosure, as I said, I worked there from the beginning for over three years until it turned into less of an R&D project and more of a money making machine.

I am, however, free to discuss most of the algorithmic aspects of the system in some detail. So specific questions are welcome as long as they come with a certain degree of technical sophistication. ;-)

Thats Avamar