Ask Slashdot: Simple Way To Backup 24TB of Data Onto USB HDDs ?

An anonymous reader writes "Hi there ! I'm looking for a simple solution to backup a big data set consisting of files between 3MB and 20GB, for a total of 24TB, onto multiple hard drives (usb, firewire, whatever) I am aware of many backup tools which split the backup onto multiple DVDs with the infamous 'insert disc N and press continue', but I haven't come across one that can do it with external hard drives (insert next USB device...). OS not relevant, but Linux (console) or MacOS (GUI) preferred... Did I miss something or is there no such thing already done, and am I doomed to code it myself ?"

Comments

USB and disk Speed

[gagol]

May be your limiting factor here.

Re:USB and disk Speed

[gagol]

If you can achieve a sustained write speed of 50 megabytes per second, you are in for 140 hours of data transfer. I hope it is not a daily backup!

Re:USB and disk Speed

[drsmithy]

If you can achieve a sustained write speed of 50 megabytes per second, you are in for 140 hours of data transfer. I hope it is not a daily backup!

I'd be willing to bet his change rate isn't 24TB/day.

Re:USB and disk Speed

[jamesh]

If the OP's porn collection can be logically broken up at some level, eg:

/porn/blonde
/porn/brunette
/porn/redhead

then the backup software could create one job for each directory, and multiple USB disks could be attached at once giving increased throughput. USB3 also increases speed to the point where the 7200RPM disk itself will become the bottleneck.

So at 100MB/second per disk write speed with 4 disks going at once (assuming the source disks are capable of this supplying this volume of data and there are no other throughput limitations), you could do it in 16 hours, or 24 hours with more realistic margins.

If it turns out that the source data is not porn (unlikely) and is highly compressible, then it could be done in far less time.

Bacula can do all of this.

Re:USB and disk Speed

[Anonymous Coward]

Agreed. Best thing I ever did was get a computer case with a SATA sled bay, like one of these. It won't help with breaking up the files, but a plain SATA connection will be many times faster and many times cheaper than getting external USB drives (because you don't have to keep paying for external case + power supply). After you copy it over, you just store the bare drives in a nice safe place.

This assumes it's a one-time or rare thing. If you do want access or the backup process is a regular thing, then an NAS or RAID setup is probably more convenient so that you don't have to keep swapping drives in and out.

Re:USB and disk Speed

[Anonymous Coward]

Or, he could watch the content as it is copied. At 600 Mbytes/hour (assuming standard mpeg compression), it would be a month of 24/7 nonstop action!

"- Hey boss, I need to, uhh, work from home for the next four weeks to handle the backup..."

Re:USB and disk Speed

[Pieroxy]

then the backup software could create one job for each directory,

Is that what we call a blow job?

Re:USB and disk Speed

[shokk]

If he's looking for reliability in a backup, then his choice of disks is going to be a factor. A drive with consumer grade chances of URE is going to die in a handful of writes and reads. USB grade drives (Caviar Green anyone?) aren't known for their reliability. Something like a Hitachi Ultrastar RE has a very very low chance of encountering a URE, so will be much more reliable.

Re:USB and disk Speed

[Anonymous Coward]

Is that what we call a blow job?

We're talking about multiple "directories" (know-what-I-mean? notch-notch. say-no-more), so I think we would call it blow bang, DP, gang bang or orgy.

Now, imagine a beowulf cluster fuck of those... :-)

Re:USB and disk Speed

[ilikejam]

No. No it is not.

Re:USB and disk Speed

[Anonymous Coward]

It's "nudge-nudge", not "notch-notch".

Also, you left out "wink-wink".

Yes, I know, I should get a life..

Re:USB and disk Speed

[Anonymous Coward]

Bacula can do all of this

So he quantum leaps into you, and isn't allowed to leave until he performs the backup? Oh wait! Bacula, not Bakula.

Re:USB and disk Speed

[v1]

I have a setup here where the server's video media is about 8tb in size. That backs up via rsync to the backup server which is in another room over rsync. It contains a large number of internal and external drives. None of them are over 2tb in capacity. The main drive has data separated into subfolders and the rsync jobs back up specific folders to specific drives.

A few times I've had to do some rearranging of data on the main and backup drives when a volume filled up. So it helps to plan ahead to save time down the road. But it works well for me here.

The only thing with rsync you need to worry about is users moving large trees or renaming root folders in large trees. This tends to cause rsync to want to delete a few TB of data and then turn around and copy it all over again on the backup drive. It doesn't follow files and folders by inode, it just goes by exact location and name.

I help mitigate this by hiding the root folders from the users. The share points are a couple levels deeper so they can't cause TOO big of a problem if someone decides to "tidy up". If they REALLY need something at a lower level moved or renamed, I do it myself, on both the source and the backup drives at the same time.

Another alternative is to get something like a Drobo where you can have a fairly inexpensive large pool of backup storage space that can match your primary storage. This prevents the problem of smaller backup volumes filling up and requiring data shuffling, but does nothing for the issue of users mucking with the lower levels of the tree.

Re:USB and disk Speed

[deniable]

Send error messages to a Blackberry and it's a RIM job.

Re:USB and disk Speed

[deniable]

Bacula went on to be Enterprise grade software.

Re:USB and disk Speed

[postbigbang]

SATA is good, and a set of SATA cages might be the cheapest solution.

More expensive is getting SAS drives (if the interface is there) and it'll be quicker.

More expensive, but not TOO expensive (these days) is a set of SATA cages and some SSDs, which will do the job far faster still (than conventional SATA drives).

USB3, if supported, with SSDs, would do the job nicely, too.

The software? Good old tar can be used, self-managed. Tar never seems to go out of style, and using it in conjunction with drive-spanning is a well-known exercise.

Re:USB and disk Speed

[CastrTroy]

You can even get add-on cages for you existing case like this one. Big advantage is that you can fit 3 drives in 2, 5.25" slots, which very rarely have much use anymore, yet most cases seem to still have 3 or 4 of them. You can get other models that will fit fit 5 drives in 3 bays, or have 1 in 1 depending on your needs.

Re:USB and disk Speed

[MMC Monster]

Maybe he's personally backing up CERN?

Re:USB and disk Speed

[milgr]

The LHC generates a petabyte per second.

Re:USB and disk Speed

[jedidiah]

Actually, they are roughly the same price.

Although SATA is more widespread and avoids any reduction in performance you might get from putting an intermediate layer in front of the native interface of the drive. A large drive is going to require a wall wart and all of those will need to be looked after.

The problem with case+power supply is not the cost but the fact that it is something else to lose. This goes for the extra cabling too.

Plus with a bare drive you can buy with performance in mind since the drive will likely be your bottleneck.

Re:USB and disk Speed

[Laebshade]

I had to re-read this several times before getting it. Bravo.

Re:USB and disk Speed

[GameboyRMH]

And better yet you can get them as 5 1/4 bay accessories so you don't have to change your case or end up with obsolete tech permanently stuck in your case 5 years from now! (Since these things have maximum capacities they can support).

Re:USB and disk Speed

[jamstar7]

Total failure on many levels? I see your point...

Re:USB and disk Speed

[sglewis100]

More expensive, but not TOO expensive (these days) is a set of SATA cages and some SSDs, which will do the job far faster still (than conventional SATA drives).

24tb worth of SSDs? That's a lot of cages, and yes, it would be TOO expensive.

Re:USB and disk Speed

[jemenake]

You can even get add-on cages for you existing case...

I was going to suggest the same kinda thing. Easier than sliding them in and out of the main case and futzing with power and SATA cables. In fact, the ones you suggested are the very cases I use in my 2U rack servers that I build. I like 'em for what I use them for, but I'm going to make a different recommendation for the OP. The problem is that that rack requires that the drives be in special sleds/trays. If the OP is going to be swapping drives, that means swapping sleds.

The new love of my SATA rack world are these StarTech trayless babies. The doors open with a little pull-latch... like most car-doors, and there's no tray. Just slide the drive in and close the door.

Re:USB and disk Speed

[voltorb]

But only 1GB/s is recorded: http://www.itnews.com.au/News/310769,computing-for-the-large-hadron-collider.aspx

Re:USB and disk Speed

[postbigbang]

Not as much as even a year ago.... not quite half as expensive. SSDs are plentifully fast.

SAS is faster than SATA is faster than USB, generally speaking. the SSD SATAII drives are becoming almost commodity. The cage is the same cost, as it's form factor and power supply.

Re:USB and disk Speed

[Pieroxy]

you're doing it wrong

Maybe I'm holding it wrong?

Re:USB and disk Speed

[robi2106]

I have two SATA over USB docks for bare SATA harddrives. I have ~7TB of unique data I keep (video editing business generates a lot of data).

I copy data using BeyondCompare from the edit RAID array to each twin pair of SATA drives. That gives me some redundancy for the data, but is a PITA to track.

I also would love to have a software solution that lets me just number the drives, tell the system to keep two copies of everything, and then would just instruct me which drive # to pop in the dock in order to back up file X.

Re:USB and disk Speed

[oldmac31310]

A nod's as good as a wink to a blind bat.

Re:USB and disk Speed

[Immerman]

I would be hesitant to use SSDs for long-term backup though. Hard drives, like tape, are a time-tested technology, in fact they're basically the same technology in a different form factor, though hard drives add the additional point of failure in their mechanical and electrical components.

SSDs on the other hand are still a relatively new technology, especially at the feature sizes being used in modern drives. You don't have the mechanical failures to worry about - but your storage medium is essentially billions of tiny capacitors, and the one thing you can absolutely guarantee about a capacitor is that it won't hold it's charge indefinitely. And as drive capacities increase feature size decreases, and with it your charge-per-bit ratio, which can only bode ill for long-term storage.

The more I look at SSDs, the more convinced I am that while SSDs may take over many niches (mobile storage especially) magnetic storage isn't going away any time soon. On the other hand as the controller technology develops the potential of using SSDs as cache for slower storage is becoming quite dramatic, and once it reaches a mature level that may open the way for other reliable high-capacity storage technologies to start entering the field "behind the cache", where poor performance of early generations wouldn't be as big a drawback

Re:USB and disk Speed

[thereitis]

If you're just using RAID to make a bunch of disks look like a single logical unit, consider mhddfs. It's a FUSE filesystem which makes a bunch of disks look like a single unit. I've used it for storing backups - it works as advertised.

IIRC there were one or two caveats like a lack of hard link support so make sure you try all your use cases before relying on it.

Re:USB and disk Speed

[jcoy42]

Is that what we call a blow job?

Yes, that would probably be an effective way to blow your job.

Re:USB and disk Speed

[hairyfeet]

USB would be just the most retarded way to go for something like this, its too slow and he's gonna be swapping worse than when we used to have to back up things to CDs.

I'm guessing he's going USB because he don't have the cash to buy a NAS of that size but you can always jury rig you a NAS, its really not hard. We did something similar at the last shop I worked at when the boss scored a ton of SCSI drives at an auction and ended up with nearly a Tb NAS when the average HDD was 40Gb. Here is how you do it..

You take a couple of full size towers, bigger the better, preferably twinkies as it makes the job a LOT easier. You strip 'em to the frames and use a couple of spot welds to make them into one giant case along with another couple of weld to mount a shitload of drive cages into the case. Then you take a cheap server or even desktop board, all that matters is it has a shitload of PCI slots which you fill with controller cards, SCSI in our case but SATA today, mount the board along with a big PSU to feed the drives and voila! One big ass DIY NAS unit that can hold a huge pile of drives. Just to finish our white trash conversion we tied on a Walmart box fan to keep the sucker cool and stuck it in a corner, worked great.

The only software that I think would work with USB is Paragon Drive Backup as you can have it split by just about any size you want. They also have their own Linux based recovery media but damned if i know if you can get the software as a Linux installer, never ran into that situation to need it in that way. I know its worked great for me making OS images and backing up files and folders onto USB drives but if you're gonna be splitting to a ton of little drives then you are just gonna have to swap, no way out of that. If you want to fill the drives up then set Paragon to a small size, say 700Mb, but good fucking luck checking your backup as the amount of swapping you're gonna do is just insane.

Re:USB and disk Speed

[deniable]

Don't worry T'Pol, it's a human thing.

Re:USB and disk Speed

[screwdriver]

Unless you're using USB 3.0, which I use for backups. It's just as fast as if the HD were internal.

Re:USB and disk Speed

[HArchH]

24x7?

Re:USB and disk Speed

[atamido]

If he's looking for reliability in a backup, then his choice of disks is going to be a factor. A drive with consumer grade chances of URE is going to die in a handful of writes and reads. USB grade drives (Caviar Green anyone?) aren't known for their reliability. Something like a Hitachi Ultrastar RE has a very very low chance of encountering a URE, so will be much more reliable.

Citation required. Information I've read from the manufacturers themselves indicates a similar error rate and MTBF between various drive lines. The difference between consumer and enterprise grade drives tends to be in the firmware where the enterprise drive will give up faster when encountering a read error so as to not risk dropping the drive out of the RAID array.

solution

[Anonymous Coward]

1.take all hard drives out of USB enclosures
2.install in PC with multiple SATA cards
3.samba

Re:solution

[aglider]

3.samba

Uh? Why?
cp -a is all you need once you put the HDD inside the target machine.
And if you put it into another machine on the same network, then rsync is the answer.
Forget about the buggy and slow SAMBA.

Re:solution

[Pieroxy]

Agreed. Samba should be at the very bottom of the list. It is the best solution only when there's no other solution.

Re:solution

[myowntrueself]

3.samba

Uh? Why?
cp -a is all you need once you put the HDD inside the target machine.
And if you put it into another machine on the same network, then rsync is the answer.
Forget about the buggy and slow SAMBA.

cp copies file by file.

A more efficient way is something like

tar -cf - .|(cd /somewhere/ ; tar xf -)

tar treats the directory contents as a data stream. Its much faster for large amounts of files and data.

Re:solution

[AvitarX]

Is it that much faster for 3mb to 20 gb files?

Re:solution

[fnj]

No. It's slower. Informative, my ass.

Re:solution

[Anonymous Coward]

This simple tar pipe has saved my ass too many times to count!

By the way, simplified version:
tar -cC [sourceDir] . | tar -xC [destinationDir]

Re:solution

[bastafidli]

cp doesn't preserver exact timestamp. If you want to do rsync later, it will copy all files all over. Jusd do

rsync --dry-run --archive --stats --progress --whole-file --exclude "/lost+found" --delete-after /source/ /destination

which is reproducible and later on will copy only the newer files.

Re:solution

[GameboyRMH]

cp has options to preserve timestamps and everything, RTFM.

Re:solution

[Cormacus]

Well, it might not be faster for the example in question, but that's a neat idea if someone reading this has a slightly different need in mind - ie backing up many small files.

Re:solution

[bastafidli]

didnt know that, will have to take a look at it, thanks

Re:solution

[Culture20]

But it preserves more filesystem data than cp. And running a tar stream like this over ssh is easy. Of course I'd use rsync if I wanted to do it more than once.

Re:solution

[fnj]

Not any more than cp -rp

Examine the command. He's packing the files into a tar format stream, then unpacking it back into files on the other end. He's doing the same number of source opens and destinations creates, but he's adding a step.

Re:solution

[fnj]

Why?

Examine the command. He's packing the files into a tar format stream, then unpacking it back into files on the other end. He's doing the same number of source opens and destinations creates; moving the same number of bytes; but he's adding a step.

Re:solution

[Cormacus]

Ah, I had not analyzed the command. I assumed that since it was using tar that one archive file was being created, and therefore your derision of that example was due to file overhead to file size ratio related.

Re:solution

[berashith]

this one looks like GNU tar, where the original deals with the crappiness that can be the native tar in solaris. I tend to use the original posters version out of habit as it moves to the dir and unpacks here (.) . Also, it can be fun to ssh stuff to different systems this way.

Re:solution

[Richy_T]

Yes. The above tar command is really from a time when cp did not have r and p options (and still likely doesn't on some systems so it's worth knowing). OTOH, you can add in the z option (compress) if you're doing something networky (though you'll probably want to throw in netcat or ssh too in that case). Of course, if you're doing that, rsync is probably the better option if available and leads to some interesting backup options going forward.

Re:solution

[flargleblarg]

Don't use cp.
cp doesn't preserve hard links and symlinks. When cp copies a symlink, it follows the link and copies that file, rather than copying the symlink.

cp is a very bad idea for archiving.

rsync or tar are what you want.

Re:solution

[Score Whore]

tar treats the directory contents as a data stream. Its much faster for large amounts of files and data.

WTF? Tar creates a stream, but the source data is still files and still have to be read as individual files and are still scattered all over your platters like the individual files that they are. Tar cannot magically transform random reads into sequential reads any more than I can have a four way with Jessica Biel, Scarlett Johansen and Rhianna.

However if you recognize the fact that you have two different sets of media and they will have their own read/write characteristics and there is no need for them to be in any kind of lock step you could track down mbuffer and do something like this which should shave some time off the copy:

tar -cf - * | mbuffer -s 256k -m 512m -o - | (cd /destination && tar -xpf -)

Plus it gives you a nice little status line so you can see how fast it's going and how much has been transferred.

Re:solution

[Tanktalus]

But he's also setting up reads and writes in different threads (processes, actually). Opening a new file and starting to read it will happen while the other thread is still finishing its writes and closing (syncing) the previous file.

cp probably opens input, opens output, loops { reads input, writes output }, closes input, closes output. All in serial. Probably. (Well, the -p flag adds a bit more reading and writing, which, again, the tar solution would buffer between the processes.)

I'm not convinced that one method is necessarily much better than the other. That said, I use the tar solution largely because it does the job more portably: I can copy all the files, symlinks, permissions, etc., recursively, on more platforms, than cp (I'm pretty sure the -R and/or -p flags haven't existed everywhere that I've used it). Maybe that has changed, I've not looked for a while. Also, the tar command allows me to split up the work. Not just different processors, but different machines: tar cf - . | ssh bigassserver 'cd /bigassdrive; tar xvf -' (Or reverse it if I want to pull instead of push my data.) (Or both can be ssh to use my public keys to get access to two remote machines, though I think the performance will suck at this point.)

Re:solution

[amorsen]

He's doing the same number of source opens and destinations creates, but he's adding a step.

An extremely cheap step, free in fact because the task is I/O limited. If cp is unable to read one file while writing the next (quite likely), it will lose badly.

Re:solution

[amorsen]

With two tars going on different devices, either the reading tar or the writing tar will be slowest, and the other will not contribute anything much to the wall clock time. Compare to cp -r which requires threading (or built-in multiple processes, or async I/O) to achieve the same speed. I bet GNU cp isn't using any of those.

I would expect the built-in pipe buffer in a modern OS to be large enough on its own, but perhaps mbuffer will help. On Linux you can adjust the pipe buffer size with fcntl, so you could implement the equivalent of mbuffer with just a small preloaded library.

Re:solution

[Score Whore]

What you say is true in certain situations, but if the two storage mediums have similar characteristics -- both SATA or maybe SATA & SAS or SATA & IEEE-1394 -- then mbuffer will help keep all your devices going all the time. It's not going to make an enormous difference, but for the minimal effort involved, what not take advantage of the 1% improvement it gives you over a day or two of copying?

Additionally mbuffer gives you the possibility to have multiple writers, so you can make multiple copies without having to reread the source material.

Re:solution

[Demena]

cp originally did one thing with a few options. Scrubbing it up to do the functions that cpio was written for could be regarded as system bloat.

Re:solution

[allo]

but the buffer of a pipe is a few kb, afaik. so when the first process runs out of data, the second one does not have much buffer to work on. and if the second one is blocking, the first one has not buffer space to fill.

DaisyChain

[Anonymous Coward]

I believe you can daisy chain external drives together if you have the right cases.
For ease though, I'd consider a DroBo http://www.drobo.com/products/professionals/drobo-5d/index.php

Re:DaisyChain

[Captain Hook]

It's not mentioned by the Author, so I might be assuming too much but if he's trying to write to USB Drives as opposed to a RAID of some sort I figured he wanted to be able to read the drives individually, prehaps on a different machine without a network connection between them.

The drobo won't allow that, the file system is spread across all the drives.

I guess it kind of depends on what the author needs to do with the drives when he's finished writing to them.

Re:DaisyChain

[GCsoftware]

An 8 drive DroboPro with 3 TB disks might just about do it.

Check out:
http://www.drobo.com/products/professionals/drobo-pro/index.php

Re:DaisyChain

[GCsoftware]

Actually 8x4 TB disks will do it, with the overhead etc, giving you 24.96 TB usable space.

Re:DaisyChain

[Painted]

DON'T DO THIS.

We did this exact thing using WD Green drives for our 18Tb backup problem. Got two of 'em, planning on using their built-in rsync for onsite/off siting the data. Unfortunately, the units never broke 1MB/s transfer, and no amount of work with Drobo yielded faster performance reliably. Both of our units are now sitting unused, ($2500 each!), and we put the drives into a RAID-50 8 bay USB3 enclosure. The new unit runs about 150x faster, and ended up costing $400 (prices are for enclosures only, drives were additional).

Most disappointing was Drobo's support- they just seemed to shrug a lot, and were hyper-agressive about closing trouble tickets.

Re:DaisyChain

[robi2106]

your problem wasn't the Drobo, which can be connected with eSATA or GigEthernet. The problem is using crappy "eco friendly low power" green drives. If you concern is performance, then don't get the slowest most energy efficient drives you can find. Get yourself some 10k RPM drives with 64MB cache. Like these:
http://www.newegg.com/Product/Product.aspx?Item=N82E16822236243

And then use a RAID 5 setup in the Drobo.

Re:DaisyChain

[Ost99]

Seriously? Your're blaming 1MB/s on the drives?

There must be something seriously wrong with the Drobo for the performance to be that bad.
I have a 6 drive array (raid6) with WD Green 3TB drives, and I get ~300MB/s read and write speed.

Disabling the 8sec "green" head park timer improve response times, but it should not impact transfer speed in any significant way if left at it's default.

Re:DaisyChain

[grim4593]

Even an "eco friendly low power" drive is between 1-2 orders of magnitude faster than 1MB/s.

Re:DaisyChain

[robi2106]

and yes I know RAID 5 has a big performance hit. It is a reasonable tradeoff between redundancy & speed.

Re:DaisyChain

[flargleblarg]

I get ~80 MB/sec read/write on my WD Green drives over FireWire 800.

Re:DaisyChain

[StillAnonymous]

I'm tired of hearing this "crappy slow" line about the green drives. Sure, they're not as fast as the regular 7200 performance line, but 1MB/sec?? Come on, the problem is obviously NOT the drives.

I have a number of these drives, and individually they can read between 60-80MB/sec when copying video files off them. In Linux software RAID with 4 drives, I've seen writing of 140MB/sec on a crappy Silicon Image controller.

I've seen frequent complaints about the Drobo units being slow (in the 20-40MB/sec range), probably due to a slow CPU in the thing, but 1MB/sec there's something really wrong there.

Re:DaisyChain

[Circuit Breaker]

Can you say which model your new model is, and what it is used for?

I've had good results with a Synology setup.

Re:DaisyChain

[Nogami_Saeko]

Curious what that problem could have been - Drobos aren't fast, but at 1MB/s, something is seriously wrong.

Mine has 8TB of storage currently and the transfer rate is about 30MB/s. Not fast, but quick enough for media storage and such. That said, I'd probably buy Synology hardware in the future as it is quite a bit faster, less expensive and more versatile.

N.

Re:DaisyChain

[pegdhcp]

.................... Most disappointing was Drobo's support- they just seemed to shrug a lot, and were hyper-agressive about closing trouble tickets.

You know, this is what I hate most about some tech support organisations. Some idiot in the support team managment with an engineering degree and a MBA on top of that, with no field experience, sets some aggressive KPIs for support jockeys. They try to meet their quota. For the first six-nine months he would have a very happy upper management. Then customer complaints reach upper managment via sales, in the form of customer threats for moving away the business. Upper magament fires support manager. KPI stays there, as they cannot believe a customer would prefer slower service, if it means quality -remembe those guys were the ones assigned MBA holder for support management at the first place-. Company would eventually fail.... Management by objectives my ass.....

Re:DaisyChain

[Cili]

That's because raid6 actually means something. A 6-drive raid6 system should have about 4x throughput.

Re:DaisyChain

[flargleblarg]

No, it's because the drives are natively capable of that speed over FireWire. Not using RAID of any sort.

Bacula is your friend

[bernywork]

http://www.bacula.org/en/

There's even a howto here:

http://wiki.bacula.org/doku.php?id=removable_disk

Re:Bacula is your friend

[richlv]

was going to suggest bacula as well, but came a bit late :)

Re:Bacula is your friend

[Anonymous Coward]

Yes, Bacula is the only real solution out there that isn't going to cost you an arm and a leg, and that allows you to switch easily between any backup medium. As long as your mySQL catalog is intact restoration is a synch...

Did I mention it supports backup archiving as well if you want duplicate copies for Tapes being shipped off site...

Re:Bacula is your friend

[hoover]

Another thumbs up for bacula if you need more than a single backup of your data (like copying it to drives only once)

Re:Bacula is your friend

[arth1]

Yes, Bacula is the only real solution out there that isn't going to cost you an arm and a leg, and that allows you to switch easily between any backup medium.

Except for good old tar, which is present on all systems.

Most people are probably not aware that tar has the ability to create split tar archives. Add the following options to tar:
-L <max-size-in-k-per-tarfile> -M myscript.sh ... where myscript.sh echoes out the name to use for the next tar file in the series. It can be as easy as a for loop checking where the tar file already exists and returning the next hooked up volume where it doesn't.
Or it could even unmount the current volume and automount the next volume for you. Or display a dialogue telling you to replace the drive.

One advantage is that you can easily extract from just one of the tar files; you don't need all of them or the first-and-last like with most backup systems. Each tar file is a valid one, and at most you need two tar files to extract any file, and most of them just one.

Tar multivolume can, of course, be combined with tar's built in compression.

Re:Bacula is your friend

[robbie73]

"came a bit late" - for real, after all this huge amount of pr0n, no wonder...! ;)

Re:Bacula is your friend

[h4rr4r]

You can extract from just one bacula file, you do not need the first and last. Bacula really just automates all this stuff for you. It really is the way to go.

Re:Bacula is your friend

[Qzukk]

Did I mention it supports backup archiving as well if you want duplicate copies for Tapes being shipped off site...

We had a DLT tape break inside the drive about 8 years ago and we had no other backup of the backup. Back then I wasn't able to figure out how to get bacula to mirror backups so we switched to mirrored USB drives to ensure that it never happened again.

Any links to a guide for this? Googling bacula "backup archive" or "backup mirror" isn't coming up with anything helpful and I'm not seeing anything like it described in the bacula manual. I've got two very expensive tape drives just sitting idle that I could be putting back to use...

Re:Bacula is your friend

[Orsmo]

> Yes, Bacula is the only real solution

What a minute. Really?

OP is asking for a linux console application that can perform a backup over multiple block devices (in this case externally attached hot-plugable drives like USB), and Bacula is what you come up with as the *only* real solution? Obviously you've never heard of dump.

http://linux.about.com/od/commands/l/blcmdl8_dump.htm

Re:Bacula is your friend

[datapharmer]

Not to be a jerk, but RTFM: http://www.bacula.org/en/dev-manual/main/main/Migration_Copy.html

Re:Bacula is your friend

[mdielmann]

To be pedantic, you only need the tar files that contain part of the file being extracted. This could be one or more (even all).

Re:Bacula is your friend

[Anonymous Coward]

Hmm, I have a "sTARt menu" folder in my c:\users\default... folder, but no TAR command. You said it is present on all system; can you tell me where I can find it?

Re:Bacula is your friend

[Stuarticus]

RTFM, it's been too long since I've seen that advice, brings a warm glow to my heart.

Re:Bacula is your friend

[DigiShaman]

I'm evaluating ARC Serve D2D actually. I needed a BRM recovery that also supports full with incremental backups. But most importantly, I needed a solution that will backup MS Exchange 2010 and supports native 4k disk sector sizes. Acronis still does not support Exchange 2010 (WTF?, so late now), and MS 2008 native Windows Backup will not support 4k sector sizes on newer drive. Supposedly 512e emulated support may work. But with newer external drives that use 4k sector sizes, VSS writer will fail!. As for Backup Exec 2010 and 2012, I'm fucking done with what amounts to a hacked up "API". I want a turn-key product, not a fucking roll-your-own enterprise solution for a SMB market.

And to extend my rant even further. It's the year 2012. Why on this 3rd planet from the Sun do we not have any viable working backup solutions out there!!! Every G-Damned one of them has their major problems and/or incompatibilities. WTF is up with that shit?! Blows my mind.

Re:Bacula is your friend

[Qzukk]

Well, now I know what bacula calls it.

It's still not quite what I was hoping to do (something like sending the same data to two storage daemons at once to write two tapes at once) since I can't copy from a broken tape (and honestly, I could copy from a non-broken tape just fine without bacula's help).

Re:Bacula is your friend

[g1zmo]

The way to not be a jerk about it would to be to inform the guy that Bakula uses the term migration rather than the industry standard archive.

Linking to the relevant chapter in the manual was a nice thing to do, however, so I don't think you were a jerk about it.

Re:Bacula is your friend

[Onymous Coward]

s/synch/cinch/

A cinch is a sure thing, easy.

"Synch" would be pronounced "SEENK", be more generally spelled "sync", and mean "to synchronize" or "a synchronization".

Though I suppose you might mean that restoration is actually some kind of synchronization (from the backup to the live server), in which case this is a very clever pun. But I doubt it.

Re:Bacula is your friend

[arth1]

That's amazing, if only every other major archiving program like Winzip and Winrar could do that too. Oh wait...

They can't split to separate target locations, no. Splitting a 5 TB backup file into ten pieces isn't very useful if all of the pieces have to be created on a single volume that can't hold 5 TB.

With tar, that job is easy.

Re:Bacula is your friend

[arth1]

I know you tried to make an asshat joke, but I'll respond anyhow:

Yes, Microsoft provides tar (and many other useful apps primarily associated with Unix and Linux).

Quoting Wikipedia:
"Interix versions 5.2 and 6.0 are respective components of Microsoft Windows Server 2003 R2, Windows Vista Enterprise, Windows Vista Ultimate, and Windows Server 2008 as Subsystem for Unix-based Applications[1] (SUA[2]). Version 6.1 is included in Windows 7 (Enterprise and Ultimate editions), and in Windows Server 2008 R2 (all editions).[3]"

If you have XP, W2k or a lesser version of Windows Vista or 7, you need to register with Microsoft to download Services For Unix.
If you have Windows Server 2003 or newer, or Windows Vista/7 Ultimate or Enterprise, you can turn it on or off through the Windows features in the control panel.

Re:Bacula is your friend

[pegdhcp]

:) Producer of the unnamed toy OS, used to supply a toolkit free of charge (probably as an oversight) that provides UNIX utilities. I guess it was the part of their attempt to be POSIX compatible, back then. I fortunately am not with the company anymore, whose general manager believed that any UNIX jockey must be able to run M$ products as well, thus I lost track of the current status of tool box I mentioned....

use 'dd' inlinux

[Anonymous Coward]

Use 'dd' in linux

Why USB HDDs?

[Anonymous Coward]

Are you REALLY sure that you want to use USB HDDs? The cost savings of using a box of HDDs may well be offset by the hassle in finding the backup software, the manual labor of swapping them, finding the correct drive to retrieve a certain file, etc.

How about a pair of Synology DS1512+ NASes? In addition to getting all of the storage online at all times, you get RAID support, etc.

Re:Why USB HDDs?

[jamesh]

Are you REALLY sure that you want to use USB HDDs? The cost savings of using a box of HDDs may well be offset by the hassle in finding the backup software, the manual labor of swapping them, finding the correct drive to retrieve a certain file, etc.

How about a pair of Synology DS1512+ NASes? In addition to getting all of the storage online at all times, you get RAID support, etc.

No reason why they can't all be attached at once. with 3TB disks, and 8 USB3 ports, you'ld only need to plug them all in to do the backup then remove them all to take them offsite when the backup is done.

A few portable NAS's holding 4 disks each might be a better option, but don't exclude USB for its simplicity.

Re:Why USB HDDs?

[aaarrrgggh]

Hell, USB is a pain, but going with single-drive NAS units is the easiest (portable) approach I can think of. The WD MyBookLive units are Linux based, so you can even run an rsync daemon on them.

Our setup is just for offsite backup, but we run a cron job every day to detect what drives are present, determine which drives to use by even/odd week, and rsync portions of the system to each. The drives are put into sleep mode and can be rotated in the morning. If you are backing up from a server with high throuput, you can rsync running in parallel so the individual drives aren't a limiting factor

You have to plan for how the data will change; in the /blonde /brunette /redhead case, changing tastes could put more stress on a single drive... But it is easy to sub-divide later if need be as your collection grows. If however, those directories reside in a 24TB tru-crypt container you make things much less portable, needing access to all drives for any useful data transfer.

Re:Why USB HDDs?

[SQL Error]

After a lot of messing around with USB drives, we bought half a dozen Synology DS1812+s. Backup problems solved.

Split into multiple tar files?

[Anonymous Coward]

I'm guessing you don't have enough space to split a backup on the original storage medium and then mirror the splits onto each drive?

Given the size requirements, it seems that might be prohibitive, but it would make things easier for you:

How to Create a Multi Part Tar File with Linux

A Full 24TB using only 2 USB ports

[Bondolon]

Assuming you're not worried about backup speed, you could use a four-bay external hard-drive enclosure in combination with RSYNC and LVM on any linux variety. I don't know if they all do, but the MediaSonic HF2-SU3S2 supports 3TB hard drives per bay, which means that two of them could be used in conjunction to provide 24TB of backup storage. Since you can make a large volume out of the full 24TB using LVM, you could even use something like dd to write to the disk (RSYNC with the archive option would be a better choice though, imho).

RAID

[Anonymous Coward]

For that much data you want a RAID since drives tend to fail if left sitting on the shelf, and they also tend (for different reasons) if they are spinning.

Basically: buy a RAID enclosure, insert drives so it looks like one giant drive, then copy files.

For 24TB you can use eight 4TB drives for a 6+2 RAID-6 setup. Then if any two of the drives fail you can still recover the data.

Re:RAID

[dutchwhizzman]

If not a RAID (those tend to fail just as hard) get at least two, possibly three copies of each file on separate drives. The last thing you want is to wait for RAIDs to recover and watch them fail during recovery, with your only copy of a file on them.

Re:RAID

[Kjella]

For that much data you want a RAID since drives tend to fail if left sitting on the shelf, and they also tend (for different reasons) if they are spinning. Basically: buy a RAID enclosure, insert drives so it looks like one giant drive, then copy files. For 24TB you can use eight 4TB drives for a 6+2 RAID-6 setup. Then if any two of the drives fail you can still recover the data.

Yeah... though I suspect with the price premium for 4TB drives - they're huge - and the cost of an 8-port RAID6 capable RAID card you're considerably above the budget he was going for. If this is like "projects" or something I'd probably suggest the human archiving method - split your live disk into three areas, "work in progress" and "to archive" and "archive". Your WIP you back up completely every time, your "to archive" you add to the latest archive disk (plain, no RAID), and make an index of it so you can easily find on which archive disk it is then move it to "archive" on the live disk. Very low tech incremental backup but this seems like a hobby project. I certainly hope it's not a company's backup / disaster recovery plan...

Re:RAID

[DRJlaw]

We are helping someone back up his music and audio collection, aren't we?

Well, actually you're not helping someone do anything. You're just vomiting up speculative accusations.

And they won't be pirated, will they?

See above. Then ignore children, CD rot, and every other legitimate reason for backing up the optical media that you've spent your hard-earned money on.

Now run along and sue everyone who's provided actual, helpful advice. However, you may want to look up the standard for "contributory infringement" first...

Re:RAID

[Anonymous Coward]

Very low tech incremental backup but this seems like a hobby project.

Call me old, but I wasn't expecting anyone calling 24TB a hobby project...

Now get off my lawn!

Re:RAID

[MachineShedFred]

More to the point - Do what the parent post said, but use something like FreeBSD or Solaris and ZFS with a raidz3 setup (essentially RAID6), which gives you block level dedup, snapshotting, compression, encryption, etc.

Re:RAID

[Sarten-X]

As mentioned already, RAID is not a backup solution. While it will likely work fine for a while, the risk of a catastrophic failure rises as drive capacity increases. From the linked article:

With a twelve -terabyte array the chances of complete data loss during a resilver operation begin to approach one hundred percent - meaning that RAID 5 has no functionality whatsoever in that case. There is always a chance of survival, but it is very low.

Granted, this is talking about RAID 5, so let's naively assume that doubling the parity disks for RAID 6 will halve the risk... but then since we're trying to duplicate 24 terabytes instead of twelve, we can also assume the risk doubles again, and we're back to being practically guaranteed a failure.

Bottom line is that 24 terabytes is still a huge amount of data. There is no reliable solution I can think of for backing it all up that will be cheap. At that point, you're looking at file-level redundancy managed by a backup manager like Backup Exec (or whatever you prefer) with the data split across a dozen drives. As also mentioned already, the problem becomes much easier if you're able to reduce that volume of data somewhat.

Re:RAID

[the_B0fh]

You should check out ZFS. These issues go away. And with RAID-Z3, up to 3 drives can die before you have a problem.

Re:RAID

[the_B0fh]

ZFS + snapshots. problem solved. Though you do need more drives than a 8x3TB.

Re:RAID

[Qzukk]

In this case, RAID is a method of reducing the chances of a disk failure being fatal to the backup.

Re:RAID

[d3vi1]

You didn't read what I said. Yes, ZFS+Snapshots, but you also need at least Sun Cluster replication and tape backup. ZFS + Snapshots doesn't save you from fires, floods, software bugs and ill-will. It does save you from idiots, and disk failure though.

Re:RAID

[Amouth]

if you go with 3TB drives you would need 10 for the R6, you can get that + controller + enclosure for ~2,600$ or ~10-11 cents per gig. not too bad really.

Re:RAID

[jedidiah]

You can get an 8 bay SATA JBOD enclosure for about $300 and that will include an extra SATA card. It won't be a proper RAID setup but it will allow you to mount 8 drives at once and it won't cost more than your car.

Re:RAID

[the_B0fh]

Sure it does, when you have a second set of them. Where you store tapes, store the drives instead. What do you think all those virtual tape vaults are made of?

Re:RAID

[jedidiah]

10TB is certainly in hobby territory. Putting 24TB in that same territory is not really that much of a stretch.

Times change. Tech improves. Before you know it that $100K SSD you had to borrow from a client has been replicated by cheap consumer grade stuff.

Re:RAID

[jedidiah]

Do you want your media to fail?

Do you want a medial failure to be intolerable?

If not, then RAID is a reasonable part of a backup strategy. It also turns an O(n) problem into an O(c) problem as you don't have to worry about n+1 little disks and how nothing is a clean 1:1 mapping from one drive to another.

This guy is basically trying to replicate the classic 80s PC backup where you would have a stack of floppy disks except the size of the media is bigger.

Re:RAID

[kesuki]

in my experience shelved hdd live as long or longer than powered ones, but i do not have 24tb of data. at least not on hdd. then again i've never had optical disks fail either, despite having owned a 2x cdrw, ymmv

Re:RAID

[Anonymous Coward]

The problem with a large raid like that is recovering from a failed drive. Even with a raid 6.. if one drive fails and you add a new drive to rebuild.. rebuilding 4 TB of data is going to take quite a few hours of heavy heavy heavy disk usage. There is a good chance of a second drive failing during rebuild, which means as soon as you are done rebuilding the drive you added you are adding adding another drive to rebuild... Remember the larger the array the longer the rebuild time. 8 drives has a pretty big risk of failure.

Realistically, if you are backing up 24 TB of data and this is a home project and not business or enterprise (which good solutions exist, albeit expensive), you should really cut down your data. You are looking at $1000/yr to keep that data alive.

Re:RAID

[jones_supa]

Yes, but if it's not practical for someone to backup a huge amount of data at all, housing it in a RAID at least increases the reliability a bit.

Re:RAID

[ixidor]

no not in and of it self, but i would think it would be hard to argue with calling a second copy of his data not a backup. simply because that 2nd set resides on a raid should not automatically invalidate it as a backup.

Re:RAID

[louic]

As mentioned already, RAID is not a backup solution.

Nevertheless, there is nothing wrong with using disks that happen to be in a RAID configuration as backup disks. In fact, it is probably a pretty good idea for large files and large amounts of data.

Re:RAID

[Jawnn]

Granted, this is talking about RAID 5, so let's naively assume that doubling the parity disks for RAID 6 will halve the risk... but then since we're trying to duplicate 24 terabytes instead of twelve, we can also assume the risk doubles again, and we're back to being practically guaranteed a failure.

Bottom line is that 24 terabytes is still a huge amount of data. There is no reliable solution I can think of for backing it all up that will be cheap. At that point, you're looking at file-level redundancy managed by a backup manager like Backup Exec (or whatever you prefer) with the data split across a dozen drives. As also mentioned already, the problem becomes much easier if you're able to reduce that volume of data somewhat.

Quite so.
So the correct answer to the question in TFS is "There is no 'simple' way to do this."
Better questions would be along the lines of, "How can I organize my data better, so that I don't have to perform one huge monolithic backup of 24 TB?" or, "Where can I go to learn about basic principles of IT architecture?"

Re:RAID

[AJodock]

RAID is not a backup solution

RAID certainly isn't a backup solution, however he is making a copy of his data to put onto the RAID array (aka a backup).

Re:RAID

[meddle99]

For that much data, I'd recommend just keeping the original HD-DVDs and Blu-ray media for your "20 GB" files, and the original CDs for your "3 MB" files. We are helping someone back up his music and audio collection, aren't we? And they won't be pirated, will they?

The idea of re-ripping ~2400 assorted CDs, DVDs, and BR (total 15tb) really does not appeal to me. So- I back it up, even though I have every one of the originals. Just because you don't understand something is not a reason to assume the person is doing something illegal.

Re:RAID

[tlhIngan]

You can get an 8 bay SATA JBOD enclosure for about $300 and that will include an extra SATA card. It won't be a proper RAID setup but it will allow you to mount 8 drives at once and it won't cost more than your car.

For backup purposes, I'd say skip the RAID card and use software RAID for multiple reasons.

First and foremost, if the card dies, you'd be hooped in trying to find a replacement with correct firmware down the line.

Instead, use Linux's software RAID support - the formats don't change that much (and there's lots of legacy support), plus being open source, and you're able to find a PC able to run Linux enough to be able to recover data from it - as long as ONE computer can run Linux, it'll work (even booting off a CD).

Between LVM and md, they should be supported for a good long time so your backup media will be accessible for ages.

Re:RAID

[johanwanderer]

An alternative to RAID is to get a Drobo unit (http://www.drobo.com/), fill it up with drives, and copy the data over, then remove the drives.

The advantage is that the Drobo disk set can have dual-disk redundancy, just like RAID-6, and the drives contain enough meta data that you can insert them in any order. Just make sure you have the Drobo unit in hand to read them out.

I have not had great experiences with the enterprise model (B1200i) but the consumer models (4- or 5-bay) seem to work great.

Re:RAID

[Richy_T]

+1. Software raid also allows you more flexibility. At home, I have two drives, one much larger than the other. The first 4 partitions are mirrored and the last one on the larger drive is not. I get to use all the size of the bigger drive but have the security of RAID for important stuff. Also, one drive is IDE and one SATA.

Re:RAID

[Sarten-X]

Quite the contrary, and that's my point. The errors here aren't just "let's try again" failures. They're unrecoverable, final, data-is-gone-forever errors, and the chances of encountering one are very high with so much data. Resilvering such a large array is practically impossible (as described in the article I linked to). Without resilvering and having blocks spread among disks, losing one disk means you've lost a little bit of everything, so all your data is corrupt, rather than just the fraction that was stored on the failing drive. Add to that hassle the extra expense of having more disks, controllers, and setup time, and the submitter would be better off writing a few thousand DVDs.

Re:RAID

[Sarten-X]

I always seem to forget ZFS, which is a shame. It certainly seems magical, and far more reliable than plain RAID. I'm afraid I don't know enough about it, though, to determine if the same resilvering risks apply, or if there's enough additional layers of error correction to mitigate the risk of read errors.

As I write this, a question occurs to me: Does ZFS even do disk-at-once resilvering to recover from a failed disk? If not, what's it do?

Re:RAID

[darkmeridian]

RAID is not a backup solution if you are using the drives in production. The concern is that if you delete or otherwise corrupt the file system on the RAID, you have a perfectly redundant copy of garbage without no backup to restore from. However, if you write to a RAID array, then put the device into cold storage, then you have a redundant backup system. That definitely makes sense. Even if this dude gets a dozen 2 TB hard drives, he still isn't going to have redundancy. Copying everything onto RAID will make his life easier as long as he doesn't use the RAID in production.

Re:RAID

[CityZen]

I'm jumping in here to point out some possible loose ends:
- What other medium can realistically hold a couple dozen terabytes of data? (Thousands of DVDs probably isn't viable.)
- The original poster didn't say what kind of backups he needed: long term (cold storage), available (warm), or online (hot).
    But presumably, hard drives that are turned off and put away have a different MTBF than ones that are running constantly.
- In any case, the point that any single backup can fail (spectacularly) should be well taken.

Re:RAID

[NikeHerc]

in my experience shelved hdd live as long or longer than powered ones...

Hard data point: I opened two sealed 60gb disks Tuesday (2012/08/07) and stuck'em into a server. The manufacture date was 2002 on both drives. One drive is DOA, one is working normally. YMMV.

Re:RAID

[the_B0fh]

yes yes yes and more yesses. It is bloody magical.

I just upgraded from 7x1TB to 7x3TB drives by replacing them one at a time, wait for reslivering to finish, and then swap next drive out (telling ZFS that I've replaced each drive with the new one of course).

In place replacement. When all of them are done, I suddenly have a 12 TB RAID-Z3 disk set. Raid-Z3 = 3 redundant disks. Also, peak reads and writes at 500MB/s. Bloody impressive.

And if you want to mirror a snapshot, it's just ome command to send that mirrored snapshot to an external array, or another computer or another network.

And end-to-end checksums, so your data *IS NEVER* corrupted, as long as you have sufficient copies of it around, unlike Raid, where if you read the corrupted data in, and write it back out, you're screwed.

Re:RAID

[dfghjk]

...so let's naively assume that doubling the parity disks for RAID 6 will halve the risk..."

Why assume that when it's grossly in error? If you're going to spend effort teaching us about RAID not being a backup solution, why not put a little effort into getting things right?

Re:RAID

[bingoUV]

True, but -1 for RAID just for s

Re:RAID

[bingoUV]

My bad , submitted by mistake.

-1 for RAID just for security. This makes your SATA drive perform equal to the IDE drive for writes. Also, a periodic rsnapshot from IDE drive to SATA drive would give more security and from more vectors (data corruption through misbehaving software, human operator or hardware).

Re:RAID

[bingoUV]

There is no saviour from idiots yet known to mankind

Re:RAID

[atamido]

Granted, this is talking about RAID 5, so let's naively assume that doubling the parity disks for RAID 6 will halve the risk... but then since we're trying to duplicate 24 terabytes instead of twelve, we can also assume the risk doubles again, and we're back to being practically guaranteed a failure.

Bottom line is that 24 terabytes is still a huge amount of data. There is no reliable solution I can think of for backing it all up that will be cheap. At that point, you're looking at file-level redundancy managed by a backup manager like Backup Exec (or whatever you prefer) with the data split across a dozen drives. As also mentioned already, the problem becomes much easier if you're able to reduce that volume of data somewhat.

Use Greyhole.
http://www.greyhole.net/

It distributes files across multiple storage locations, with a user defined level of duplication (lots of options for per file type and location duplication levels). Full drive failures aren't terribly common, but read errors are. To survive a drive failure AND a read error reduplicating/redistributing files to recover from the failed drive, then you would need 3x redundancy. If you used a 3x redundancy with 4TB drives, it would be 18 drives, which is a lot but not unreasonable.

Really though, that's for more of an onsite backup. To duplicate the data for taking offsite you'd only need six 3TB drives because you only need a single copy in the backup. Set up a system with the six drives, and tell Greyhole to store one copy of every file on that system. When you're ready, shutdown that system and take it offsite. For best results have two of these systems that you rotate. When you hook it up to the network, the main onsite Greyhole system will see it show up and start syncing files to it. The best part is that a full drive failure on the backup doesn't impact files on the other drives. Just replace the failed drive and wait for everything to sync back up.

In a crisis where the onsite location is destroyed, and all but one backup drive fails, you can still hook up the one working drive to any system and copy the files off.

Julian?

[WinstonWolfIT]

Out on bail mate?

git-annex

[Anonymous Coward]

You might want to look into git-annex:
http://git-annex.branchable.com/

I've not tried it, but it sounds like an ideal solution for your request, especially if your data is already compressed.

NAS Box

[second_coming]

http://www.synology.com/products/product.php?product_name=DS2411%2B&lang=uk Still portable enough to do your backup then take offsite.

Re:NAS Box

[symes]

I have a Synology NAS and I'm very pleased with it. I don't have anywhere near the volume of data the OP has though. One thing with a NAS is that you'll be subject to the networks available bandwidth and, depending on your set up, this could make backing up lots of data pretty darn tedious. And might annoy admin (and other users). So while a decent portable raid might be the better option, it might be better to find one that just plugs in rather than use the network. Might find one that can be setup to use SSDs as well.

Re:NAS Box

[second_coming]

I use a DS1511+ at work with 5 x 3TB in RAID5 setup and I love it. It's rock solid, good performance and reasonably cheap to setup. I have all my Centos servers rsync to it everynight.

Re:NAS Box

[Overzeetop]

Easy - this is his personal music, photo, and movie collection - most mp3 and older digital camera files are 3MB, and BR discs are about 20-25GB when ripped to HD.

I'm at about 9TB at the moment (well, 7TB actual usage, 11TB total disc space, 2TB of which is parity) and I have a fairly small collection. With thousands of RAW family photos, several thousand music tracks (most from a 30 year collection of CDs), and about 400 DVDs & BR discs I've ripped, it's not hard to see how I've ended up with 7TB. I know many movie collectors with libraries several times the size of mine.

I presume he has a bunch of old HDs he can put into USB enclosures or a docking station - perhaps having upgraded from a 16-20 disc 1-2TB/disc NAS to a 6x4TB NAS. There are several people on the unRaid forums with larger arrays. Rather than pitch the old hardware, they could be used for a backup.

Re:NAS Box

[jedidiah]

The problem with small drives is the fact that every drive needs it's own port or needs to share part of a common bus. Old drives also tend to be slow drives. So you end up with something that is closer to dying, slower, and more complicated.

Sometimes it's just time to chuck the old stuff.

If a drive is going to keep your solution from being able to saturate a gigabit network connection then it needs to go.

Re:NAS Box

[Richy_T]

The 200GB range drives in my main server have been trundling along for many years while I have a pile of 0.5-2TB hard drives I need to go through and get warrantied (three of them Caviar blacks). Not impressed with the big drives.

Re:NAS Box

[bingoUV]

Closer to dying is still of positive value if it is only holding a backup copy. Slower doesn't matter if it will sit in a usb2 enclosure when used and cupboard/some place away from home to be sort of offsite backup when not actively being used. And of course, a huge data amount for hobby might make most people value money, so chucking will be seen as wasteful.

So there are lots of people for whom your advice doesn't make sense.

Tape?

[mwvdlee]

Why not tape, backup RAID, SAN or some other dedicated backup hardware solution?
24TB is well within the range that a professional solution would be required.
Given a harddisk size of ~1TB, making a single backup to 24 disk isn't a backup; it's throwing data in a garbage can.
More than likely atleast one of those disks will die before it's time.

Re:Tape?

[Lumpy]

Yup. spool to tape. get a SDLT600 tape cabinet and call it done. if you get a 52 tape robot cabinet you will have space to not only hold a complete backup but a second full backup in incrementals that will all run automatically. Plus it has the highest reliability.

And anyone whining about the cost. If your 24Tb of data is not worth that much then why are you bothering to back it up?

Re:Tape?

[mwvdlee]

Assuming the 24TB is worthy to backup as a single backup.
Your movie collection is probably (A) not worthy of backup and (B) far more easily backed up as individual movies.

Re:Tape?

[Anonymous Coward]

No kidding. For $2400, you get 24x TB HDs and a bookkeeping nightmare if you ever actually resort to the "backup." For $3k, you get a network-ready tape autoloader with 50-100TB capacity and easy access through any number of highly refined backup and recovery systems.

Now, if the USB requirement is because that's the only way to access the files you want to steal from an employer or government agency, then the time required to transfer across the USB will almost guarantee you get caught. Even over the weekend. You should come up with a different method for extracting the data.

Re:Tape?

[aaarrrgggh]

It depends on the purpose, but I am guessing 20 3TB NAS drives in a weekly rotation at $4,000 is going to be cheaper and more portable than even a drobo. The reliability, assuming it isn't used as JBOD and each drive is independent, will likely be better than you can get with anything else.

If it is for "live" data, then the argument goes out the window and the OP should go with a SAN. But I have trouble finding a case where tape is a better solution.

Re:Tape?

[SlashDev]

I don't think tape is a good idea for large backups, mostly because of the restore times it would take, tapes are sequentially read/written so a search and restore for data at the end of the tape would take a very long time.

Re:Tape?

[SecurityGuy]

Riiiight. And what do T10KC drives cost again?

Re:Tape?

[mjwx]

Yup. spool to tape. get a SDLT600 tape cabinet and call it done. if you get a 52 tape robot cabinet you will have space to not only hold a complete backup but a second full backup in incrementals that will all run automatically. Plus it has the highest reliability.

And anyone whining about the cost. If your 24Tb of data is not worth that much then why are you bothering to back it up?

And having 2 full backups + incrementals in your tape drive when the office burns down is going to be a great help.

Backups are useless unless you cycle them. At the very least the monthly's must go off site.

GIS produces a shitload of data that needs to be preserved. 30 GIS analysts can produce 8 TB in no time, Thank Buddha that LTO4 came out when it did. I was backing up 18 TB in a full backup with a weekly change of 4-6 TB (depending on work volume) back in 2008. I could have a single 24 tape loader do full backups, I insisted on 2 but they never found the money...

Yep, I handled up to 6 TB of unique data generated per week and they still called me a storage nazi.

Re:Tape?

[Minwee]

Riiiight. And what do T10KC drives cost again?

If you are the kind of person who asks questions like then then Oracle is clearly not for you.

Re:Tape?

[Lumpy]

if you are still housing critical data gear in buildings that can burn, you are in the wrong business. Cement bunker/ box building. steel roof. nothing at all will burn and when the halon bombs go off, the only fire that is a concern is a magnesium fire. and we have a rule not allowing magnesium in the server building.

Mitigate your fire risk. and yes it's built on a hill so no flood risk either. no a tornado can do squat to it, this is the cool part of using an old Civil Defense shelter as a server farm. we even think it's bomb proof.

Re:Tape?

[mjwx]

if you are still housing critical data gear in buildings that can burn, you are in the wrong business.

Dont ask me, ask management who decided the ideal place for a server room was a wooden shack mounted to the second story of the side of their warehouse.

I cringe every time I see this.

tar --multi-volume

[jegerjensen]

Evidently, our UNIX founding fathers had similar challenges...

Tar already does this

[cyocum]

Have a look at tar and it's "multi-volume" option.

Re:Tar already does this

[leuk_he]

multi volume tarJust mount a new usb disk whenever it is full.

However to have reasonable retrieve rate (going through 24 TB of data will rake some days over USB2), You better split the dataset in multiple smaller sets. That also has the advantage that if one disk chrashes (AND Consumer grade USB disk will chrash!) not your entire dataset is lost.

For that reason (diskfailure), do not use some linux spanning disk feature. File systems are lost when one of the disks they write on are lost. Unless you use a feature that can handle lost disks (Raid/ Zraid)

And last but not least: Test your backup. I have seen myself cheap USB interfaces failing to write the data to disk without a good error messages. All looks ok until you retreive the data and some files are corrupted.

Re:Tar already does this

[coldsalmon]

Yes, and don't forget the danger of bad blocks in a job this size. A single bad block can take out an entire archive. I used to backup my 300GB of data into a single .tar file on external USB drives. Unfortunately, when I tried to restore after a system failure, I discovered that BOTH of my backups had bad blocks about 30GB into the .tar file. After the bad blocks, the rest of the file was unreadable. I attribute the bad blocks to the high temperatures inside fanless external HDD enclosures (this is also a danger in your scheme). Now I use rsync, so bad blocks won't kill more than one file at a time. If you must use an archiving backup program (like tar), break up the backups into small files, and make sure that they can be restored individually.

Re:Tar already does this

[Richy_T]

Tar can seek past bad data FWIW (though if you zipped, you're in trouble).

Linuxquestions thread on multi-disk backups

[Anonymous Coward]

Here's a Linuxquestions thread outlining multi-disk backup strategies.

The gist of the discussion is to use DAR.

No.

[AdmV0rl0n]

I'm not sure if you posed the question out of being nieve, or if its just being daft. You don't want to be moving 24TB over the USB bus. End of discussion really - at least in terms of USB.

Whoever or however you ended up looking at USB for this was wrong/wrong way.

You have lots of choice in terms of boxes, servers, NAS boxes, locally attached storage. 24TB is in the range of midrange NAS boxes.

Once you have this, you can start to make choices on the many backup, replication, and duplication bits of software that already exist, both free and proprietary.

Re:No.

[ledow]

USB 2.0 provides 480Mbps of (theoretical) bandwidth. So unless you go Gigabit all over your network (not unreasonable), you won't beat it with a NAS. Even then, it's only 1-and-a-bit times as fast as USB working flat-out (and the difference being if you have multiple USB busses, you can get multiple drives working at once). And USB 3.0 would beat it again. And 10Gb between the client and a server is an expensive network to deploy still.

Granted, eSATA would probably be faster but there's nothing wrong with USB for such tasks if you *don't* want to provide Gigabit connections everywhere and (presumably) greater-than-gigabit backbones.

Re:No.

[Bill_the_Engineer]

You don't need a Gigabit connection everywhere, just on your computer and the NAS directly connected to your computer.

USB2 is not a very good option. For some reason, I've been getting poor performance from Linux with storage mounted via USB. Your best bet is eSata. If you can't install eSata, but have a Gigabit eithernet connection then go that route. USB2 is the connection of last resort when talking about backing up 24TB.

Re:No.

[tippe]

USB 2 is half duplex and from what I'm told has quite a bit of overhead, not to mention the fact that depending on how things are connected together (i.e one controller vs many), the available BW might actually be shared amongst multiple devices. Ethernet can in some cases suffer from some of these problems as well, but it is at least full duplex, and in this day and age, who doesn't run Gigabit Ethernet?

Re:No.

[asdf7890]

USB 2.0 provides 480Mbps of (theoretical) bandwidth. So unless you go Gigabit all over your network (not unreasonable), you won't beat it with a NAS. Even then, it's only 1-and-a-bit times as fast as USB working flat-out (and the difference being if you have multiple USB busses, you can get multiple drives working at once).

The 480Mbps is nowhere near what you will see in practise, unlike network speeds which are far closer to the rated maximum. Most USB drives I've seen top out at somewhere between 25 and 30MByte/sec, and if there are no other bottlenecks it isn't unusual to see 100Mbyte/sec from a gbit switched network. My main desktop pulls things from the fileserver at around 80Mbyte/sec, which is as fast as local reads tend to be on that array. So you are right about 100mbit networks: that'll be the bottleneck not USB, but gbit networking should outdo USB2 by at least a factor of 2, possibly 3, maybe even more if you have better drives in you main storage array than I do.

Before trying to run several USB drives to max out your network bandwidth, consider that you will taking the source disks too. Unless they are SSDs having 2, 3, or more concurrent bulk reads going on may not be any faster than one concurrent read as all the extra head movements will wipe out the bulk speed potential. If the OP's 24Tb is spread over numerous physical drives this need not ban an issue though (with planning careful enough to ensure there aren't two bulk processes reading from the same physical devices.

And USB 3.0 would beat it again.

That it would. I have an SSD in a USB3 enclosure, and it can happily consume 80Mbyte/sec read over my little network. It might even be able to do better than that: I've not measured a bulk write read from the internal SSD yet.

And 10Gb between the client and a server is an expensive network to deploy still.
Granted, eSATA would probably be faster but there's nothing wrong with USB for such tasks if you *don't* want to provide Gigabit connections everywhere and (presumably) greater-than-gigabit backbones.

If I wanted more speed than USB3+gbit can provide (due to the size of data being backed up on each run) I'd be plugging the backup device(s) in locally to the source (vie eSATA, USB3, or such) rather than using the network (though again taking note to be careful how things are done if trying to use more than one backup device at once).

For the size of data being described, I'd not want a set of USB drives to be my primary backup solution though.

Re:No.

[AdmV0rl0n]

You just wrote that because you think you're clever and you like an argument. You're not clever, and its no argument.

Gigabit ethernet and a midrange NAS box roughly complment their throughputs, and having a NAS thats setup, possibly offering RAID is a far better solution that your idea of entertaining multiple drives over USB. And theoretical bandwidth is meaningless. USB2 real world tends to be limited to 33MB/s ballpark. Doing something badly multiple times doesn't stop it from being bad.

You are better on your eSATA point, but I regard that as lost given you wrote drivel about thinking USB2 is an ok way to go for this.

Its not.

And who the fuck gave you (3) for your posting. Idiots

Re:No.

[jedidiah]

> That it would. I have an SSD in a USB3 enclosure, and it can happily consume 80Mbyte/sec read over my little network. It might even be able to do better than that: I've not measured a bulk write read from the internal SSD yet.

Unless you've got a slow SSD drive, it should have more performance potential than just 80MB/s.

I have gotten 100MB/s transferring between two SATA spinny drives across my gigabit network.

Now this is a lot of data we're talking about. Creating the backup and restoring the backup both will be performance sensitive. Every little bit you can do to help push things along will be very worthwhile in the end. It probably makes sense to be as close to the bare metal as you can be.

You might even want to buy more expensive drives to get better performance. Perhaps a better RAID card too.

Re:No.

[jones_supa]

Whoever or however you ended up looking at USB for this was wrong/wrong way.

Wrong/wrong? You could have written "wrong/{wrong way}" to make clear which words are part of the two options.

Re:No.

[asdf7890]

The 80Mbyte seems to be a network limit, not a bulk disc throughput limit. A quick test reading ~10Gb (in two large files not many small ones) locally on the server gets just under 120Mbyte/sec overall. I might be able to get better from those drives but I wasn't overly careful about stripe alignment and such (they are 4K sector 2Tb disks using Linux's software RAID). I'm not sure if those particular files are closer to the slow end of the spinning metal or closer to the fast end. Pulling the same files to the Windows box from a Samba share is where I mentioned 80Mbyte (in fact a quick test with the same two files shows it top out at just over 65Mbyte). The server doesn't have anything SSD but the desktop box uses them for internal storage, I've not benchmarked copying from them to the external SSD though (I have copied large things that way, but just not paid attention while it happened, where I have watched the transfer rates while copying from the server. The server has no USB3, so plugging the drive in directly would not get any benefit from the local connection due to USB2 throughput limits. I've not investigated where that drop from ~120Mbyte/s to ~65 comes from. It could be a mix of Samba not being terribly efficient (it does sit eating >65% of a CPU core while the operation proceeds), me having cheap NICs and a cheap switch, and me not having jumbo frames enabled anywhere (the network is shared with devices that I think don't support larger frames), and probably some other factors I've not considered.

You know...

[marsu_k]

Porn is a renewable resource, there's no need to store so much of it.

Re:You know...

[Anonymous Coward]

So renewable, you can even make your own..

Re:You know...

[couchslug]

"Porn is a renewable resource, there's no need to store so much of it."

There was only ONE Beatrice Arthur.

Yeah, you missed basic common Linux knowledge

[Anonymous Coward]

Script your own solution for your specific problems.

That’s kinda the whole point of having a computer... as opposed to a set of appliances that happen to run on a computer you never use directly.

Seriously: Build your own homebrew NAS.

[Qbertino]

What your attemting isn't easy, it's actually difficult.
Buy a cheap and big refurbished workstation or rackmount server, install a few extra SATA controllers and maybe a new power supply, hook up 12 2TB drives, install Debian, check out LVM and your all set.

Messing around with 12 - 24 external HDDs and their power supplys is a big hassle and asking for trouble. Don't do it. Do seriously go through the possibilty of building your own NAS. You'll be thankfull in the end and it won't take much longer, it might even go faster and be cheaper if you can get the parts fast.

My 2 cents.

Re:Seriously: Build your own homebrew NAS.

[DRJlaw]

What your attemting isn't easy, it's actually difficult.
Buy a cheap and big refurbished workstation or rackmount server, install a few extra SATA controllers and maybe a new power supply, hook up 12 2TB drives, install Debian, check out LVM and your all set.

Messing around with 12 - 24 external HDDs and their power supplys is a big hassle and asking for trouble. Don't do it. Do seriously go through the possibilty of building your own NAS. You'll be thankfull in the end and it won't take much longer, it might even go faster and be cheaper if you can get the parts fast.

Way to redefine the problem instead of working within the specifications.

Perhaps:
1. The poster ALREADY has a NAS and wants to have airgapped or even offsite/offline backup.

2. External HDDs are fast, common, reasonably cheap, and do not have a single point of failure (e.g., the tape backup drive in many suggested alternatives)

I'm interested in this question. I use this general setup, but on a smaller scale. I cannot put a NAS in a safety deposit box. I cannot ensure that my "backup" NAS would not be drowned in a flood, burned in a fire, fried by a lightning strike...

Let's pretend the poster is not an idiot, and answer the actual question. If he has 24TB of data, IT'S ALREADY ON DAS/NAS. Geesh.

Re:Seriously: Build your own homebrew NAS.

[Bill_the_Engineer]

Let's pretend the poster is not an idiot, and answer the actual question. If he has 24TB of data, IT'S ALREADY ON DAS/NAS. Geesh.

Don't assume he was the one that created his current storage solution. It could be a turnkey solution that he purchased, like one of those movie storage devices we read about on slashdot earlier this year.

If he installed his current storage configuration himself then why did he need to ask this question on Slashdot? I don't see any particular bad answers, and no one is insulting him. I'm sure he is mature enough to filter all these responses and pick the right one for his situation.

Re:Seriously: Build your own homebrew NAS.

[asdf7890]

For 24Tb of data you want more than 12x 2Tb drives. At least 18x 2Tb so you can apply RAID5 (or some compound RAID based on arrives of RAID5) or 24x2Tb for RAID1, so that you can survive at least some drive failure situations without data loss. Be careful of array rebuild times with drives that big though. ZFS is something to consider with an array this size (with smaller drives preferably as re-image/re-checksum times will not be short with this over large drives either).

You are right that anything like this would be much less hassle in the long run than many separate external drives, and their PSUs (something I'd not considered).

Re:Seriously: Build your own homebrew NAS.

[gl4ss]

it's pretty probable that the data is already on such device, but needs to be moved around to somewhere else on something cheap.

Re:Seriously: Build your own homebrew NAS.

[Joce640k]

Let's pretend the poster is not an idiot, and answer the actual question.

On slashdot...?

"Ask slashdot" is never about reading the question or formulating a relevant answer.

Re:Seriously: Build your own homebrew NAS.

[DRJlaw]

Then put your backup NAS in the garage or a garden shed, only power the NAS while needed and yank out the network cable if you have to.

Attached garages burn and get broken into. Tenth floor condominiums don't have garden sheds. I can knock down any alternative that you propose just as easily as you can ignore the fact that while alternatives are nice, the Ask Slashdot request was for a solution to a particular problem, not a general method of backing up data.

External HDDs means USB, and USB 2.0 is not fast (30-35 MB/s, faced with 24 TB source storage this is very slow, GBit/s Ethernet can give you 80 MB/s to 110 MB/s).

External HDDs means USB or eSATA, and USB3.0 is as fast as any 1-3 HDDs. Hence the Seagate GoFlex 3 TB USB3 drives that I purchased for $99 each at Best Buy last Black Friday.

RAID5/6 and lvm2 (zfs if you like an are already familiar with it) and you're done in an easy and maintainable way

Except that any two or three drive failures, such as caused by a lightning strike, can eliminate all that data. Particularly if your NAS and your backup NAS are in the same location. My 'NAS' is already RAID6. Fat lot of good it'll do me if the house burns down.

Dealing with ~10 external disks guarantees just one thing, that you'll never actually do the backup.

No, it guarantees that you won't have a current backup of frequently changing data. If your NAS has a large quantity of stale but online data (because you want the availability, or it's a media collection, or it's 5 years of home movies from a camera-head), frequency may be almost a non-issue.

External HDDs have a number of advantages that the "insane" may desire:

1. If it's not hooked up to power, it can't be hacked and can't be fried except by EMP.

2. If its not bundled together in a RAID, nothing short of complete failure will cause complete data loss.

3. External HDDs are very easy to haul between home/office and a remote site.

4. (becuase other comments keep raising this issue) If you buy the same model eHDDs, you're only managing as many power supplies as you need simultaneous connections. The aforementioned Seagate drives also use essentially the same power supply as my prior FreeAgent 750GB USB2/eSATA/FW400 drives.

Organizing your storage so that the live and changing data is well segregated from the live and stale data eliminates much of this problem. For example, a live data drive with operating system, client computer backups, user profiles, and other live data could be automatically backed up to an identical eHDD or -- crazy, I know, -- iHDD installed in a hotswap dock. A stale data section of a NAS could be manually backed up on eHDDs. The stale data backups, and any live backup that you happen to have at the time, can be dropped off in your saftety deposit box -- which for most normal people will not fit a NAS.

So... there is a sane option in answering the stated problem rather than proposing your own defective solution based on your own priorities and parameters. You simply don't know a relevant answer and choose to attack the question instead.
Grade: C-.

Re:Seriously: Build your own homebrew NAS.

[rastoboy29]

You have never worked with business grade tape.  It is way more reliable than HDD.

And it can be put in a safe deposit box.

There is a reason it still exists--there is nothing better, yet.

Bash....

[djsmiley]

First bash script to grab the size of the "current" storage;

compress the files up until that size;

Move compressed file onto storage;

request new storage, start again.

----------

Or, if you've got all the storage already connected; bash for 0..x; do { cp $archive$x /mount/$x/ }; done :D

we generate a lot of data (3 GB/min)...

[acidfast7]

... by employing a detector with a size of 2463 x 2527 pixels (6M) at 12 Hz (12 times / sec). When run continuously for a set of data (roughly 900 degrees) ...

we collect 900 frames in roughly 2 minutes including hardware limitations for starting/stopping.

In proper format for processing, this works out to about 6MB/image and roughly 3GB/min for 2 minutes.

With an experienced crew of 3-4 people ... one handling the samples, one handling the liquid nitrogen, one running the software and one taking notes (overall monitoring also) ... we can run through 600 samples in a 24 shift ...

Which roughly works out to about 600 x 6GB = 3.6 TB on a "working" day.

To answer your question ... we never make physical copies of stuff ... the data stays online in multiple places on multiple continents ... and when something is published the data becomes publicly available in a central database

Why do you need a physical copy anyway?

Re:we generate a lot of data (3 GB/min)...

[ledow]

I'm not the OP but:

Because downloading 3.6Tb to restore from a backup for just one day is pretty ridiculous for someone on a home broadband?

Backup to external servers is ridiculous for anyone without university-sized access to the net. Hell, the school I work for try to back up 10Gb to a remote server each night and it often fails because it took too long (and we're only allowed to do that because we're a school - the limits for even business use on the same connection are about 100Gb a month).

Absent a stupidly fast connection for a home, you have to have a physical copy that you can put somewhere else.

The fact that you *don't* see that, tells me that you probably have far too much hardware and connectivity available to you.

Re:we generate a lot of data (3 GB/min)...

[acidfast7]

Two quick things:

1. Why do a complete restore of the 3.6TB? Just take the files that want to use again/have been lost.

2. Why work at home? It's home, not work.

Re:we generate a lot of data (3 GB/min)...

[Bill_the_Engineer]

I agree. I have similar data generation and storage requirements. Our science team is spread over two continents and we too use multiple storage locations for our online backup. Having said that, I just had our IT guy install a brand new high capacity tape drive so that we can start archiving the data into long term storage. So the two backup schemes aren't mutually exclusive.

Re:we generate a lot of data (3 GB/min)...

[History's Coming To]

1: Yes, syncing makes more sense than full backups, but you still have to do the initial 100% "sync". If you lose your whole HD then before you can restore a single file you need to reinstall the OS and apps, plus any custom scripts and other customisations you might have made. A full disc image is a life saver if your main desktop goes to crap.

2: In my case, because my job doesn't have a physical location. If I didn't work from home I'd be working from the beach, the car or the top of a mountain. It's happened, but home is much more comfortable, has all of my books etc, and I can claim back rent, electricity, phone bills etc against tax. I could hire some office space, but I'd have the extra expense and a 1.5hr commute each day, and I couldn't claim back some of my household bills.

Re:we generate a lot of data (3 GB/min)...

[acidfast7]

1. not an issue

2. sorry man, that kinda sucks.

Re:we generate a lot of data (3 GB/min)...

[Richy_T]

2. Huh?

Re:we generate a lot of data (3 GB/min)...

[flargleblarg]

> Because downloading 3.6Tb to restore from a backup for just one day is pretty ridiculous for someone on a home broadband?

3.6 terabits isn't that much.

> Hell, the school I work for try to back up 10Gb to a remote server each night and it often fails because it took too long

100 gigabits isn't that much.

USB is not for backup

[aglider]

USB is for a second working copy.
Backups should also ensure durability of the copy, while USB HDD have a shorter lifespan than a normal HDD which in turn has shorter lifespan than tapes, the usual medium for durable backups.

Re:USB is not for backup

[Anonymous Coward]

USB HDD have a shorter lifespan than a normal HDD which in turn has shorter lifespan than tapes

What planet are you from?

Re:USB is not for backup

[phillymjs]

I think he might mean that a HDD sitting in a server and running 24/7 will likely last longer than a HDD that's in an external enclosure and gets physically moved around and powered on/off frequently.

Re:USB is not for backup

[History's Coming To]

Are you confusing Flash storage (eg a USB stick) with a normal platter-based HDD which uses USB cables for transfer? USB != flash, HDD != USB, ethernet., Firewire etc. OP is talking about HDDs connecting via USB as far as I can tell.

Use DAR or KDAR

[pegasustonans]

If you don't want to invest in new hardware, you could use DAR or KDAR (KDE front-end for DAR).

With KDAR, what you want is the slicing settings.

There's an option to pause between slices, which gives you time to mount a new disk.

Use purpose designed backup media.

[gallondr00nk]

Backup tapes were designed precisely for the problem you have. LTO-5 tapes are about 1.5TB, if I remember right. Stored correctly they shouldn't give any problems when you come to retrieve whatever is backed up. Most archiving efforts use backup tape, and they can't all be wrong :)

Re:Use purpose designed backup media.

[Antique Geekmeister]

Actually handling all those tapes and recovering data from them is very expensive in manpower and time, and can be very awkward for recovering data. Those tapes, and tape drives, are also _expensive_. They're useful for sites that require secure off-site storage, or encrypted off-site storage, but for most environments today they are pointless. Easily detachable physical storage has become very inexpensive, far more economical, and is far less vulnerable to the vulnerabilities of mishandling SCSI connections. I've seen far, far too many SCSI setups for tape drives and external media fail due to misconfiguration, miscabling, and the very poor driver integration of SCSI controllers in far too many operating systems. USB has proven startlingly simple, resilient, and _cheap_ to manage.

I use the external drive approach very frequently for data center migration and virtualization OS image migration, though usually I only back up the configuration files from the virtualized hosts, not the complete images. It's very effective. 24 TB is bigger than I've personally done this way, but it's certainly feasible if it's not treated as a single lump. If the the data can be factored reasonably before transferring, don't simply duplicate it every time. Split it up into reasonably sized chunks and _mirror_ it onto the USB drives, so the first backup is lengthy but following backups are far more efficieint.

Assuming that the backup system is Linux based, the ""rsync" tool can be written into a script to see which media is attached, to mount those media, and to mirror the contents of a set of directories to those media.. It's also reasonable to use a USB hub to allow mounting multiple USB devices simultaneously so it can be done all at one time, rather than having to swap media.

Re:Use purpose designed backup media.

[tomtomtom]

Actually, for a data set this large it will probably work out only very slightly more expensive - and the benefit to be gained is worth it IMHO (in speed if nothing else - USB disks are *slow* and eat a lot of CPU). I live in the UK so I'll work in GBP. I think US prices are likely to be cheaper but the relative sizes will be similar.

I'd figure around ~£1100 for drive and SAS interface plus £500-700 for 24TB worth of media. Throw in an extra 2TB drive to spool to before you write to tape as well for say £150 (if you are buying SAS) and you get to less around £81/TB (which works out roughly the same as current external hard drive costs). If your data is precious though you'll want double the amount of media so you can store offsite (or at least have a spare backup). Then the lower marginal cost of tape vs disk will become apparent.

Yes, tape can be harder to configure correctly and swapping tapes over etc will be a pain for a set this large. But that's equally true for disk; and we all know that it's not a backup until you've checked that you can restore from it. User error in configuration of the backup scripts is way more likely to cause an issue than any kind of hardware error and for that reason alone, you are stupid in my opinion if you don't test your backups. If you test them, then you will spot any SCSI misconfiguration etc immediately.

I agree that for moving data around, disk (or network) is much much easier. But that wasn't the submitter asked about.

Re:Use purpose designed backup media.

[Antique Geekmeister]

The slow disk is why you use rsync or other such efficient mirroring technologies. The tapes have a limited lifespan, they require significant maintenance, and have been prone to far too many mechanical failures and expensive downtime in my experence. The disks can actually be simultaneously connected for casual "read" access with a reasonable USB hub and possibly an additional USB card.

You've also left out the cost of recovery time for users. Swapping tapes to get recovery of arbitrary files is rather awkward. and requires physical access to the server. Either you invest in a tape library that can hold that 24 TB of tape online and provide user access to the tape library (which means two drives in the library, one for backup activity and one for read activity), or you invest in onsite manpower to swap tapes. If you've staff already in the computer room to swap tapes for you, that's workable, but it's very expensive in a small environment, and it interferes with backup operations if you've only the one tape drive.

Swapping USB ports, or even leaving the multiple backup drives constantly on, is much quicker to access the same data, and leaves the full backed up systems or designated pieces of it directly accessible for network exporting, such as with a read-only NFS mount or read-only CIFS mount, for clients to easily recover their own files rather than requiring backup system access. You do have to be cautious with export configurations for performance and security reasons, and to leave the drives easy to swap out, but it's workable.

Re:Use purpose designed backup media.

[Bill_the_Engineer]

No matter the backup solution being proposed there are hardware costs involved. When the backup size reaches 24TB, dedicated backup devices become quite competitive. Especially when you consider that some of these tape drives are designed for autonomous operations with a tape jukebox. Sure the hard drive may be slightly less expensive, but they consume much more electricity and require more cooling than a tape drive. Also if you do purchase one of those jukebox tape drives, you won't have to waste time changing media every time you want to make the backup.

The easier and less time consuming the backup procedure is, the more likely regular backups will be made.

Re:Use purpose designed backup media.

[tomtomtom]

Whether tape or disk is appropriate really depends what you are intending to use the backup for and how important your data is. You might even choose to use a mixture of the two.

If it's your only backup, I would suggest that it's not wise to leave it permanently online in the way you suggest; that leaves you open to any number of potential issues which your backup is supposed to protect you from (OS bug, misconfiguration, lightning strike, power failure, overheating, ...). Tape libraries have the same issue although at least there you are exposed to a different set of software bugs and the other tapes in the library might be OK if they are not physically in use when the worst happens.

For the inadvertent file deletion, you can cover this with better tools using true online storage - effectively some form of regular snapshotting (ZFS snapshots, rdiff-backup, Windows VSS, etc) to keep a (shortish) recent history. This should cover a good proportion of restore requests depending on how much history you can keep. For the rest, you're right that if you need to restore files very regularly then you might need a second drive and/or robot. Whether you need to do that or not will just depend on your use case.

Even if you do go with disk, make sure you use something which can properly keep multiple versions of files - just rsync'ing a big directory onto another disk is a recipe for disaster. My personal favourites are rdiff-backup and DAR (which can handle multiple volumes as others have pointed out) but there are others out there too, eg bacula.

Re:JBOD or more accurately, spanned volume

[sumdumass]

how transportable is that though?

I mean, if i copied 200 gig across 3 drives in a jbod raid, could i plug just one drive in to access the information on another machine? Suppose my laptop only has 2 usb ports and i do not have a hub plus i'm running a different OS, does this mean i can't look for information on the set?

I have never used JBOD for raid, I have however used regular mirrored and stripped raids with and without fault tolerance (raid 5 and 10 or a mirrored stripe for instance) and know this can be a problem. In fact, I've even seen issues reading a complete raid set across systems when you aren't using a true hardware raid controller.

PAR

[fa2k]

I have just seen "PAR" a couple of times here on slashdot, haven't used it, but it seems great for this: http://en.wikipedia.org/wiki/Parchive . You need enough redundancy to allow one USB drive to fail. And I would rather get a SATA bay and use "internal" drives than having to deal with external USB drives. Get "green" drives, they are slow but cheap.

24TB?

[codman1]

Some sort of NAS or tape would be your best option without knowing more. How often do you need to do the "backup"? Is it really a "backup" or data replication eg. are you needing to restore the data after a serious failure. Have a look at this seems to have some good advise and i think could be a solution to your issue, as i see the big problem is the amount of time and the restorability of the data after a failure. http://www.smallnetbuilder.com/nas/nas-howto/31485-build-your-own-fibre-channel-san-for-less-than-1000-part-1

NAS

[Wolfling1]

A 24TB NAS is not very hard to assemble. Relatively cheap, and basically transfers data at Gb speed - assuming that you populate it with fast disks. Set one up with RAID and you're away. Personally, I would do it with a low end server and a big-ass RAID array. That way, you can really control its behaviour via the OS. Linux is ferpect for this kind of thing.

eSATA

[tmshort]

Your best bet for speed is likely to be eSATA.

Have you looked into something like this:
http://eshop.macsales.com/shop/NewerTech/Voyager/Hard_Drive_Dock

The cost becomes noise when you consider how many drives you will end up needing, and per TB, will be cheaper than USB solutions.

I don't know how your data is organized, but if possible, you may want to back it up by project/directory/etc.

There are also online backup systems that can do what you want, but it'll take an extremely long time...

Re:JBOD or more accurately, spanned volume

[hawkinspeter]

Seems like a very bad idea to me. You'll have trouble creating a JBOD device without connecting all the drives simultaneously. Also, you're basically increasing the chance that the entire JBOD volume will be broken as the number of drives goes up. If you've got one drive failing, you'll be lucky to get any data back at all.

To my mind, Bacula would be a good choice as you can set up virtual tapes that will correspond to the drives and you can set the backup to wait for the operator to swap over the drive and then continue the backup. Also, once you've got Bacula installed and working, it's easy to do incremental backups and thus not need to write out the whole dataset again.

I know

[Anonymous Coward]

The iCloud! ;-)

Going about it all wrong

[Charliemopps]

Get an old computer... anything will work really. You have to know someone that has one laying in their basement. Plug your drives into that. share the drives on your network. Use any general backup software and sequentially backup what you need to backup over the network. Now it will do it overnight and you really don't care how long it takes. It can even do it every night. If you want it safe from fire and such.... build a box out of 2x4s and Drywall scraps form homedepot. Make it 5 sheets thick and it'll withstand any housefire you could possibly have. If you really want to go hardcore you can pour a box out of concrete, but that'll be hard to move.

Re:Going about it all wrong

[DarkOx]

build a box out of 2x4s and Drywall scraps form homedepot. Make it 5 sheets thick and it'll withstand any housefire you could possibly have

I find that statement suspect. I am not saying you are wrong but extraordinary claims require extraordinary evidence.

I have seen some pretty nasty house fires, the kind were the fire department sprays water on the neighbors houses to keep them from catching rather than try to do anything about the one that is actually burning. With all the modern synthetic materials in furniture, carpeting, and other flooring a house fire can hit 600 degrees and stay that way for hours.

If five sheets of dry wall ( .5" or .75" ?) was sufficient to insulate a hard disk such that it will useable later, in what amounts to a 600 degree oven for an extended period; I suspect all our outside walls would be five layers thick with the stuff. Cheap as dry wall is the energy costs in terms of heating and cooling would be recouped in a few years if it were that good.

Read it from Torvald's lips

[zapyon]

"Only wimps use tape[*] backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it ;)"
Linus Torvalds (1996) http://en.wikiquote.org/wiki/Linus_Torvalds

(Isn't that prescience of "The Cloud"?)

––––––––––
* replace this with your favorite backup media of today ;-)

Re:Read it from Torvald's lips

[rvw]

"Only wimps use tape[*] backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it ;)"

Linus Torvalds (1996) http://en.wikiquote.org/wiki/Linus_Torvalds

(Isn't that prescience of "The Cloud"?)

––––––––––

* replace this with your favorite backup media of today ;-)

"Only wimps use ftp[*] backup: real men just upload their important stuff to the iCloud, and let the rest of the world mirror it ;)"

An Amazon support employee (2012)

Re:Read it from Torvald's lips

[wvmarle]

Linus should have patented his idea! He could have become a rich man.

Kim Dotcom

[Albinoman]

It's a little late to be asking that now.

Count Bacula

[freaker_TuC]

Count Bacula as your friend ;) -> http://www.bacula.org/

What do you have now?

[DarwinSurvivor]

Sometimes the easiest way to duplicate (back up) data is to simply duplicate the hardware it's already on. If it's on a 16-disk (x 2TB) NAS system, build another one. If it's on tape, buy more tapes, if it's on random HDD's scattered all over the place, then you have bigger problems to deal with first (like building a NAS box)!

Backup advice

[Anonymous Coward]

I do things like this all the time with a data set about half of that, ~ 12TB. You didnt say anything about what the data is but from the request and the fact you mentioned USB I would gather this is your typical warez hording mp3/flac, mkv, apps and also a personal picture and video collection of fam.

Here is a checklist i would execute similiar to mine. I find the most reliable way to keep your data over the years is by following a checklist or procedure and choosing when to move to the next storage platform.

Step 0: Get USB out of your head. Pop upon the drive and attach it to the native bus, PATA, SATA. if SATA may want to invest in ESATA cases. Its not solely the speed. I have done stupid things like this, in which the data backup takes over 2 days, and on the 2nd day some unrelated event affecting my USB bus causes all kinds of problems with the transfer. Over time doing cheesy things like this affects other things, like doing stupid shit in real life, usually with duct tape or guerrilla glue, then you have your wife on you. Right now your wife may not catch on to this, but it will escalate. Just do shit the right way.

Step 1: Organize. Actually understand what you are backing up. I never got into these tools like google desktop that allow a user to accept the fact that he/she has no idea where their files are. Understand and make an effort to organize your files before you back them up and know the capacity of each 'genre' of crap you are backing up. Run a tool like 'jdiskreport' to find this information out after you organize. Create a mapping on paper of where shit is going, zork style. If you have really important shit like family pictures, taking up say 200GB, and your mkv collection is 12TB, you may want to make 2x copies of your family shit. Anything you download off the internet is easily replaceble despite how obscure your tastes may be and will turn up again. I would question even backing it up but that is another conversation.

Step 2. Label your drives accordingly to your documentation.

Step 3. Format the drives in the most likely native format you will use and are familiar with. If you are a noob linux guy who runs Windows 7 all the time, dont be an idiot and experiment with your backup on ext3. It is not that ext3 is a bad filesystem, but you may not be the most skilled in restoring your data in various scenarios. For example im a linux and solaris geek but am just getting into macs --- im not comfortable enough with mac failures enough to store my crap on a mac fs. Whatever your skillset is, dont use the most optimal file system on paper, use what you know, even if it is NTFS (which imo is very reliable).

Step 4. Copy your shit over using your knowledge of your data organization and native OS commands or tools.

Step 5. Run a checksum on your important stuff and store the hashes to verify everything is fine over time. Odd situations occur when backing up data. I have run into cases where i didnt realize the files i was about to backup were bad/corrupt until i saw the good copy on a backup drive i was about to incrementally overwrite.

Step 6. Store the shit somewhere else if you can reasonably do this and feel confident in the security of your data. If you have to start encrypting your crap, you add some more complexity that can effect the reliability of your restoration, but again if you proceduralize and keep up on it you will be fine.

Backup design and integrity is hard work and serious business when dealing with large volumes. It reminds me of the Seinfeld episode where he goes to the car rental place and they dont have his car and he goes into his "Anyone can take the ticket" diatribe. Anyone can back up their data. But can you get it back? I am not an expert in this area and dont pretend to be, i am just a seasoned IT administrator who has performed alot of backups in my day and have managed to keep most of my data safe over the years.

Keep it simple

[jampola]

# rsync -avz /this /that. Split your directories corresponding to the sizes of your drives. If on Linux, run smartctl -H /dev/sdX to check your disk health and if possible, take the HDD's our of their usb enclosures and connect them directly to SATA for faster xfer speeds. These drives will 9/10 mount just like a normal drive since usually they are just a normal drive housed in an enclosure.

Good luck :)

Re:ZFS

[jampola]

Problem with ZFS is that he'll need a shit ton of RAM. Once ZFS eats into swap, everything goes slowmo!

man tar

[flyingfsck]

Plug all the disks into a USB hub. Ensure that each one has a unique volume name eg bak1, bak2... The old skool way is to make a little tar script and use volume spanning. Otherwise, configure all the disks as a single JBOD and run DejaDup.

Linus did become ...

[zapyon]

a rich man by NOT patenting stuff (i.e. using the GPL2 for Linux). So why shouldn't he do the same with other stuff? Also, I guess there is loads of "prior art" regarding this cloudy PR talk of today.

Re:Linus did become ...

[CastrTroy]

I guess it depends on your definition of "rich". While he probably isn't hard up for cash, neither are most extremely competent software developers. Bill Gates on the other hand is undeniably rich, although he is probably not as good of a software developer as Linus Torvalds. I don't think that Linus wrote Linux to become rich. Linus probably has a lot of money, and he didn't patent stuff, but correlation is not causation. A smart guy like that could have probably been quite a lot richer if he had chosen the other side.

Fishy

[PopeRatzo]

"Hi there ! I'm looking for a simple solution to backup a big data set consisting of files between 3MB and 20GB, for a total of 24TB, onto multiple hard drives (usb, firewire, whatever)

Private Manning, is that you?

Damn!

[robbie73]

Damn, that's a lotta pr0n!

arj -xvf

[gl4ss]

was it that?

and to extract arj -va.

there, problem solved.

ZFS snapshots

[the_B0fh]

Why go through all that? Set up a ZFS volume, and snapshot it to another ZFS volume, and then offline that. Put it on a sata cage, and you can just take it with you when it is done.

One word: cpio

[SlashDev]

Check out cpio under Linux or many Unix flavor OS, cpio can span backups over multiple target media. Make sure to test backup AND just as important: restore.

LVM

[bastafidli]

LVM is another possibility. If you can get SATA drives and plug them all in, you can then create LVM volume spanning all the drives and just simply copy the data over to one large volume. LVM will take care how to span it across the drives.

Re:LVM

[Wolfrider]

--LVM is not RAID. Trust me on this, I speak from experience. LVM in a virtual machine can be nice. However, after having an entire Linux LVM fail IRL, I switched to Freebsd + ZFS and never looked back.

Disk Utility RAID in OS X

[Anonymous Coward]

I have actually had to do this in an OS X environment before. We have an xserve hosting up about 30TB of data in small files, and we are scheduled to move away from the system, but we need backups in the meantime. My solution for the short-term was to create a concatenated "RAID" of 35TB worth of external hard drives connected via firewire AND usb, (the external drives range from 6TB to 12TB), and use retrospect to back up to the resulting volume. There is no room for anything but an up-to-date backup, but it's getting the job done until we move to a large RAID with offsite backup.

Apple's software RAID as configured through Disk Utility is surprisingly versatile, and though my transfer speed is slow when the data hits a USB drive, it is entirely transparent to the software when switching between FW and USB. It is also fairly robust, because if there is a hardware failure on our server, we can take the disks, plug them into another mac, and the RAID configuration is maintained without any futzing around (as the config is listed on the beginning of each volume).

Now, before everyone goes apeshit on me for using a concatenated set instead of a RAID solution, there were a couple limiting factors in my decision to concatenate rather than RAID 5/0, the major one being the range of sizes of external drives that we have, and a lack of funds available to purchase more. OS X's software RAID goes by the lowest common denominator (6TB in my case), so I would lose ~1/2 to 1/3 of my space if I used ANY of the RAID options, and I didn't have any space to spare.

I feel your pain, and good luck.

Rar files?

[Agent0013]

I guess there could be issues with space while making the rar files, but they can break the archive up into chunks of any size you desire. You will need all of them accessible to unpack them again though. Perhaps it isn't the greatest solution, but it may do what the poster wants.

The Cloud?

[fongaboo]

Just put it in the cloud... *rimshot*

BackBlaze

[minijedimaster]

A cloud backup service released information on how they build their own disk based backup servers. Maybe something that would help with your endeavor? http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/

Online backup: USB 3 ... or thunderbolt.

[sonamchauhan]

Get a PC with 6 USB 3 ports, connect a powered, 4-port USB 3 hub to each PC USB port. Then connect 24 1TB external USB HDDs (or SSDs) to the hubs, format as necessary and run your backup software.

Thunderbolt may be higher performance and have daisy chaining capabilities. But the USB solution should work just fine.

Simple answer: don't.

[dandaman32]

I work for a data backup company as a dev monkey/admin/jack-of-all-trades.

Do you ever want to restore these backups? If the answer is "yes" (and it should be, otherwise why are you backing up in the first place...?), then you need to be guarded against failure of an individual disk. That means you need some sort of RAID solution.

For reference, Datto's 3U nodes store 20TB across 14 2TB drives, and the next larger size of node we have is somewhere around 55TB in 4U. No, I'm not trying to sell you our hardware (we only sell to resellers anyway) but hear me out. You really are going to save yourself some headache if you build a NAS device.

USB 2.0 is SLOW AS BALLS. I see our USB seed drives (HDDs we mail out to customers to get their initial datasets up into the ether) max out at 20-30MB/sec on a good day. By comparison, Gigabit Ethernet will give you 112MB/sec after NFS/TCP/Ethernet overhead -- much better. For this reason, and because it's just so impractical to handle large collections of failure-prone USB drives, our largest round trip drive that is shipped as USB is 4TB. After that, we actually ship our customers NAS devices (usually a returned/development box with a different OS image on it).

Go with NAS. You need the resilience against disk failure, you need the additional speed, and while yes, it's a greater investment, the alternative is utter agony when one of your 12 2TB disks takes a dump.

How many USB ports do you have?

[DarthVain]

I know you are likely trying to do this for a cheap alternative, but just don't. It is really an unworkable solution for that amount of data.

Some have mentioned Tape, which I know very little of. However I would simply build another RAID machine to copy to, or use a NAS if you can find one big enough, as it amounts to pretty much the same thing, but more specialized.

If this isn't sensitive data, another option might be to cloud it. Amazon and a few others have some competitive prices. The advantage here is you additionally get off site backup.

I guess one of the key factors in your decision will be how refreshed this 24TB of data is. WIll it only get occasional updates, or will a big chunk need to be backed up regularly. That is the other question, how often will you need to back up? Lastly, how quickly do you need recovery?

Re:ZFS

[repetty]

Problem with ZFS is that he'll need a shit ton of RAM. Once ZFS eats into swap, everything goes slowmo!

RAM is free, or nearly so.

JBOD Drive Array

[Brewster Jennings]

Sans Digital Makes an 8 slot drive enclosure with either a PCI-E or USB 3.0 interface for about 350 bucks. Put 8 3tb drives in it, run it JBOD. You can buy the cheap 3tb drives because you're going to run them JBOD. At 150 bucks a drive, Your total cost is about $1600.

You might be able to get Windows to do Incrementals to those drives, although I haven't tried it myself. And remember to run the enclosure sparingly, because non-enterprise drives aren't rated for the same number of spin-up hours.

Of course, it's not as safe as putting everything on a billion optical disks. But even using a BD-rom (at 46gig a pop), you're talking about 534 Blu-rays, and that's pretty much ridiculous, unless you have an intern you really dislike or something.

Data Deduplication

[Aphonia]

USB seems inane and insane for that level of data. How redundant is this 24 tb of data as well? Running it through a data de duplicator (possibly to reduce storage requirements depending on the type of data) and then a tape drive or raid array may be a cheaper and more time effective option.

Re:Just plain old rsync...

[ducman]

Don't use btrfs. My approach (I only have 8Tb) was two low-powered linux file servers. The "main" one was running btrfs over a mixed set of disks with a nightly rsync to the backup server. A power outage that was more than my UPS could handle resulted in a corrupted btrfs filesystem. After a couple days of trying to fix the btrfs filesystem, I gave up and restored from the backup server. Fortunately it was using LVM and ext4.

Now I have the "main" fileserver running LVM and ext4, and a backup server running FreeBSD with zfs. The two are in physically different locations, and I use rsync with --only-create-batch and a USB hard drive to move changes from the main server to the backup.

Not simple! Geez.

[whitelabrat]

"I'm looking for a simple solution to backup"

And USB drives are your idea of simple? Seriously? Please hand the lady your Admin card at the door when you leave.

For 24TB if you wan't to have a job after someone asks you to restore a chunk of that you'll want to insist on tape. Or perhaps a equally sized NAS or SAN array. USB? Hope your resume is up to date.

External RAID enclosure

[kimvette]

You buy one of these:

http://www.newegg.com/Product/Product.aspx?Item=N82E16816322007

populate it with 4GB drives and create two RAID5 (or one RAID6) array, then you've got 24 or 28 TB of backup space, without having to change drives or break up your backup into smaller chunks.

But really, your backup methodology is broken; you need to organize the data into manageable chunks because aside from a large dedicated backup server/SAN, there is no reliable (don't tell me tape is reliable) backup solution for a such a large quantity of data in a single chunk.

What I do for backups: in my 24-bay server I have eight large drives in a (HARDWARE) RAID5 array (were 4TB drives available at the time I'd have gone RAID6) and rsync the virtualized server contents to that, then archive them into tarballs, and send copies of them across the LAN to another server that is running (HARDWARE) RAID5 as well. Every once in a while I back up the critical data (source, scripts, financial data, production web sites, /etc, and so forth but not the program binaries nor system binaries which are easily recreated or reinstalled, respectively) to optical media and external hard drives.

So what I have in summary is:
* Massive server with a backup array separate from the production array
* Separate backup server running another array (again, using a quality HARDWARE RAID controller. Safeguard your data and don't bother with Intel, Adaptec, Promise, or Highpoint "hybrid" RAID)
* Periodic backups of non-recreatable data to USB drives and optical media that are moved off site.

Re:External RAID enclosure

[AmiMoJo]

Hardware RAID is a bad idea. If the card fails you need a compatible one. With software RAID any machine running the same OS can read it back.

Re:External RAID enclosure

[jamesskaar]

and i've heard that linux software raid outperforms many hardware raid varieties...

Windows 8 Storage Space

[Barlo_Mung_42]

Plug the drives in. Tell Win8 to treat them as one large drive. Good to go.

its possible, but risky

[Supp0rtLinux]

If you're willing to deal with the time it will take to write it all out, then its doable. You need a backup software that supports VTL (virtual tape library). With this, the physical drives are seen as tape devices. So it will start writing to drive #1 and when its full it will say "out of media" and it *should* pause for new media. You "eject" the drive, attach a fresh one, and hit continue. Then wash, rinse, repeat til complete. As others pointed out, it will take some time. You can speed it up with eSATA or USB 3. If you're on a Mac, you can speed it up using t-bolt. I believe Arkeia still offers a free version and they did/do support VTL. Haven't been current on free backup wares for a while. One thing to bear in mind as well once you write this 24Tb to a collection of media any single media failure will result in all data being unrecoverable. So you might opt for doubling your backup window and making a duplicate copy. Otherwise your best bet is to put all the drives in a NAS configuration (think FreeNAS) with a RAID6 structure, then have the backup s/w use this as its destination. You could do this with an 8 drive chassis of 8x4Tb SATA disks (2 lost for RAID6, leaves 6x4TB=24Tb raw). A similar idea could be accomplished with ZFS, but its future is somewhat unknown with Oracle these days. If you need longevity, I'd stick with a more open/compatibly filesystem. If you manage to setup it correctly and use exFAT, you could mount the backup volume to any current Linux, Windows, or Mac system and if the backup s/w runs on all platforms you'd have a lot more compatibility and recovery options.

Re:its possible, but risky

[Wolfrider]

--Dude - ZFS has been ported to FreeBSD, and has been running on that OS for some years now. And as ZFS is the best thing to come along since the invention of the hard drive, Oracle would have to be completely insane in the membrane to try and drop it.

Use a portable file system like ZFS

[sh3rp4]

we do much the same thing. we have a backup nas that we then rsync to a set of "offsite" drives.

My recommendation would be to investigate ZFS. (picture software raid and LVM rolled into one with filesystem encryption and compression built in.) Easy to compile and install on linux.

We created a pool for the offsite drives, then rsync to that file system. "Export" the file system and take the drives out. (Hot swap in trays, buy extra trays for rotation drives.) When you need to put in the next set just put them in and import it. Order and placement does not matter as long as enough drives are in. You could even have one or two parity drives in case a drive fails.

We have a cron job that rsyncs to the offsite drives, then exports them and emails the admins that it is ready for rotation. We keep 2 sets, one is in all the time and the other rotates offsite. You can swap on whatever schedule you are comfortable with. With compression, depending on data you could easily cut your drive requirements in half. Turn on encryption to keep your porn safe while in transit. All you need is a hot swap JBOD chassis. you could backup directly to the removable filespace, or do what we do, backup to a set local (local to datacenter, not to machine) filespace and rsync it over regularly.

It is something else to learn, etc. But it is a system that works well.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:ZFS

[sh3rp4]

I don't think 12 gig is a shit ton. if you are not doing dedupe ram usage is reasonable. and there is no reason to do dedupe on this.

skip the USB

[whitroth]

If you've got that much data, with a setup like that, you can afford to buy something better than USB. Consider eSATA, though I, personally, would push for a simple, fast backup server.

However you go, and I admit to not having read all the comments, you didn't mention how often the backups need to occur. Here, were we've got terabytes of data on many systems, we do a nightly rsync, and use hard links, which speeds it up and decreases space usage.

                mark

Re:JBOD or more accurately, spanned volume

[leptons]

"I mean, if i copied 200 gig across 3 drives in a jbod raid, could i plug just one drive in to access the information on another machine? Suppose my laptop only has 2 usb ports and i do not have a hub plus i'm running a different OS, does this mean i can't look for information on the set?"

This falls outside of what OP is requesting. He just wants to backup 24TB of data onto multiple USB drives.

USB can support up to 127 devices connected to a single host controller, so with a few hubs OP could connect all the drives he'd need for the back up all at once. I've run my own drives via external USB for a time, probably around 8TB of various sized drives using cheap USB-to-SATA adapters ($3.00 on ebay), and cheap 7-port USB hubs ($5.00 on ebay). It's not the fastest solution but it never gave me a problem. It was an experiment to see how many drives I could hook up with cheap Chinese parts. I had it running for a year before I started switching things over to USB3.

The only answer is

[herve_masson]

42

8 disk USB3 enclosure + 4TB drives

[ekimminau]

Buy 8 x 4TB drives + this enclosure
http://www.amazon.com/Mediasonic-H8R2-SU3S2-ProRaid-External-Enclosure/dp/B005GYDMYQ

compressible porn

[davidwr]

If it turns out that the source data is not porn (unlikely) and is highly compressible

Would dirty photos of his blow-up doll count as "compressible?"

I guess it depends on HIS definition of "richt":

[zapyon]

quoting Linus again: "First off, I'm actually perfectly well off. I live in a good-sized house, with a nice yard, with deer occasionally showing up and eating the roses (my wife likes the roses more, I like the deer more, so we don't really mind). I've got three kids, and I know I can pay for their education. What more do I need? The thing is, being a good programmer actually pays pretty well; being acknowledged as being world-class pays even better. I simply didn't need to start a commercial company. And it's just about the least interesting thing I can even imagine. I absolutely hate paperwork. I couldn't take care of employees if I tried. A company that I started would never have succeeded – it's simply not what I'm interested in! So instead, I have a very good life, doing something that I think is really interesting, and something that I think actually matters for people, not just me. And that makes me feel good." http://en.wikiquote.org/wiki/Linus_Torvalds

That should have been "rich" (n.t.)

[zapyon]

no text

Re:ZFS

[ixidor]

serious question: i am getting to the point where i will have 24 x 2tb drives connected. i have heard a rule of thumb you need 1 gb of ram per 1 tb of drive space, will i really need ~48 gb of ram? am considering a server grade board that can support 32gb ...

Re:Just plain old rsync...

[mark_osmd]

Your UPS wasn't set up correctly, it's supposed to signal the server to gracefully shutdown before the battery goes dry.

Build a USB array

[freshlimesoda]

Use FreeNAS to manage RAID on the array. And rsync. Yes, you may have to do some handywork yourself. GTG!

Another option

[sootman]

I know this has nothing to do with USB and maybe the OP has very good reasons for wanting it on USB. In any case...

Amazon S3 pricing:
First 1 TB / month: $0.125 per GB
Next 49 TB / month: $0.110 per GB

(1 x 0.125 + 23 x 0.11) * 12 = about $32 per year for 24 TB. That's a lot less than buying a bunch of hard drives.

Re:Another option

[Synesthes]

I think you might be off with your maths there.

1TB/month @ $0.125 per GB would be $128/month
Next 49 TB/month @ $0.110 per GB would be $lots/month

Re:Another option

[sootman]

Ugh. Everyone is right, I'm an idiot. I mid-read the whole thing. I got as far as "First TB" and stopped, so yeah, I'm way off. Never mind me.

I thought that sounded pretty low, but then I just thought "meh, maybe Amazon is just so huge they can do that."

Question about throughput

[TheLoneGundam]

"I'm just a simple caveman, ..." with a mainframe background, so I have a question of curiosity here At what point does the bandwidth/throughput of the DMA start limiting the performance of your backup? In my world, DMA for I/O is called a "channel". We have many, and while there are a lot of nuances we could discuss, basically we try to segregate the I/O for the input to backup (disk) and the output of backup (usually tape) , and have the backup task process in parallel as much as possible - my nightly backup, for example, runs 9 parallel tasks, 9 being the limit that this particular backup program has. I could run multiple instances of the program, but then I have to have mechanisms to make sure I don't back up the same disk twice between two concurrent executions; with one instance and 9 tasks I can just say 'back up everything that's online at the moment'. So, the throughput is limited by the performance of the slowest devices, multiplied by the parallelism we are able to achieve. In the PC / server environment, does the DMA limit the I/O capability?

Re:Question about throughput

[amorsen]

At what point does the bandwidth/throughput of the DMA start limiting the performance of your backup?

With USB2? Never, it does PIO. The CPU gets to babysit the transfer. That is very much the exception in the PC world of today, every other bus in use today offloads nicely.

In general, storage buses on PC's are rarely fast enough to saturate system buses, and therefore DMA does not limit throughput. If you are really lucky you can get perhaps 2GBps through a fast PC, but you can probably copy memory around at 10 times that speed. So just like the mainframe, it will be limited by the performance of the slowest devices multiplied by parallelism.

Backup Exec

[Vrtigo1]

Backup Exec does exactly what you are asking for. Free 30 day trial.

MHDDFS FUSE module

[n3xu5]

I ran across a FUSE module (mhddfs) that seemed relevant when I wanted to combine several USB drives into a single file system. My main goal was to make each drive usable independently for file recovery if I had to move it to another system.

The module appears to be a fairly thin wrapper over an existing file system. It only appears to choose which of the sub-file systems to write new data to, automatically writing files to whichever drive has the most space. This provides nothing in the way of redundancy, however.

What is nice is that you can easily access the files on a drive without needing the other drives. May be helpful for someone.

http://romanrm.ru/en/mhddfs

Skip usb

[Outtascope]

Honestly, I know it isn't your question, but skip USB. Too slow. WAY too expensive. Get yourself a rocket raid card or similar, a sas expander, and an 8+ trayless disk enclosure. I use a 12 disk enclosure (8 for regular backups, 4 for all the one off stuff I do) with 2TB drives. I wrote a program in Java using NIO that stripes the backups across the disks so that it can saturate the bus. A solution like this will ultimately be faster and cheaper. One day I will port the code to native as the Java program was just a proof of concept that has worked so well I haven't gotten around to it. This setup works exponentially better than the VXA-3 tape backup we were using before, and couldn't imagine having to do it with usb drives, either from a cost or a logistics perspective.

Re:No offense meant, but ...

[Outtascope]

That's a load of crap brought to you by the people who would rather that you pay 10 times as much for 1/10th the performance and 1/10th the capacity.

But let me rebut that in a more logical fashion. Tapes take considerably longer, meaning the backup strategy ends up backing up less than is optimal.

Sure, disk backups are fragile. But if you system is going to be borked by the failure of a backup volume or two, then I would posit that your backup strategy is a disaster waiting to happen regardless of the media that you use.

USB is trouble

[EricScott]

USB will copy files, but not identical copies. Firewire is better.

But the best/cheapest solution, is a Dell MD-1000. It will take 2tb generic drives.

Re:USB is trouble

[gl4ss]

are you implying that usb inherently corrupts files?

nfs over lvm2?

[jamesskaar]

if you build a small system, cheapish, an itx with 6sata, each connected to a port multiplier, os on flash... you could have 30tb from 1tb drives. set it all up as an lvm2 volume, then you can slap the drives back in a new system any way you want, and they'll come back up in the right order. rsync(backupmypc) will keep the backup in good condition, you'd of course need spares, in case the verify shows a drive fail, a duplicate system. using linux raid to turn the two nfs mounts into a raid 1 array would be nice, but parity would be better, yet it'd take even more drives. yah, bigger drives would be a good first start.

should be modded up

[gl4ss]

this sounds exactly what the guy is looking for..

The Easy Way

[jep305]

Step 1: Buy yourself something like this: http://www.aberdeeninc.com/abcatg/Stirling-X339.htm
Step 2: Install it
Step 3: rsync
Step 4: Go do something else -- this is going to take a while

Well not too hard...

[niftymitch]

At $35 each you get a dozen Raspberry Pi's.

While not fast you have a USB port and can connect them
via ethernet and ssh and start tinkering.

A good USB hub can turn one USB to four
The local Costco has 3TB USB disks. Yes
you have to organize your data into 2.8TB chunks
or so with some script foo but rsync can help
verify the bits.

N.B. this is 10/100 ethernet not GigE and USB2 (at best)
and they share a single USB link to an onboard USB hub.

But you could automate the thing and not have to
swap out USB cables for a week.

MD5 checksums and an index...

Let us know how it goes. ;-)

No matter what you do you will have to do some
scripting. Do label each of the USB disks
(physical and logical names that match).

Did someone way that this was a marginal
idea? Backing up to USB has some value but does not
sound magical and error free.

Since 24TB is a lot of junk -- good luck
but with the crazy big USB disks -- what the hey.

DLT?

[HArchH]

Shouldn't you be looking at DLT devices for this kind of dat set size?

Network attached removable harddrive units

[BagOBones]

http://www.high-rely.com/

We ran some of these for off siting data in rotation... Way faster than tape and designed for swapping... Might not be the best for long term storage.

Re:JAFM

[voltorb]

I take it you have never logged into CERN.

It's nice

[aglider]

to see someone here knows this.