Slashdot Mirror
Affordable Home Backups for 10-100G Systems?
MichaelJames asks: "Ok, I have my MP3's streaming, all our digital pictures up, and a file server running on one machine in the basement. What would be the best way to do simple backups of the system and data? Get a tape drive Get a CDRW or DVDRW to backup the MP3 and pics, but use the old Zip drive for the file server data?" With drives in the 10-20 gig range only getting smaller and less expensive, what are we to do for backups, that have yet to scale well in the same range. For home systems with up to 100G of storage, what do you use to back up that much data, with a solution that's affordable to the average computer user? Have DVD writers become cheap enough for serious consideration as a backup media?
DVD media is about $6 per 4.7GB disk now, but do you really want to use 20 pieces of media to back up a 100 GB disk?
One thing some people do is back up their HD to a second HD.
Zip disks seem practically useless these days--recordable CD is just too cheap and universal by comparison.
Tape drives are the high-end solution, but expensive.
I'd get a cheap raid card, set up raid 1 simple mirroring and poof... some backup but not off site...
I have a 100BaseT network, and a server computer that resides in a different room from the rest of my systems. I rotate backups using those aluminum drive caddies. A pair of 60G drives turned out to be MUCH cheaper than the equivalent size tape backup. Every day, I rotate out the drive at the end of the day, and swap with the other. The spare I keep in a fireproof safe. Just tarball the appropriate directories. Done. Poof. Much faster than the average DDS3 tape drive too. Runs at night and I don't even notice it.
Actually, what I do is make the new (largest one I own) drive the backup drive, put the old backup drive into use as the primary drive, and retire the smallest one. Just make sure the new drive is as large as the others added together.
CD-R's are OK, but why bother with the hassle? Just run a cron job to copy the files every evening/hour/whatever.
If all this should have a reason, we would be the last to know.
Onstream 30 or 50 GB ADR Tape backup.
Pros:
Can be found for under $100
Linux Support!
Cons:
Tapes are expensive
Ultrium tapes can backup 100GB Native but the price tag is way out of line for small buisness or home use (5000$+ a 100GB drive, ouch).. same goes with any dataloading systems... The only cheap tape backup I've found that was giving the best storage/price (aside from buying those used DSS 4/8 gig drives) is those 33GB Native VXA drives.
--- Metamoderating abusive downgraders since my 300th post.
This will also work for laptops. I recently purchased an external drive enclosure with a PCMCIA connector (also available in Firewire and USB), and a separate 3.5" hard drive. The cost of the two together was less than these external drives they advertise for backing up your notebook, plus I can reuse the drive enclosure for any 3.5" hard drive.
The drive enclosure was a bit more expensive than the rack mentioned above (under $100 with shipping) however it did come with a two sets of enclosures - one for the drive itself to use externally, and another to put in a PC cabinet if you want to hot swap the drive with it instead of using the card slot.
I'm not a journalist, but I play one on slashdot
I did exactly just that for a small office that I consult for. It backs up the entire server hard drive, and all critical information on all networked workstations, such as e-mail, etc... And for the backup software, I simply used Powerquest's DataKeeper software, that happened to come free on a driver disk for a Firewire hard drive bay I bought on a whim and never used. Compared to the tape back-up and other options, this was a no-brainer, both in the price and ease of use departments. Running the backup was reduced to a one-click wonder app, perfect for the "mature" staff in this office. ;-)
Does anyone have a better bang/VS/buck solution? I know CDRW's are quite cheap, but one has to factor in speed, as well as human intervention. Swapping disks requires attention, which requires a wage. The backup system simply is launched every friday after work, and does it's thing on it's own time, without need for people. And with 100GB, no sanely priced tape drive comes close.
Get norton ghost, get a hard drive to fit your size.
Once a week, remote dump your house to a central backupserver. No only does it offer compression, its fast and very easy.
There are several programs allowing to use your DV camcorder for backups. For example:
http://dvbackup.sourceforge.net
Given that MP3's and digital pictures don't change with time like databases, documents etc, why not do what we do and back the whole lot up once. Then as you download new MP3's, pictures, whatever. Put them into a directory say Not_Backed_Up, and burn that to CD also when you have enough to fill a CD. Then migrate this with the rest of the data and start fresh with the next CD to be filled.
FWIW if you have the money to buy 2 identical drives RAID 1 might be the way to go.
Agreed that just backing up to another HD provides the best overall method for creating a complete backup of 100MB of disk storage.
However, I would suspect that most users don't change a huge percentage of their HD's content on a daily basis, unless you are routinely d/l'ing or ripping MP3s and MPGs on a daily basis (and I note that when I do generate that kind of traffic, it is usually because I am making a compilation CD, and while this does generate a few GB of "new" files on my HD that day, that data doesn't need to be backup up because I've got the original CDs anyway).
As a result, it seems to me that a reasonable solution is to create a "baseline" backup, say to a CD or DVD, at system install time, when there is (relatively) little on the disk, and then each day (or week, depending on needs), do an incremental backup of changed data only to another CD.
This approach is obviously quite inefficient if you have a complete HD failure, in that you have to recreate a new drive by starting with the first backup CD and then restore EACH ONE thereafter until the final CD restores the disk to it's last backed-up state, but for a more common problem of losing or corrupting an individual file, since that is more likely to happen with a recently modified than a remotely modified file, you are likely to be able to restore a last good version within only a few CD's of the most recent incremental backups.
A lot of people have mentioned that disk to disk backup seems to be the best way to go.
I agree.
What hasn't been mentioned is rsync, which makes disk to (local or remote) disk backups fast and easy.
It is trival to set up a second disk that is a "stale" mirror of your primary disk(s) that backs up nightly, and will boot off a floppy. This captures some of the advantage of RAID (quick recovery) while being an actual backup, not just fault tolerance.
Rsync can use ssh as a transport, so you can securely back up remote disks as well.
-Peter
RAID offers good protection for some things: hardware failure (ie: HD crash) and uptime. That aren't the only woes, however... You can loose data in a lot of ways:
Disaster (fire, quake, flood, nuff said)
Hardware failure (disk, controller, ...)
OS failure (FS corruption, ...)
Application failure (User space applications malbehaving, virii, ...)
User failure (accidental deletes, experimental children - trust me on this one ;-)
:)
...) are a plus, but more cumbersome.
RAID will protect you from the second, but will happily add nothing in case of any of the other failures. Backing up to another media is a necessity.
Adding an extra disk (or two, or three), and some tar/cpio cronjobs will add basic protection. (No disaster recovery for you, unless it's off-site
Removable harddrives (firewire, frames,
Tape is considered a more 'trustworthy' backup medium because the mechanism and data storage are separated (ie: tape drive / tape), while in a HD it's in one single package, and it's not as easy to replace the logic board/stepper motor if this flunks. With tape it's easier: just get a new tapedrive.
Anyhow: don't rely on RAID to save your data - it won't.
Okay... I'll do the stupid things first, then you shy people follow.
[Zappa]
RAID using rsync from my other computers, (or just drag-and-drop to the Samba server from the Win2K box).
Cygwin has RSync for Windows. Works really well.
Moneyed corporations, non-working 'poor' and criminal prisoners are turning productive citizens into tax-slaves.
Backing up your hard drives onto other hard drives sounds like a good idea (price, amount of storage, etc.).
If you a backing up things that would be inconvenient to loose but not mission critical then this method may be fine.
If your are backing up irreplacible data then I would suggest using a different method.
I have experienced the meltdown of two systems where in both cases it was the power supply that failed. Unfortunately, the when the supplies failed they took out pretty much every component in the systems (including the hard drives). I can only guess at the voltage that was pumped through everything but between the two systems I found 5 chips that were cracked or blown apart (1 of them on a hard drive controller board)!
These incidents happened at different times, in different locations, on different types of computers.
If you do use hard drives for backup consider removable hard drive bays and don't leave the back up hard drives in your system.
Just my 0.02.
--
Can't be bothered to login.
There are two ways you can go relatively cheaply, and IMHO a far better solution than CD-R or CD-RW.
Pick up a DLT2000XT (15gb native) off ebay for about $200. Tapes are dirt cheap, about $5/ea and the media is extremely durable, nearly indestructible.
Pick up a DLT1 (40gb native) off ebay, about $500. Tapes are moderately expensive at around $20/ea, but again the media is extremely durable.
DLT is industrial strength backup, the drives are built like tanks and the tapes can take incredible abuse.
Its all standard SCSI and works great with linux, no problems whatsoever.
I considered buying hard drives for backups, but they are far too fragile for long term backup and off-site storage. Most drives arent designed to be spun up and down lots of times either.
Last thing you need is for your backup harddisk to go splat when youre trying to power it up to restore your main system from a data loss.
With DLT, this isnt likely to happen.
Don't rely on MTBF scores, they're not too reliable.
That's actually pretty funny - the reliability metric isn't reliable.
I agree, don't trust hard drives. They're a bad solution for backups, as they don't store as well as real backup media. However, RAID systems are also not suitable because you can't archive data in case your system burns, and it doesn't protect you from filesystem corruption or inadvertent "rm -rf" commands.
They're steep in price, but DLT or AIT are optimal. The newer models are capable of 100GB, except you can't get that much on there if the data is uncompressible (like MP3s are). If you wanted to be able to back up less than 100GB, you could probably get an older DLT/AIT for a grand or less. I've seen 8 or 10-slot AIT changers for less than $3kUS, a config which can store almost a terabyte. If you can afford that kind of cash, you probably won't have to worry about backups for a long time to come.
I have had bad luck trying to backups with
Dantz Retrospect or Veritas to a yamaha cdrw, to a sony cdrw, and to a creative cdrw. IMHO, the firmware in these drives seems to hang if there is a media problem, and dantz just acts like its waiting. Veritas blue screens.
If you are going to try this, use cdrw's or you may end up with 35 coasters.
On the other hand, I shelled out $140 for Dantz because it supported my Onstream drive. This works the best, but sometimes the $250 computer i dedicated to doing backups hangs, expecially when I back up the full 120 gb.
You may want to look at this paper by Sanders et. al. titled Redundant arrays of ide drives available on xxx.lanl.gov under hep-ex/0112003. They report tests of raid-5 under linux for backup of physics data and find it price comparable to tape at the 10^12 byte level
dvbackup is a utility that lets you store up to 13gb of data on commodity miniDV tapes. With the use of a Digital Camcorder.
:(.
This is not exactly a dirt cheap solution, but if you have/want a digital camcorder anyway, there's only the cost of extra tapes.
Make sure the camcorder works with dvbackup before buying one though. It doesn't work with any JVC's that I know of, or at least not mine
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-U
If your data is fairly static and you just need to mirror it, buy an extra hard drive and run a dump periodically.
If your data changes quite often, and you need the ability to do "point in time" recoveries of particular pieces of data, try Amanda.
Tapes are 113$US (shipped) for a 4 pack.
For those of you following along at home, that is ~300$US for 120GB of backup.
400$US for two sets of 120GB.
Not bad, and it is a DAMN FAST drive...
______
Once: you're a philosopher. Twice: a pervert.
I have over 100 GB of hard storage, however most of that is mp3s, videos, and other things that do not change. I have burned on CD all of my mp3s and other things that don't change, but I don't want to live without if my hard drive goes down. Then every night I have a script that does an incremental backup and zips it, PGPs it, and then ftps it to my friends hard drive, both at college with a fast connection and a 100 miles apart. Also once a week I do a full backup. I figure the chance of both of are computers getting struck by lighting, bruning in a fire, or failing at the same time is pretty rare.
Its not the best of all solutions, but I don't have anything I would die without and it was really cheap (free).
Some thoughts:-
1) IME, Racks cause heat build-up and kill HDDs. I won't use them and I recommend others don't either.
2) If you're going to permanently mount and power the drive, then it stands as much chance of dying through power cycles and usage as your main drive.
3) If you're going to leave the drive unpowered most of the time, then you probably won't bother backing up because it'll require a shutdown, an opened case, a boot and another shutdown.
IMHO, If you're going to use another drive, you might as well go for RAID and have the backups done automatically and continuously.
--
You say it's unlikely, and it is. It happens, though. It happened to me. I was working in a bulding that got torched after a squatter got evicted. The entire building is gone (99 Moody St. for those of you in Waltham, MA). We saved a ton of stuff *only* because a firefighter was fortunate enough to find a machine w/ an undamaged HDD. It happens.
3ware 6800 Escalade IDE RAID is working great, and it's only $350 or so.
It's a PCI card with *8* IDE drive slots, which you can configure in a RAID 5 array for huge, failsafe backups.
I've got 8 60-gig IDE drives on it, in a RAID 5 array. Gives aboout 420 gigs. Shows up as a SCSI device in OpenBSD. Works great with Linux.
Churning away wonderfully.
I've backed-up 200 gigs of files on it so far.
420 gigs of RAID5 storage = $1100 USD. ($300 for the card. $800 for 8 60-gig drives.)
Here's my post on the OpenBSD list about it
Since when is a harddrive not a semipermanent media that can be easily taken off site? I'm surprised this comment got modded up so high. And since when are tapes such a reliable media compared to a hard disk? So burn-in the drive for a few days before using it for backups. And use a S.M.A.R.T. utility to diagnose the drive before each backup to reduce the chance that something is getting ready to fail.
Your best option is to put all data on a 2-disk mirrored RAID and use another drive as a removable for an off-site or fire-safe backup. The probability of 3 hard disks failing simultaneously, one not in use, is so incredibly small it's laughable. And for that non-zero chance, if it happens, you can pay to have the spindle of one of the failed drives transferred to a new drive in a clean room.
Advantages:
- Standardized TCP-based protocol, uses authentication.
- Lots of free "backup software" available - e.g. pan, newspost, tin...
- Centralized backups over the network! Only one server directory to back up.
- Backups are automatically dated by the server, and can be signed and even encrypted if you integrate pgp into your posting script.
Disadvantages:It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
You know, the funny thing about /usr/share is it is supposed to be where you put "data" which can be shared by users, right?
No. /usr/share and /usr/local/share are for files installed by an application and shared between the different binary architectures the application can be compiled for (such as map files for a game). To share data among users, use /home/johndoe/pub, or use groups.
Will I retire or break 10K?
Anyway I think there should be a standard place for stuff which will be shared between users (everything from documentation to icons to MP3s to programs which users have compiled themselves)
/usr/local: /usr was for stuff that came from the vendor, /usr/local/ for stuff that the local site added to it. At the site I learned Unix /usr/local was often a separate partition from /usr to make it easier to share/duplicate/backup.
I believe this was the original purpose of
-- Sigs are for losers
This may not be the best way to do it, but it works for me...
I have a "backup" hard drive in my server. This drive is always unmounted so that there is no chance of filesystem corruption from the operating system.
I just use a crontab to run a simple script that mounts the drive and coppies whatever specified backup files to it, then unmounts it. The same method slightly modified could be used to back up this same backup disk to another location on the network on regular intervals.
You might want to look into an Iomega Peerless. The disks are pretty small (maybe about 5"x3"x.5") and the disks are 20 or 40 gb a piece. I'm running one a Windows 98 machine, I couldn't tell you about Linux compatability. It connects to a USB hub and has sustained data transfer of 12 mg a second, I think.
Tim ODonnell (trying to be the most
rdiff-backup and RAID it, dude...
http://www.stanford.edu/~bescoto/rdiff-backup/
From the web page:
What is it?
rdiff-backup is a script that backs up one directory to another. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, symlinks, special files, permissions, uid/gid ownership (if it is running as root), and modification times. Finally, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted.
Promise IDE RAID controllers saved my live more than once. Three 100GB drives in a RAID 5 config still leaves with you just under 200GB and a safety net.
It occured to me nobody mentioned online solutions such as Streamload or MyPlay (great for mp3 storage)..
:(
Too bad iDrive & Freespace.com went offline
I've got all the hardware, but I can't seem to find a piece of software that's built for incremental backup to non-tape media. The closest I've found is NTI BackupNOW, but after much frustration I discovered that even their software won't support DVD-R(W) for some months to come.
Has anyone actually **SUCCEEDED** in setting up such a backup system? Bonus question - has anyone had to restore data from this kind of setup?
Of course you'll still need something like 150 CDRs to get everything, inceremental or not - and that's just the first dump that will keep growing.
BTW, to the other replies to this - one of the companies I worked for, we did full, weekly backups of all the important systems (ie the actual OS and software - not the data) like the domain controller, database servers, fileservers, firewall, the backup servers themselves, etc. and then daily incrementals on them.
For the actual data on the file and database server, plus everyone's personal workstations (something like 100 people) it was a full backup once a month (maybe less) and nightly incrementals, which worked great because here people usually wanted just a couple of files their dumb asses deleted a week ago (usually perfectly timed to want the tape that just went off to off site storage :) ) All in all that achieved very good granularity. Of course the incremental tapes were recycled after about two months or so.
The database servers were a bit more tricky so additional full dumps were done manually, when, let's say, 500,000 new compounds were imported. There were also databases attached to data collection robots that had their own separate system going.
All in all, this took three backup servers (one for the data collection databases, one for individuals PCs and one for the other stuff) with their respective tape loaders, plus individual tape drives for the main database servers (we also had things like SGI workstations, but the scintists on those did their own thing), and the full backups did take up almost the entire weekend. But it was setup well enough that it took only a few minutes a week to maintain the whole thing - get a new tape, write on it what the server tells you to, put it in the loader, take the specified one out, send it off - you are done. And it did work - they had to recover after a fire once (before my time) and did it without any problems.
Anyway, the point of the story was this - it's not as simple as "do incrementals" or "no! do full backups" when you are talking about actual companies, it's a bit more tricky than that.
sic transit gloria mundi
This has been my choice for low-cost backup solutions for a couple of years now.
The drives support three different flavours of media - 12GB, 20GB and 33GB, and come with IDE, SCSI, or Firewire interfaces. The IDE is cheapest, at $699, with the media costs being $80, $45, or $35 depending on the capacity.
Is it the absoulte top of the line as far as tapes go? No. But the cost can't be beat. And you get a reasonably fast (3MB/sec) drive with very nice reliability (take a look at the independent testing on their site; e.g. soaking a tape in hot coffee for a minute, rinsing it, drying it and reading the data off.)
They also recently merged with Exabyte, who will be positioning it as their new value solution. Hopefully the Exabyte name will expand the market enough to drive the prices down on these even further.
Matt
This is what I do (man levels of redundancy):
1) Use resierfs -- it'd stable now and recovers better then ext2 in small "incidents"
2) RAID 5 array
3) Run a nightly script that hard links all the files into a . (hidden) directory -- protects against rm -rf''s
4) Run mirror in every directory from cron-- if you lose or mangle one file you cna recontruct it from the contents of the mirror and the other files (works a lot like XOR in raid 5 arrays -- aka a RAID for files).
5) DDS3 incremental back-up; complete backup at regular intervals -- one set of tapes stored off site.
This protects you from multiple levels of failures -- with the catastrophic redundancy being the tapes. You don't always want to rely 100% on the tapes for all your redundancy.
You can get a DLT drive that will do 20-40G per tape for under $200 on Ebay. Add an external case if it doesn't have one, and grab a SCSI card. Cheap, reliable, tested.
Don't listen to him. Tapes always let you down!
Some of the problems with tape are:---
Granularity -- i.e. you accidentally deleted "xmascard.lst" how do you know which tape it was on and how long will it take to scan several tapes to find your 2K file?
How can you be sure the tape is readable. In my experience something like 5% of backup tapes cannot be read by any drive except the one that wrote the tape.
No standard format, no backward compatabilty, short market life for thevarioustecnoligies. You may be very smug that you have an "acme" format backup stored in a safe deposit box, but after your house is flooded (mentioning the f*r* word upsets some people) you discover "acme" drives are no longer sold anymore, you just have to hope someone is trying to unload a working drive on e-bay.
Expense -- the drives, the tapes etc. will all cost several times as much as a couple of external hard drives.
Old COBOL programmers never die. They just code in C.