Slashdot Mirror
Affordable Home Backups for 10-100G Systems?
MichaelJames asks: "Ok, I have my MP3's streaming, all our digital pictures up, and a file server running on one machine in the basement. What would be the best way to do simple backups of the system and data? Get a tape drive Get a CDRW or DVDRW to backup the MP3 and pics, but use the old Zip drive for the file server data?" With drives in the 10-20 gig range only getting smaller and less expensive, what are we to do for backups, that have yet to scale well in the same range. For home systems with up to 100G of storage, what do you use to back up that much data, with a solution that's affordable to the average computer user? Have DVD writers become cheap enough for serious consideration as a backup media?
Given that a 100G hard drive is cheaper than any removable media solution, why not just buy another hard drive and install it in a removable (not hot-swappable, just removable) rack?
Racks are $20 at my local Fry's, and inserts for other hard drives are $10.
DVD media is about $6 per 4.7GB disk now, but do you really want to use 20 pieces of media to back up a 100 GB disk?
One thing some people do is back up their HD to a second HD.
Zip disks seem practically useless these days--recordable CD is just too cheap and universal by comparison.
Tape drives are the high-end solution, but expensive.
Just get a lot (A LOT) of 1.44MB floppy disks...
----
WWJD...For a Klondike Bar?
One more thing...First Post! :-)
20 January 2017: the End of an Error.
I'd get a cheap raid card, set up raid 1 simple mirroring and poof... some backup but not off site...
Seems to me that, anymore, the only cheap solution that's got any speed to it is more hard drives. OF course, if you're looking to do weekly's or something then it's no fun changing out a whole drive for it every week, but then it's not THAT much work either...
since drive are so cheap, maybe systems should start coming with a standard raid set up?
also you can by many tape solutions.
The Kruger Dunning explains most post on
Borrow a copy of Veritas Netbackup Datacenter from work. Get a tape drive and some tapes on ebay for cheap.
You can get inexpensive (ide) raid motherboards and a couple of big discs. Cheaper than a dvdrw solution (although not as flexible and neato)
Hey, I've got all these 5 1/4" floppy disks sitting in boxes in a back closet. I bet if I added them all up, they would amount to close to 100 GB.
I don't really think CDs and DVDs really aren't big enough for regular backup of large sets of files. It's just too inconvenient to have to setup a bunch of different 5GB backups, one per DVD (or swap DVDs). The only convenient solutions are to do what the first poster said: use a second harddrive, they're relatively cheap. Or buy a tape drive to store the backups.
Personally, I backup to a second harddrive.
have "become cheap enough for serious consideration as a backup media". (I know, technically the '.' doesn't belong there. I'm putting it there, deal with it.)
CD-R/CD-RW are too slow and too small, plan on spending a day or so swapping disks. You can always mirror to another hard drive, get a basic RAID card or just use a Ghost-like program to do manual backups. But tape is still cheaper per megabyte and more reliable. Sure, you can damage a tape, but it's harder to do than with a hard drive. SCSI tape drives are more expensive than another drive, but fast enough, and allow you to keep multiple versions or copies of your backup. Try that with hard drives and you need arrays. Tape starts looking REAL cheap then.
Ignorance is the root of all evil.
I have a 100BaseT network, and a server computer that resides in a different room from the rest of my systems. I rotate backups using those aluminum drive caddies. A pair of 60G drives turned out to be MUCH cheaper than the equivalent size tape backup. Every day, I rotate out the drive at the end of the day, and swap with the other. The spare I keep in a fireproof safe. Just tarball the appropriate directories. Done. Poof. Much faster than the average DDS3 tape drive too. Runs at night and I don't even notice it.
features on motherboards, just incorporate a mirror or RAID5 array for your data as a means of "backup" with an added increase in performance!
I have to say that this is coming from someone with a total of around 280gig at the house, but...
Out of 100gig, how much do you really NEED to back up?
The vast majority of my space is taken up by MP3s (where I converted my CD collection), but that could easily be replaced. To tell you the truth, of the things that I would need (documents, pictures, etc), I could easily fit it all onto a CDR. Well, maybe two. (I take lots of pictures)
Basically it boils down to, do you really need to shell out the money for that extra drive?
:^)
'Life is like a spoonful of Drain-O, it feels good on the way down but leaves you feeling hollow inside'
Actually, what I do is make the new (largest one I own) drive the backup drive, put the old backup drive into use as the primary drive, and retire the smallest one. Just make sure the new drive is as large as the others added together.
CD-R's are OK, but why bother with the hassle? Just run a cron job to copy the files every evening/hour/whatever.
If all this should have a reason, we would be the last to know.
Onstream 30 or 50 GB ADR Tape backup.
Pros:
Can be found for under $100
Linux Support!
Cons:
Tapes are expensive
I've encountered the same conundrum when faced with backing up my machines... Removable media drives seem to be not that cheap. I've actually acquired a backup 80 GB. ATA hard drive, which I use to copy off any important data (.mp3's, video files, etc.) With the large size of DV files (pre-Final Cut 3,) and the amount of footage I like to archive, I've considered purchasing an external firewire drive: Problem is, not all machines have firewire (like my machine @ work that needs backing up, too!)
I got an external usb drive for $248 for an 80G unit. It's the cheapest, fastest, least hassle way of backing up your data. Yeah, of course, you can buy internal drives at a much lower cost but I also needed a way of carting the data to other machines easily. Even though I don't have a USB2 interface, backing up my 40G or so was complete overnight. I'm sure if I had usb2, it would have only taken 2 or 3 hours. Sure as hell beats tape. Tape is a thing of the past. It's NOT cheap.
Since most people here have another box somewhere on the network, the answer is simple. A hard drive. Set up an old machine as a linux box and backup over the network. Even schedule it. Some motherboards will power on at a certain time of day so schedule the thing to power on in the middle of the night, suck your data over, then turn itself back off. Perfect off-line storage, nice and quick, nice and automatic. Or, find an old DLT drive or something. Used tapes, while often still good, are especially cheap. Use the same system... Automated remote backups. You'll just have to go tape switching every time you do a full.
-Steve
Ultrium tapes can backup 100GB Native but the price tag is way out of line for small buisness or home use (5000$+ a 100GB drive, ouch).. same goes with any dataloading systems... The only cheap tape backup I've found that was giving the best storage/price (aside from buying those used DSS 4/8 gig drives) is those 33GB Native VXA drives.
--- Metamoderating abusive downgraders since my 300th post.
First, on a typical system, not all data is really worth backing up; the OS and all applications can be reinstalled in the event of a crash (for Linux, it might even be slightly beneficial, as you'll reinstall newer versions and get rid of various cruft you've forgotten you have). Some data has been saved just because it's convenient or simply less bother than having to actively remove it (for me, I tend to collect old logs, various mails I never will look at again and documentation that's several revisions old). A lot of mp3:s and movies may already be burned onto CD:s. That filled 40Gb drive may actually 'only' contain 4-5 Gb of data that actually needs to be backed up.
The data I actually need to back up I manage by having the important stuff an specified directories, then mirroring them over the net to my machine at work. By doing it incrementally, there is little time or bandwith wasted.
/Janne
Trust the Computer. The Computer is your friend.
I haven't heard much about the quality or reliability of these drives, but the price sure is right. Media is big and cheap and the speed seems OK too. If anyone has experience with these drives please post it.
Oh shit! I forgot to click "Post Anonymously"...
Another solution might be to pair down what you backup. It isn't strictly necessary to backup everything on your hard drive. OSes and programs can be reinstalled, but the data that you create with them is the more precious commodity. Of course, figuring out exactly what you need to backup is the problem, and you still lose some information that isn't easily backed up or restored (like settings for programs in the Windows registry). It's taken several reinstalls of Windows before I figured out exactly what I needed to save...
To minimize your storage costs, try using LZip: Lossy Compression. Sure, you won't be able to restore your system to EXACTLY the same state, but you can compress your files to as little as 0% of their original size!
I have several Machines at home and instead of using one as a file server i use one machine for mp3s, one for Ware^H^H^H^H Applications, one for videos.... etc. Each of those drives are mirrored. That way hey if everything gets scewed up on a machine including the backup I dont lose everything. I know you guys have at least 3-5 machines at home.
"All I can tell the "lesser of two evils" folks is that if they keep voting for evil, they'll keep getting evil."-Lp.org
Perhaps you should consider the Torvalds method? =)
"Backups are for wimps. Real men upload their data to an FTP site and have everyone else mirror it." -- Linus Torvalds
I think not: (prices in Aussie $ and are guestimates)
100GB HD - $500 (real rough guess, however I saw a 80G for $449 the other day)
-------
$5/GB ongoing
Burner - $150 -> $300 (depends on the quality)
100G of CDs - 156 cds * $1 = 156 (prolly cheaper in bulk)
-------
max outlay $300
$1.56/GB ongoing
The strange thing is that I was thinking about this the other day when contemplating buying a new HD and how I was going to back it up.
I went through this dilemma last month, and settled on the 30/60 GB IDE tape drive from Onstream. I ordered it last week and can't wait to crack it open! Tech support of the firm who wrote the software that comes with it (Yosemite Technologies -- tapeware.com) says it will run over mapped drives via samba from a PC or via smbmount/smbfs from Linux...
Get norton ghost, get a hard drive to fit your size.
Once a week, remote dump your house to a central backupserver. No only does it offer compression, its fast and very easy.
Until all the problems with them are worked out (and Linux drivers are widely avalible, of course :-) ), I would recommend staying away from them. Untested Technology == Bad Nasty Crap
Everything is mainstream now.
Why are you bothering to back up your data?
/` or a virus)? If so, then a RAID solution is useless.
That may seem like a stupid question, but you need to consider the reasons you want to have a backup before you settle on a method.
Are you afraid of your drive failing? If so, then using a RAID solution should cover you.
Are you afraid of losing your whole system (perhaps due to lightning or theft)? If so, then your backup must be kept physically isolated from your system.
Are you afraid of accidentally deleting files (such as `rm -rf
Are you afraid of having your system down for an hour or two while you replace a drive? If so, then regardless of other issues, you need a RAID setup.
Do you want to use your MP3s with some other device? If so, you probably want CD-R copies.
Of course, there are other considerations that I haven't mentioned or thought of.
The cheapest backup media now is more hard drives. Really!
So I got 4 x 60 GB Maxtors (cheapest MB/$ ratio when I purchased), two Promise Ultra TX2-100 controllers, and set up a 180 GB software RAID-5 under Linux. I'm running ReiserFS on it, so I don't have to fsck 180 GB if I crash the system running an experimental kernel, and I keep the whole thing on a big UPS. Total cost for hardware was about $500.
I mirror data onto the RAID using rsync from my other computers, (or just drag-and-drop to the Samba server from the Win2K box). I think this is cheaper and more effective than a tape backup system for 180 GB of data.
The Linux software RAID gives me the reliability - I've inadvertently tested it when a power connector popped loose from one of the hard drives - I didn't even notice until I read the kernel log (for a different reason). After powerering down, I plugged the drive back in, restarted, did the "raidhotadd" command, and away I went again. No data loss, no hassle.
For stuff that I really, really want backed up beyond the reach of thieves and fire, I use CDR's and a safety deposit box. Luckily there is not too much stuff that falls into this category - source code and documents for projects I've worked on, my email, stuff like that - it still all fits on one CDR.
If you don't use Linux, I think Win2K can do software RAID too. Never checked though.
Torrey Hoffman (Azog)
"HTML needs a rant tag" - Alan Cox
Along the lines of other posts, it seems that the only storage technology that scales with the speed of growth of hard drives seems to be other hard drives. Your best bet is probably to get an external hard drive enclosure, and swap out the hard drives like you would a tape drive....only the hard drives are faster and cheaper.
You can get one such from Addonics.
They have USB1.1, USB2.0, Firewire, PCMCIA, and CardBus interfaces. Plus the thing supports Linux and Solaris (although not yet in Firewire, but check with the Linux Firewire people about that...I think they are supported now).
"You have the option of insanity. I do not. And that makes me crazy!" - Brian to Angela, My So-Called Life
LTO is still to expensive for the home user and I hate having my backups on multiple tapes. Get an IDE Raid controller and a HD caddy and some extra HD's. Use mirroring and exchange disks regulary. Simple, cheap and effective
Clearly the answer, for easy backups of a 100G drive, is 21 iPods.
sulli
RTFJ.
There are several programs allowing to use your DV camcorder for backups. For example:
http://dvbackup.sourceforge.net
I would rather recommend a working data destruction system for the inevitable emergency.
This is much more difficult then you think because law enforcement authorities know about IBAS/Ontrack labs and use them.
The other way for security is not storing illegal data. I think this is the safest.
Owner of a Mensa membership card.
I bought a SCSI DDS-2 Tape drive off of E-Bay a year or so ago for under 100 bucks. (4-8 gigs a tape) ...It's kinda slow, but it works fine for weekly or monthly backups, plus my work had a ton of old tapes they weren't using anymore, so I have *well* over a terrabyte of storage... Most E-Bay sellers will toss in a tape or two and a cleaning tape too.
At work we have several external scsi drive bays that hold 4-6 scsi drives that is attached to an external scsi port. This would be perfect for backup if scsi disks werent so expensive,. Does anyone know of any external solution like this that uses ide drives in the bay instead of scsi?
The 100GB Western Digital Special Edition hard drive with 8MB of cache.
:)
Supposedly, it performs just like a SCSI drive, but it's IDE. A couple of these in the aforementioned rack in a mirrored RAID combo would make a perfect backup.
I'm definitely swapping out my current configuration for two of these once I can afford the $600.
Given that MP3's and digital pictures don't change with time like databases, documents etc, why not do what we do and back the whole lot up once. Then as you download new MP3's, pictures, whatever. Put them into a directory say Not_Backed_Up, and burn that to CD also when you have enough to fill a CD. Then migrate this with the rest of the data and start fresh with the next CD to be filled.
FWIW if you have the money to buy 2 identical drives RAID 1 might be the way to go.
Agreed that just backing up to another HD provides the best overall method for creating a complete backup of 100MB of disk storage.
However, I would suspect that most users don't change a huge percentage of their HD's content on a daily basis, unless you are routinely d/l'ing or ripping MP3s and MPGs on a daily basis (and I note that when I do generate that kind of traffic, it is usually because I am making a compilation CD, and while this does generate a few GB of "new" files on my HD that day, that data doesn't need to be backup up because I've got the original CDs anyway).
As a result, it seems to me that a reasonable solution is to create a "baseline" backup, say to a CD or DVD, at system install time, when there is (relatively) little on the disk, and then each day (or week, depending on needs), do an incremental backup of changed data only to another CD.
This approach is obviously quite inefficient if you have a complete HD failure, in that you have to recreate a new drive by starting with the first backup CD and then restore EACH ONE thereafter until the final CD restores the disk to it's last backed-up state, but for a more common problem of losing or corrupting an individual file, since that is more likely to happen with a recently modified than a remotely modified file, you are likely to be able to restore a last good version within only a few CD's of the most recent incremental backups.
A lot of people have mentioned that disk to disk backup seems to be the best way to go.
I agree.
What hasn't been mentioned is rsync, which makes disk to (local or remote) disk backups fast and easy.
It is trival to set up a second disk that is a "stale" mirror of your primary disk(s) that backs up nightly, and will boot off a floppy. This captures some of the advantage of RAID (quick recovery) while being an actual backup, not just fault tolerance.
Rsync can use ssh as a transport, so you can securely back up remote disks as well.
-Peter
I already have 2 30GB drives, and after hearing horror stories about hard drives crashing recently, I've decided my next step is to get another 30GB drive and run RAID5 across them all. Linux can do this in software by the way, and this way you can be assured that your data will stay intact if one of the hard drives crash or not, plus you won't lose half of your drives to backup.
If you are concerned with recovering deleted files, simply use tar or something similar to backup to either a separate directory, or create a separate partition on the RAID array. Another advantage is that you can always increase your storage by slapping in an additional drive, partitioning it the way you want, and then adding it to the current array.
Tape media is quite expensive in those higher capacities. They arent that useful either unless you are interested in archiving multiple sets of data for long periods.
A better solution would be to set up a cheap drive mirror with a cheap RAID card. Or better yet, put a backup drive in your computer, and use Norton Ghost to save an image of your data drive to it. Set the backup drive to spin down after a short period to save wear on the drive.
Make it a malt liquor. I want to be as clever and handsome as possible.
This is really only a short term backup. Since the storage media and the reader are one unit, if either fails the back up is toast. If you have two drives, you might as well setup mirrored RAID.
Real backup is done on semipermanent media (>10 year storage) in a format that can be taken off site easily.
If your box is full, put the backup drive in an old 486.
You'll be lucky if the BIOS on that 486 can see beyond 4GB on an IDE drive. The BIOS on many high-end Pentium Pro servers can't see beyond 8 GB or so.
my $0.01. 2 cents is 2 expensive in this economy.
Go onto ebay... you can pick up a DLT (15/30GB) drive for around $150. scsi card $50. media are a little expensive (~$25 ea), but for around $275 and a little opensource backup software, you can get reliable backups.
RAID offers good protection for some things: hardware failure (ie: HD crash) and uptime. That aren't the only woes, however... You can loose data in a lot of ways:
Disaster (fire, quake, flood, nuff said)
Hardware failure (disk, controller, ...)
OS failure (FS corruption, ...)
Application failure (User space applications malbehaving, virii, ...)
User failure (accidental deletes, experimental children - trust me on this one ;-)
:)
...) are a plus, but more cumbersome.
RAID will protect you from the second, but will happily add nothing in case of any of the other failures. Backing up to another media is a necessity.
Adding an extra disk (or two, or three), and some tar/cpio cronjobs will add basic protection. (No disaster recovery for you, unless it's off-site
Removable harddrives (firewire, frames,
Tape is considered a more 'trustworthy' backup medium because the mechanism and data storage are separated (ie: tape drive / tape), while in a HD it's in one single package, and it's not as easy to replace the logic board/stepper motor if this flunks. With tape it's easier: just get a new tapedrive.
Anyhow: don't rely on RAID to save your data - it won't.
Okay... I'll do the stupid things first, then you shy people follow.
[Zappa]
I setup a few different networks with a Promise U-ATA Hot Swap Kit, comes with a RAID card, 2 External Hot Swap Bays, and cables. Then you just get 3 identical drives, install one internally, and have the second inside of a hot swap cage. Set the system up w/ RAID mirroring. When you need a current, bootable, up to the second backup, simply pull out one of the drives, and plug the 3rd drive in (which is already inside of one of the hot swap bays) and let em re-sync. For about 5 - 10 mins the system runs a bit slow while re-synching, but its a no down time, 100% backup, redundent, and off site backup!! I love it! Sorry bout crappy format of post, heading home. :-)
No I didnt spell check this post...
That's a lot of 5 and a quarter floppies dude
the problem with raid (1) is that both drives are running all the time. this can lead to both drives failing from use after a while. the way i have it, all my mp3s are backed up onto a 2nd hd that sits on the shelf. no chance of it getting worn out while sitting there. if the drive in use fails, rip it out, shove it the like-new backup drive.
A few folks have mentioned that for 100G you would spend a day swapping CD's. This would only be true for the first backup. After that you would be doing incremental backups that could probably easily fit on a single CD or DVD. Of course every few months or so it would be wise to do another complete backup and start the process over. The problem with using a harddrive as a backup medium is its reliability. CD removable media is good for 5years or so. A harddrive can crap out the next time you power it up. So if you do decide on a harddrive make two backups just in case.
I've gotten burned for the last time with my cheap hard drives going down. My solution is two fold.
1) Throw together a cheap linux box and just have weekly uploads of whole directories to it. You could even go so far as to do your whole computer.
There are great programs out there that will just 'diff' all of your files and then update what's necessary. Over 10/100 ethernet, it's a good deal.
2) Then for your 'mission critical' files; I opt for a weekly offsite backup. Just another computer that I have access/control over at work/school. Since most of my mp3's and movies I don't consider to be important, they don't go there, but things like photoshop files, 'my documents' and photos so.
I figure $300 (minus drives) for the machine-in-the-closet and the cost of hard drives (if that) at your preferred offsite location.
I'm also assuming you've got broadband. =)
-- dK
Backing up your hard drives onto other hard drives sounds like a good idea (price, amount of storage, etc.).
If you a backing up things that would be inconvenient to loose but not mission critical then this method may be fine.
If your are backing up irreplacible data then I would suggest using a different method.
I have experienced the meltdown of two systems where in both cases it was the power supply that failed. Unfortunately, the when the supplies failed they took out pretty much every component in the systems (including the hard drives). I can only guess at the voltage that was pumped through everything but between the two systems I found 5 chips that were cracked or blown apart (1 of them on a hard drive controller board)!
These incidents happened at different times, in different locations, on different types of computers.
If you do use hard drives for backup consider removable hard drive bays and don't leave the back up hard drives in your system.
Just my 0.02.
--
Can't be bothered to login.
I'm currently backing up about 12GB under Win2K using an Onstream 15/30GB tape drive. They sell a 25/50 model as well, but neither they nor anyone else sells anything bigger than that, for a price that a home user would find reasonable. The IDE version of the 15/30 model goes for about $200, and tapes are about $30-$35 each (I have three and cycle between them). Add a couple of hundred dollars for the 25/50 model, or for a SCSI model.
Given those prices, I've got a different plan for when I outgrow my tape drive. I'm thinking about just buying 3 hard drives (whatever I can get for $100 each; right now that's 40GB), a copy of DriveImage, and a removable drive bay (with three drive enclosures, one for each hard drive). When I want to back up my system, I'll pop in a hard drive, use DriveImage to make an image of my system and store it on the removable hard drive. Like with my tapes now, I'll cycle through the three hard drives.
That strategy will probably cost me about $400, which is less expensive than an Onstream 25/50 (plus three tapes). It will also hold more data than the Onstream, and will be substantially faster as well.
In an enterprise sense. I know you are talking about home backup use, but for people at work and backing up enterprise data, it is good to have media that you can store off site in case of natural or unatural disasters. A lot of good RAID will do to protect your data if your server(s) are molten slag on the floor after a fire, etc.
It's like "looking busy" at your employment - it's actually easier to do real work than to fake it. - bmo
Personally, I have a lot of data to backup, plus I serve a lot of data to other people. The tape drive was worth it to me. I bought a refurbished DDS-2 drive on ebay for $180 with a full Seagate warranty. Tapes are about $3. I have about 80GB of data to back up, but can split the data into small chunks that will fit on a DDS2 tape (~6GB w/compression). Amanda takes each slice and determines when in my 2-week cycle it should get a full backup. Each slice gets backed up incrementally each night and a full at least every two weeks. Works beautifully and has saved my ass several times.
I would suggest looking on E-Bay for a DLT drive. You can pick some up pretty check. The only catch is that the media is pretty expensive.
Even with E-Bay the drive at most would hold around 80 Gig compressed. So, you would have to use 2 tapes.
The buy-another-hard-drive answer is easy for home applications, but it is also true in large installations too.
We have about 2.0 TBs of RAID storage that need to be backed up. I built the RAID servers using IDE Promise controllers and 80-GB Maxtor hard drives (the largest back in the day). Right now, hard drives make the most cost effective sense for backups on this scale too. The current plan is to build a backup server with 4 to 6 removable drive bays and set them up as backup devices using scripts/tar or bru.
Right now, I am testing hot swap bays with Linux to verify that the system stability will acceptable. Also, I am looking into cheap gigabit interfaces to connect the raid servers to backup server. This could be a nice backup scheme for larger applications too.
What do you do when the building burns down?
Is there something you are trying to keep secure?
Why do you want to keep your data safe?
Is an encryption device utilized with a harddrive or an application?
Where did you obtain all of your software?
Are you looking to copy to a device that has the ability to encrypt files?
If you are looking for a portable back-up device, why do you need it to be portable?
Do you travel extensively?
When you do travel, do you primarily travel by air?
Do you have a digital camera?
Do you have a mobile phone?
Have you ever encrypted an email message?
Have you ever deleted an email message?
If so, have you had data rewrite over the sector(s) containing such message?
What was the title of the last book you purchased?
"There ought to be limits to freedom"
Of course, data file should not be installed on a local drive, so that you can implement some sort of a disk imaging solution for the base installation. The disk image should be of the main drive with the core installation folders or mounts.
This way if someone screws up the system, you can blow out the main drive, replace it with a known good config, and then add the two or three apps you need, with the datafiles safely someplace else. This could even be done from a bootable CD, if needed.
"It is a greater offense to steal men's labor, than their clothes"
IMO there are two types of backup:
1) In case of hard disk crash, you want the data that was there restored as currently as possible.
2) To create a historical record of your documents so you can revert to older/uncorrupted versions.
For purpose #1 your only practical approach is to mirror all of your high-capacity space onto another HDD which is basically a RAID setup. (Array of INEXPENSIVE drives)
For purpose #2 are your media files really modified over time or do you just need a cdr for frequently edited documents?
If you only need a backup in case of break down, fire, etc, put a cheap second hand computer on the network with a removable hard disk, that you can store in a fire-proof safe, or off-site if you are worried about floods :)
Then the only possible way to lose both lots of data is if the lot is stolen, or lightning strikes your power lines when both hard drives are in operation.
Your backup is useless if it doesn't restore, and from experience, tape has failed too many times. The beauty of the hard drive backup is that it is quick, and you can image your production hard disk to the backup, and use something like Norton Ghost to restore it.
DOWN WITH TAPES!
http://pcblues.com - Digits and Wood
Mirrored disks are the best way to go right now. Mirrored disks will protect you from data loss if one of the hard drives fail. Tape solutions in the 100GB range are way out of consumer reach. If your concerned about other sources of data loss; such as fire, flood, etc. consider another suggestion made here and backup only critical files to CD-R(W).
I'm sharing my cable modem via 802.11 with all the neighbors and since I am the local "neighborhood helpdesk technician", they often come to me for advice. Recently, one of them wanted to know how to go about backing things up properly. It dawned on me that hard drive space is abundant and most people are buying much more than they need (the person in question has an 80 gig at about 20% capacity). So I worked out a deal so that everyone is backing up to each other's PC at night on a weekly basis. The 802.11b connection keeps drive thrashing to a minimum yet provides enough speed for complete backup on an overnight basis.
I should start charging for these ideas... Can't wait for the proliferation of freenet!
Life is the leading cause of death in America.
Everyone these days wants a technical solution to a simple problem.
The guy wants to back up MP3 collection - he's already got the original CDs, what's the problem?
He wants to back up digital pictures - he could print them out and save them in an album.
He's got important documents - nothing like a good file cabinet, you know.
It's just not clear why the answer to such a simple question needs to be so technically involved. If he just needs to save the data in digital form, he could have just gotten a back up hard drive and this 'Ask Slashdot' could have been answered in one post.
Magnetic storage : the data is sensitive to electromagnetic fields, heat, and the magnetism in the ferrite particules decays over time (i.e. the data has a reasonable chance to get corrupted after 5 to 10 years, even if the media is properly stored). Also, reading a magnetic medium can wear out the medium, but that's not an issue if you just want backups. The bit density on these media is good, and the price per megabyte is excellent.
Optical storage : there is no decay in theory, but I read somewhere that pressed CDs actually have a lifespan of 20 to 30 years even when properly stored, and CDR(W) even much less, especially when stored exposed to sunlight. I'm not certain there is much real-life data on the durability of DVDs. The bit density is good, but not as good as magnetic storage, and the price per megabyte, I guess, is comparable, but mastering optical media requires more effort.
I'm not considering the fact that magnetic storage media are re-writable, since you're talking about doing backups. So I guess, the question really boils down to whether or not you want to have backups that are more reliable over time, and whether or not you want to spend the time and effort to create CDs or DVDs of your data.
Then of course, there is the question of obsolescence : CDs have been around for years, and I don't think they're going to disappear any time soon. However, tapes for example can become unreadable because nobody makes readers anymore (ever tried to re-read a 80Mb Colorado tape recently ?). Same goes for hard-drives, although I'll admit IDE and SCSI interfaces will be around for quite a while longer.
Then of course, if you really badly want your data stored safely for the longest time, you can get yourself an old punchcard deck, a very large box of punchcards, and there is a fair chance that some archeologist in the year 4000 will find them in a dig just above where you house used to be ;-)
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
I'm currently using double sided DVD-RAM media on my Panasonic DVD-R/DVD-RAM drive to do backups.
Its not the fastest, but its great for doing backups of my web stuff and important files at night.
DVD-RAM media is pretty pricey. I think my 9.4 gig disk was 25 bucks or so. Not bad all things considering though. The media lasts for a long time and is durable (in a non-removable caddy).
I actually SCP from my web servers and back up the SQL dumps to DVD-RAM nightly too. DVD-RAM has 100,000 rewrite capability. Using UDF formats, you can format it FAT32 and it is recognized just like a zip disk.
EThan
I'm supprised that nobody suggested this sooner. Sure, many people have suggested adding another HDD for backup, but RAID would be a far superior option. It would improve your performance, and it offers excellent reliability to boot. Plus, updates occur on the fly--saving you the time of actually doing the grudge work. If you don't have the hardware or you don't want to invest in it, then just use a software solution--all of the latest versions of Linux and NT (through XP) support it.
# They who can give up essential liberty to obtain a little temporary safety, deserve neither liberty nor safety. --Fran
For all those moderating the posts about setting up a RAID system as 'Redundant', please think first if such moderation is not 'Redundant' itself, since RAID is obviously 'Redundant'. Hmm. OK, it could be RAID-0, but noone is talking about stripping. :)
Looks like the cheapest Kingston/StorCase offering (which supports ATA/100, bonus!) costs around $80 mail-order.
Are there any other reputable manufacturers that sell a cheaper solution for IDE?
o/~ Join us now and share the software
If you value your data, then a tape backup strategy is important. Use a mix of weekly full and daily incremental/differential backups. That way you can go several generations back and recover from corruption or a virus that slipped in a while ago. Take your tapes off-site, and you'll always be safe.
My itty-bitty biassed preference for SOHO tape backup is VXA from Ecrix, now merged with Exabyte. Great reliable drive and very good restore results...
To Terminate, or not to Terminate, that's the question - SCSIROB
I have a 120 gigs of data on an old Pentium 200 that I dont want to lose. You can guess what it is.
There are two ways you can go relatively cheaply, and IMHO a far better solution than CD-R or CD-RW.
Pick up a DLT2000XT (15gb native) off ebay for about $200. Tapes are dirt cheap, about $5/ea and the media is extremely durable, nearly indestructible.
Pick up a DLT1 (40gb native) off ebay, about $500. Tapes are moderately expensive at around $20/ea, but again the media is extremely durable.
DLT is industrial strength backup, the drives are built like tanks and the tapes can take incredible abuse.
Its all standard SCSI and works great with linux, no problems whatsoever.
I considered buying hard drives for backups, but they are far too fragile for long term backup and off-site storage. Most drives arent designed to be spun up and down lots of times either.
Last thing you need is for your backup harddisk to go splat when youre trying to power it up to restore your main system from a data loss.
With DLT, this isnt likely to happen.
For my home network, I use removable internal hard drives. There are several manuafacturers that make units that will turn an IDE or SCSI drive into a removable unit. I've got a dataport IV which has a component that fits in one of the 5" bays in my PC and connects to the motherboard via an IDE cable. A second component opens to hold a standard 3" IDE drive and plugs into the first component. I've got several of those. Back-ups are straight-forward: 1) Shut down the machine, pop in a drive module. 2) Boot 3) Do back-ups either locally, or across the network. 4) Shutdown, pop-out drive module and place it in a drawer.
It would be nice if I could hot-plug the drives instead of having to reboot. If I was more thorough (or paranoid), I'd take the drive off-site and put it in a safe deposit box. Dataports are made by CRU-INC, but there are other similar products.
[Insert pithy quote here]
I know this is a windows solution however I'm running two w2k advanced servers and have one on site one off site at my home.
They are clustered. First created with a 100mb crossover link. Then moved one offsite with a modem link between them (direct modem to modem). AS the files don't change much there is not a lot of traffic. (1 file in 3 days maybe).
There is also software link doubletake and surviveIT for the corporate world.
The MyTh - I am a figment of the Imagination - [Im Probably even not here]
I just bought a Buslink 80GB Drive. It was the cheapest solution of that size I could find (http://str8buy.com/bus60usb35in.html is the cheapest place...I have no financial interest here).
Reasons:
- Removal - I can take it to work or put it in my fireproof safe so my stuff is really safe.
- Removal (again) - Keeps you from accidentally deleting all that stuff, or a virus from doing the same. RAID is nice, but doesn't account for this.
- Large - Will hold all of my MP3s, etc that I have so far...when I need more, I'll just buy another, or 4 more...I'm not limited by the case, only my budget.
- Decently fast - If I want, I can keep it connected and it can easily keep up with a 100mbps demand from my home network.
Good Luck.
What I do is you Ghost to image the hard drive (you can even compress the image) and then save it to another hard drive or even a bootable cd/dvd that can later restore your hard drive image.
With HDD prices so low and RAID controllers available on many decent motherboards (e.g., the ASUS A7V133 has a decent Promise controller that will do RAID), I'm using mirroring. I've never had a disk crash or had any data loss to speak of (says he, tempting fate).
Back in my day, we only had single sided, single density 8" floppy disks. And I ain't going to calculate how many of those that would take.
I am Slashdot. Are you Slashdot as well?
Bringing up the system is less of a problem with newer OSes, since you can usually, at minimum, get to your data. Configuring the database, webserver, and firewalling depends on how good you are with the OS. However, when I worked at a former company there was no real plan to get a working system back in place. We were using Novell with Arcserve -- unfortunately, you couldn't get to the data without a working system.
Next I usually try to segregate rapidly changing stuff versus things that are pretty much static. E.g, my mp3 collection is relatively static. I occasionally buy a fresh CD and rip it, but I'm pretty much satisfied with my collection as it is. I put these on CDROM. It takes a while to create them, but it's cheap and safe. If you want to keep everything up to date, you can run a script to save only files not included on the CDROM.
Finally, I back up my constantly changing stuff such as CVS, MySQL database, etc. to 4MM tape. It's cheap (hardware and tape) and most drives are pretty well supported.
On most harddrives you can use a softbios in the harddrive, on my fileserver I used that to get a 6.4 gig drive to work when the comp only took 512 mb.
Since I installed that everything has worket perfectly so I see no problem to use a drive up to the 137 meg boundary on an old comp.
1 buy a printer with a duplex unit
2 for i in `ls *mp3`
do
mimencode $i | lpr
done
3 if you ever loose your data just OCR the page and decode it
(4) you could even save place with mpage
but sumthin like this was just posted on ask slashdot recently...
actually, most of the points made in the earlier story apply here.
and as i'm sure others will/have jumped in to say, there's a big gap between failover/redundancy afforded by a raid setup or external hard disk and tape backups. it, like so much else, depends on what you need.
i'm pretty discouraged personally. i don't really see an affordable way to do real backups of a couple hundred gigs of data. it's probably going to have to be a mixed setup. most of the data is static: flac & mp3's. so maybe that just needs one offsite backup that's done to a couple hard disks--basically a mirror. then get some kind of tape system for the other 40gigs or so of slowly or quickly changing data that i also want to keep a couple snapshots of for historical purposes, as well as rotate offsite. still not sure what the latter solution should be.
i have an old dds2 drive. but that's 8gb max compressed. dlt or ultrium is way out of budget. maybe an onstream. but i don't know anyone who's actually got one--any firsthand reports on these? and how likely is whatever onstream uses likely to exist/be supported in say 5 years or beyond?
I have had bad luck trying to backups with
Dantz Retrospect or Veritas to a yamaha cdrw, to a sony cdrw, and to a creative cdrw. IMHO, the firmware in these drives seems to hang if there is a media problem, and dantz just acts like its waiting. Veritas blue screens.
If you are going to try this, use cdrw's or you may end up with 35 coasters.
On the other hand, I shelled out $140 for Dantz because it supported my Onstream drive. This works the best, but sometimes the $250 computer i dedicated to doing backups hangs, expecially when I back up the full 120 gb.
Even with grip, ripping, categorizing, encoding more than a few cd's is a pain in the butt.
Jesus was all right but his disciples were thick and ordinary. -John Lennon
Recipie for an aneurism:
1) Take 1 RAID 0+1 file server and add 1 finacially challenged geek.
2) Baste with 50 Gig ADR Tape backup until Onstream SC50 is worn out.
3) Add spoonfulls of data until fault tolerant setup is nearly full.
4) Perform self-lobotomy.
5) Move all data to multiple machines across network.
6) Reconfigure server to RAID 0 to create more disk space.
7) Move all data back to nice big empty (non-fault tolerant) drive.
8) Cry in horror when one of the RAID's 75GXPs fails
9) Skip over shock and go to pure amazement when you discover that you're ADR media has failed.
10) ***Look at the monkey!***...BANG!!!
P.S. Can anyone recommend a good data recovery shop where I can have a failed 75GXP resurrected?
Formerly part of a RAID 0 stripe set in case you didn't guess...
- Fido
"It could be that the purpose of your life is solely to serve as a warning to others."
Advice is like cooking. You should try it before feeding it to others.
Get an external 1394 hard drive enclosure and throw a 100GB+ hard drive in there... thes are cheap ($80) at places like http://www.apdrives.com or you can buy an all-inclusive unit such as the ones Maxtor makes.
If you need to protect against fault tolerance then just get another hard drive. IDE RAID on the cheap can be done via software or another drive controller.
However, if you need to backup data with the ability to rollback changes or deletions then you are most definitely looking at a tape system.
Keep the Classic Slashdot.
I actually use 80GB firewire drives for my backups. They're ~$250 and much cheaper than JAZ, DLT, or even DVD-R. My 2 cents anyway...
Currently we've got a storeroom full of boxes and boxes of CDRs. I want to make an archival backup of them.
Negatives will last for a hundred years if they're stored properly. Just the other day we made reprints of one of the first sessions our studio did, more than 20 years ago. However, CDRs do not last this long. Assuming nothing catastrophic (fire, CDs breaking) happens; CDRs are only made to last a few years at most. (Rough estimates put the shelf life of CDRs at 10 years or so.)
What form of storage is the most archival?
Consider that we've got probably a thousand CDRs sitting in boxes. Some way of doing a batch backup (CD towers?) would be great.
Username taken, please choose another one.
I use Tivoli at work to backup our unix boxes but there are lots of gpl'd backup software solutions out there. At home I put a controller card and a 80GB HDD in a POS 233. Works fine and although it looks terrible sitting on top of the AT case at least it makes it easily removable ;) Provided the rebate ever gets here total backup cost to me will be a one time $150 including the controller. Everyone has to learn their own data requirements though. If you need more data integrity than availability then a stack of CD-RW's kept in your desk at work may be an even cheaper and more appropriate solution. My only real contribution then I suppose is: decide on what you're looking for in a backup and find a gpl'd backup app that fills the bill. The right app can GREATLY decrease the amount of time your backups run and maybe also decrease the footprint of the data itself.
with raid, next time a virus hits your pc that destroys .jpg and other random files, you're screwed. We all have also accidentally deleted files...and raid does not prevent this.
No protection against fire / theft either.
I'm classifiable as an audio addict, having taken my entire personal
/boot
/home
/pchome
/pub
/pub/mp3
/scratch
/pub/mp3_2
/pub/software
/etc/cron.hourly/rsync_with_fumus script:
/pub
/pub
/pub
:-)
collection of CD's and ripped them to MP3's at 320 bit, and wanted to
have them stored in a central place, accessible from any machine in my
home. Currently this collection is at approximately 620 full CD's of
music, and I'm pushing right at, or just above the 80 gigabyte limit.
Now when you factor in personal files, financial records, games,
downloaded material, installation software you don't want to lose,
etc...etc... Well, see for yourself. Here's my space breakdown for the
partitions on my main file server Fumus (Smoke, in Latin):
fumus:/pub/mp3 # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda3 3.0G 2.1G 804M 72% /
/dev/hda1 129M 6.8M 115M 6%
/dev/hda5 9.8G 1.8M 9.3G 1%
/dev/hda6 20G 13G 6.3G 67%
/dev/hda8 40G 22G 17G 57%
/dev/hdb1 75G 38G 33G 53%
/dev/hda7 1.9G 20k 1.8G 1%
/dev/hdc1 74G 34G 40G 46%
/dev/hdd1 74G 36G 37G 49%
So, here's what I looked at:
Tape: For the size I'd need: Way WAY too expensive. When I brought
the media down into the range I'd afford, I'd be swapping tapes all week
to get a backup done. Not time effective.
CD-R: Faster, yes, but at 650 megabytes per media, same problem as
tape, only you've traded magne tic for optical.
Extra hard drives in the same machine: Originally, this is exactly what
I had done with a single file server running Reiser file systems in the
more experimental days. I got the scare (and lesson) of my life when
Reiser went a bit nuts, and started corrupting some of my data. I only
lost about one percent, but I vowed, never never NEVER again would I
backup data on a critical machine on live media in the same machine.
Okay, so here's what I finally DID select as my solution: A second
machine called Ignis (Fire in Latin) that uses the absolutely identical
configuration, right down to the types and number of drives, partition
sizes, everything. They both connect into my 100Mb network switch, and
Ignis rsync's from Fumus every hour on the hour thanks to scripts in
/etc/cron.hourly
In fact, here's Ignis'
rsync -arul --one-file-system --quiet fumus:/pub/mp3_2
rsync -arul --one-file-system --quiet fumus:/pub/mp3
rsync -azrul --one-file-system --quiet --delete --force fumus:/pub/software
rsync -azrul --one-file-system --quiet --delete --force fumus:/pub /
rsync -azrul --one-file-system --quiet --delete --force fumus:/pchome /
Is this a bit extreme? Yes. But... if, gods forbid, Fumus really does
let out its magic smoke, or Ignis does catch on fire, and the physical
media were actually damaged, hopefully the damage would be limited to
*one* case, and wouldn't end up taking both machines out. Then I really
would be crying the blues.
Oh yes, and each machine is on their own 900VA UPS. I'm not playing
THAT game.
Another principle of good backups is to have another copy in another location, since having an extra hard drive won't help you if your house burns down.
OK, so buy more than one back up HD for each application and don't leave it in the box. If you want, take the extra copy home or put it in a safe deposit box. His point is that the extra HD is still cheaper than removable media for most people right now.
This is all lost on me right now. The largest projects I have easily fit on a CD, source executable and data files uncompressed. My photos are largely segregated by where and when they were taken and are archived that way. The one or two home movies I've made are no more than 10 minutes long and of such pathetic quality that they are less than 100MB each. This may change in time, but right now the 10 cent CD and a central FTP server are more than adequate. Thingy goes to FTP, then gets CDed several times and removed.
I doubt I'll ever pay to have someone else paw through^H^H^H^H^H^H^H^H^H store my data.
DMCA, Hollings, Palladium. What might have sounded like paranoia is now common sense.
Not to distro-bait, but Debian in particular shines here because apt makes it so damn easy to bring a system back to the state you wanted. For myself I have created a meta-package (.deb) which does nothing but depend on the applications I want installed on every desktop system: galeon, gnucash, xchat, gaim, xmms, vim-gtk, and a handful of others. Then I back up my meta-package, all of 10k including a few shell scripts I wrote for myself. Install my meta-package on a new system, and voilá, apt fetches and installs every app, that I need to continue working, dependencies included.
It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
I use an AIT system at home. You can get an AIT-1 drive for a low enough cost if you hunt around. I got my fist one for 300$ and my second for 65$ (but that was a rare deal). Tapes cost a bit, 45$ for a 35/70gig tape, but thats not too bad given the performance of the system.
"Have you ever thought about just turning off the TV, sitting down with your kids, and hitting them?"
Juste buy a 100 GIG HD dumbass
A true backup solution involves taking the data our of your house. This is VERY important if you are trusting years of digital photographs to erasible electronic media.
Your best option is to backup with high capacity tape. Leave a copy a your parents house, or in a bank safety deposit box.
Actually, it would be exactly 73,560.88963 disks worth of space required (or, essentially, 73,561 disks).
Using true GB's (1024^3 bytes), and with a 1.44MB floppy holding exactly 1,459,664 bytes, that's what it works out to.
*I* am the one with too much time on my hands.
----------
Darryl Ballantyne
http://www.darrylballantyne.com
Backup has always been expensive. You have two choices:
1) Go cheap (i.e. Zip drive, MO drive, CD-RW, etc.) and only backup the files you NEED (i.e. home directory, "My Documents" folder, etc.)
2) Shell out for tape. This way you will be able to make multiple backups, keep them offsite, maintain them long-term, and back up most of your system. I find that the sweet-spot is usually to go 1-2 generations old (i.e. right now I'm using a DAT24 drive), that way you only have to pay about $400-600 instead of $1500+ for the mechanism.
Don't fall for the "just use hard drives" trick. Hard drives have a number of problems.
1) Mechanism + media in one unit. I've seen hard drives whose heads STUCK to the platters, rendering them useless, after only sitting around for a couple of months. Oops! Data gone! They are also sensitive to static and to environmental changes -- if your backup drive gets zapped accidentally, you can't just plug the media into a new mechanism! Data gone!
2) Backups limited. If you backup a 100GB system onto a 100GB hard drive, you're limited to one physical backup. This creates all kinds of problems... Many's the time I've had to go back four or five generations in backups to find an old file that I didn't realize I'd deleted months ago. Not to mention that you are limited to one backup stored in one location -- no redundancy.
3) Even RAID has problems -- I've had bad SCSI cables that filled a RAID filesystem with corruption before we were able to track down the problem and switch the cable. If that happens when you're just depending on RAID to preserve your important data... oops! There's a reason why many RAID-enabled datacenters also maintain backups of critical data in a second medium!
So that's my personal take. I think either tape, maintaining at least 5-10 backup generations at a time, or MO/CD-RW just for keeping your critical files. Or both, even.
STOP . AMERICA . NOW
Right now I _encrypt_ my files and put them on a CDR. So if I loose the CDR, I can be sure no one is readking my journal :-)
NOw, I always wanted a
- reliable
- cheap
- plenty of
- unix friendly
online backup. I tinkered with myspace.com and virtual hard drive and they are not worth while
- for the amount of spam you get
- the tiny space (10Meg, yeah right!)
- most of them want to install clients (win xx only)
Does anyone know any good alternatives? I think this is a lucrative business. Can not understand that no good service is available!
thanks
LInuxLover
I'm at almost exact same bind right now.. lots of the data on the hard drives, and trying to find the best to archive/backup them. adding a second HD might be the simple solution, but the hard drive method lacks the transportability. (when you want to move certain data to another location, you'll need to power down and move the HD). and in my situation the data grows larger daily, so it'll out grow the HD soon. CDR right now is the cheapest solution, but it's tedious.
Wouldn't it be great if something like this would do the trick?
Most of you are missing this problem (fire/theft). What are your various solutions to account for and protect against this?
h ol d=1&commentsort=0&mode=thread&cid=2689576
I use a removable drive that can be taken to another building or put in a fire safe. Any other options out there? I'm sure we're creative enough to have some decent options.
More info on mine (don't want to re-type)...go to
http://slashdot.org/comments.pl?sid=24768&thres
(sorry, you have to cut and paste cause I'm lazy)
You may want to look at this paper by Sanders et. al. titled Redundant arrays of ide drives available on xxx.lanl.gov under hep-ex/0112003. They report tests of raid-5 under linux for backup of physics data and find it price comparable to tape at the 10^12 byte level
For instance someone recommends OnStream, the 30GB or 50GB models.
Well, those models are really 15GB and 25GB, the 30GB and 50GB numbers are based on expected compression.
Of course you can use the same compression to fit more on ANY backup media!!!
Why not call DVD 4.7GB media as 9.5GB media?
Or CDR media as 1.4GB?
Remember JPG, MP3, AVI, MPG, ZIP, RAR, GZ and so on will not compress at all. So you begin to see what a farce tape capacity is.
Given the REAL capacity, how much do you pay for a Onstream tapes per real GB?
15GB for $30
25GB for $40
So the best price is $1.6 per GB.
Then there is the drive cost, figure $300 for the 50GB model. So let's say you buy 12 tapes, and a drive, 300GB of backup storage for $780.
That's a real cost of $2.6 per GB, for data that can only be read on another tape drive, and is not random access.
Harddrives are cheaper per GB, faster, and can be plugged into any computer. The price keeps falling everyday. So only buy what you need now, and pay much less in a year.
Uncomfortable with IDE? Go Firewire, it's hot plug and play (no reboot), about $3 per GB (and falling).
Network Attached Storage would be a good solution here. Ideal would be a pair of 100GB drives that the user could configure to Raid 0 or 1, and replace drives as needed. Unfortunately, the cheapest NAS out there today is about US$500 for 40GB....not a good cost/capacity ratio. If SOMEONE would start selling a NAS for the home user (linux driven, web based config, scaleable and upgradeable), priced slightly higher than the cost of the drives installed, they'd sell a lot.
:)
At least that's what I think!
I think the answer is heterogeneous & requires some thought.
1 system for on-line mirror backup - perhaps nightly incremental or full day old
Use another drive or multiple for this
Cheapest & fastest
Same machine or over a wire to a second machine where you can verify the stuff is backed up.
Have you tried testing your backups recently? How about backups of the system/OS?
1 other system for multiple versions/checkpoints
Use appropriate media CD-ROM, DVD, etc.
My plan is to never have to use the backup.
Use RAID 1 or RAID 5 on the main file server.
Do what I say, not what I do.
Seuss - I'm telling you this 'cause you're one of my friends. My alphabet starts where your alphabet ends
First you must give consideration to what you really need to backup. I only do my home directory. I have a tar/gzip of /etc there. I burn my mp3's to cdrom as necessary. How much stuff do you really need to backup after that?
RAID 5 is not foolproof. I just had a server lose one disk which caused a second disk to go offline which toasted the whole array. I restored from tape.
Extra disks are not foolproof. I tried to boot SCO unix the other day and the harddrive no longer functions.
Tape has been the most reliable as long as it was written with a tape program that still exists. I have read 5 year old BRU tapes successfully.
I still use a 4MM DAT. It's small, so I don't backup much, only the necessities. But I know I can get it back!
PK
dvbackup is a utility that lets you store up to 13gb of data on commodity miniDV tapes. With the use of a Digital Camcorder.
:(.
This is not exactly a dirt cheap solution, but if you have/want a digital camcorder anyway, there's only the cost of extra tapes.
Make sure the camcorder works with dvbackup before buying one though. It doesn't work with any JVC's that I know of, or at least not mine
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-U
My main server is a SCSI system, with close to 200G of storage. I poped in a Promise RAID controller (Just becuase I like this card) and a couple of Maxtor 100G drives in a removable mounting kit.
The removable mouting kits I got off eBay for about $10 each (Needed two). The raid controller cost me about $50 and the drives were each $329.
Effectivly, I've got less than $750 into a 200G backup solution. Evreything is in a mirror set and works great.
I actually had to test it once too - SCSI drive failure. I replaced the SCSI drive, reassigned the new drive to the mirror set and the rest was automagic!
Good luck!
-brian
-brian -- Brian D. McGrew { brian at visionpro dot com } --- > But his grip on his santiy hovers somewhere bet
For my current network setup, I have a server that uses GraniteDigital's Hot Swap Firewire solution. It is a drive enclosure that has a removable tray that you can stick an IDE drive (any size) into and backup across the Firewire connection. Has Linux and Windows drivers. Very reliable, fast and cheap, with hard drive prices constantly plummeting.
for i in `cat rsync.list| egrep -v "^#"` /vol/backup/$HOSTNAME/$DATE
/vol/backup/$HOSTNAME/$DATE
do
HOSTNAME=`echo $i| awk -F: '{print $1;}'`
DIRECTORY=`echo $i| awk -F: '{print $2;}'`
DATE=`date +%A`
install -d
rsync --numeric-ids --compress --rsh=/usr/bin/ssh --recursive --archive --relative --sparse --one-file-system --compare-dest=/vol/backup/$HOSTNAME/current $HOSTNAME:$DIRECTORY
done
Then once a week we run a similar script that updates the 'current' directories and uses --delete
(rsync.list contains entries like "hostname:/some/mounted/partition")
Get high-bandwidth Internet access, buy two more hard drives, find two other people who do the same and a few kilometers away from you (each one in the opposite direction). Start exchanging encrypted backups (for example, tar files postprocessed using GnuPG).
No doubt about it. Ripping a whole CD collection is a pain.
I just see this immersion in technology that most posters seem to be involved in (notice how people can say "Get a RAID system" without batting an eye?) as a sort of slow death. Technology's promise is the easement of difficult jobs, freeing up valuable time to spend with those we care about or doing things we care about. It seems that the result of technology hasn't been that, but rather that we spend our time servicing the technology instead of being set free by it.
Whatever did we do before home automation?
get a spare hard disk and run it in raid, mirror mode. done.
Liberty.
I do almost the same thing. The thing that made this work for me was getting three fire-wire enclosures for IDE HDs. Now I have a cheap wasy to back up my data, I have one copy always off-site, and it is hot-swappable. Firewire PCI cards cost $50, the firewire enclosures cost $90 per drive.
If my data set grows, All I have to do is get bigger hard drives and put them in the enclosures.
Even when buying 2 drives every time my data grows, i am still coming out way ahead on the price of tape based solutions. I don't even use compression, just live, scheduled file copies.
I used to mirror everything, but that doesn't help when I accidently delete the worong directory and loose my wife's sister's wedding photos. (OUCH, I am still hearing about that one.)
If your data is fairly static and you just need to mirror it, buy an extra hard drive and run a dump periodically.
If your data changes quite often, and you need the ability to do "point in time" recoveries of particular pieces of data, try Amanda.
To backup a 100GB drive, you require...
- 6 DVD+RW (18 GB) discs, or
- 20 DVD-RAM (5.2 GB) discs, or
- 158 CD-R discs, or
- 72,818 HD 3.5" floppy discs
My car gets 40 rods to the hogshead, and that's the way I likes it!
We've had quite a few of these both at our store and sold to end users. I've even had one of the 30GB IDE flavor myself.
Out of the 40-50 that we ever sold I can't think of one that is still out there and functioning properly. We went through 5 of the 50GB SCSI models in our store's server alone.
We finally realized why they were so cheap, because they are made cheaply.
We now use a 40GB Seagate DAT drive. It's still not perfect, but it sure is a lot more reliable.
Duris MUD - The best pkill MUD. Ever.
Probably want to back up /etc also, but I basically agree. It doesn't change as much, but should probably be backed up whenever something in there changes.
I have over 100 GB of hard storage, however most of that is mp3s, videos, and other things that do not change. I have burned on CD all of my mp3s and other things that don't change, but I don't want to live without if my hard drive goes down. Then every night I have a script that does an incremental backup and zips it, PGPs it, and then ftps it to my friends hard drive, both at college with a fast connection and a 100 miles apart. Also once a week I do a full backup. I figure the chance of both of are computers getting struck by lighting, bruning in a fire, or failing at the same time is pretty rare.
Its not the best of all solutions, but I don't have anything I would die without and it was really cheap (free).
This is one segment of the computer market where the industry has failed to provide a solution.
And so - I will continue to hear stories from co-workers, friends, and relatives about the x months of data that were lost when their computer crashed.
Some hotshot venture capitalist with some geek buddies ought to jump at this opportunity.
Manufacture a removable storage - external device - writing to cheaply manufactured slower hard drives (why spend top dollar on 10,000 RPM drives when you're doing backup as a batch process overnight? - oh that's right, because the industry doesn't make slow, high-capacity drives anymore - they assume that all applications are high-end).
Bundle the box with some cross-platform data management software. Users could just plug in the box and set it off once a week. The rest of the time, keep the box in a fireproof safe, unplugged, or something like that.
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
What someone needs to do is figure out how to use Digital Video cameras as a back up medium. Think of it like this something like 600x800x32 for each frame, is 15360000 bits. At 30 frames per second, for one hour is 1658880000000 bits for a tape!
Now that is something like 207 GB / tape. And that is not including the audio track. And since many video cameras are firewire you get 400 MB/s.
Now I see why people complain about stories getting favoured by certain people. I submitted this question back in September.
UNIX/Linux Consulting
I just plug in an 80GB Firewire drive to my OSX system and run an Applescript that uses Stuffit to compress and backup up text, source code, and most other files that have modification dates less a number of days I enter in the dialog box. The digital video and audio clips along with 3D models and renders don't compress that much so I keep them on anohter Firewire drive and copy local to edit. Works for me, but then maybe you don't use a Mac with OSX. Oh well...
I keep planning to get fancy with Perl one of these days
Didn't they go out of business awhile back?
Buy 3 IBM Deskstars. Your data will be safe on them. :-). Oh and thay are hotswapable. So just use some cradels and switch one of the two on a RAID 0 setup. Come on its sure fire way to love backing up.
The guys over at kuro5hin?
I bet they have some warm stories they'd like to share. Just watch the raw wounds and you should be ok.
m
RAID is to provide either additional speed and/or hotswappable capability. RAID really stinks as a backup, since RAID doesn't care when some program deletes most of the hard drive, when some user removes too many files, or when the OS barfs. Sure, RAID will save your DATA if one HDD fails, as long as whatever caused it to fail didn't affect the other drive, but for the reasons already listed, this doesn't mean RAID is a valid method of backup.
However, a HDD in an external enclosure could be considered a valid backup, however, for true redundancy, you better have two drives you swap, and you better be doing surface tests regularly. A drive, properly treated, should last many, many years. Also, you could combine a drive with monthly or quad-yearly backups to CD-R, just make sure you do your research on the inks used in CD-R disks, some don't last as long as others.
Just my $.02
What causes an EMP? Nuclear explosions. With all the terrorists and their suitcase nukes around, you're magnetic data's not safe! Go Optical. Better yet, break out the punch-cards!
</fearmongering>
Seriously, if your backup media is magnetic, be it a tape, mirrored harddrive, or a vast pile of old AOL 3.5" floppy disks, you've got to watch out.
Case in point: A company I know of stores off-site tape backups at a reputable insurance firm's lock-up. There are all sorts of gurantees against fire, flood, tornados, etc...
What there is not a guarantee for is Larry, the night watchman, who brings his ancient portable television up to work with him every night. He sits it directly on top of the tapes he's supposed to file that night.
The previous didn't happen, but it *could*. There are all sorts of accidents that can corrupt magnetic media that wouldn't harm CD or DVD media.CD-R's are especially cheap, and you can reasonably back up everything short of massive databases or large AV projects on one. If you regularly make massive databases and/or large AV projects, you can probably afford DVD-RW.
Tape is getting obsolete and DVD's are getting cheaper all the time.
The next Slashdot story will be ready soon, but subscribers can beat the rush and slashdot the links early!
Intel, who created it, are going with IEEE 1439 instead. (Firewire) No one else is really interested in it either.
When using hard drives as a backup medium, you must remeber that you have to buy TWO extra hard drives, not one.
The pitfalls of a 2-hd backup system are apparent when you consider the following:
Hard drive A is in your computer. You're backing it up onto hard drive B, when hard drive A fails during the backup...
...and now you're hosed. All the data on both drives is gone, which is a potentially FATAL scenario if you're a small business.
The solution? Like I already said, get two backup drives, and alternate them.
--R
XML causes global warming.
Just because a solution doesn't depend on digital devices doesn't mean that it's offtopic. In fact, backing up photos in albums is a much better way of safeguarding them than sticking them on a server somewhere.
What happens when your intern scratches your delicate DVD?
I forget...are we at war with Eurasia or East Asia?
That worked really well for backing up our 80MB drives onto stacks of 1.44MB floppies, since you would really only need to insert about 5-10 floppies during your weekly backup, just to get the files that changed.
So why not just do incremental backup onto CD-Rs? Even with 100GB of archives, most of those are static. You probably won't need to use more than one CDR per week (maybe two) to track the changes. It's cheap, relatively painless if you've got the right software (and it wouldn't be hard to throw together incremental backup/recovery scripts in Perl if you're into that sort of thing.) and you've probably already got a CD burner.
If less than 650MB of files change in a week, the rest of the CDR can be filled up with files that were on earlier CDRs (this way your backup set can remain finite and you can throw out the earlier CDRs as they become obsolete. Or if you keep them all, you can reconstruct that state of your hard drive at *any* time, not just at the last backup.) This seems ideal to me--why is everyone else talking about expensive solutions like tape drives, DVD-RWs, and second hard drives?
I have a positive modifier on Troll. When I mod someone Troll their karma should go UP!
I just went out and got a cheap old machine, installed a simple Linux install (no x, the bare min) and have it do Rsync every hour. Then I also have my computer at work do an rsync with my home computer every night. Works great for me. Haven't lost anything yet.
I have to wonder whether (first of all) why in the heck anyone would need to have 100GB of disk space on a home system. But then I have five systems networked together and have more storage than I would have thought sane a few years a go though I have a bit of a ways to go before I will run into the poster's backup problem. It wasn't too long ago that, if you could afford 100GB, you could probably afford a SCSI array controller that would let you do a lot of RAID, hot swapping, automatic drive replacement, etc. With today's cheap disk prices you don't have to be wealthy to have an ocean of disk space. (I can remember the days when we thought having 900MB on a MicroVAX II was extravagant.)
You could always do it the traditional way and get some tape drives. Unfortunately, they're much more expensive than you might think when you have to backup that much disk space. You certainly wouldn't want to go cheap and be feeding 90m DAT cartridges into a drive all night (it'll start feeling like you're backing up to floppies before long). A good high capacity tape drive can get, what, 20GB onto a single cartridge? Not bad. And I think that at this point in time, tape is more cost effective than DVD-R. (Something tells me that the MPAA, and maybe the RIAA, will try to keep it that way too.)
Mirroring disks can be helpful. Hard disks are getting cheaper and cheaper. Heck it's almost scary mow much disk space you get in a typical PC sold at Best Buy nowadays (and without a backup device; it's almost criminal). If you're running mirrored disks you'll forestall the inevitable disk crash that takes all your data with it. Question for the Linux folks using the `md' driver: Does it allow adding a third member to a mirrorset? And, if so, can it be done while the system is `live'? (The third member gets removed and taken offsite in case there's a disaster.)
One final thought: The poster wasn't actually running a 100GB filesystem were they? I'm thinking that a power glitch could cause a world record to be set for the longest fsck-on-reboot run. Plus I'd think that backing up such a beast would be a challenge. I tend to keep my filesystem sizes no larger than what I can fit on a single tape cartridge... just to make life simple. (I'm used to having to pipe `df' commands through `more' at work so I don't mind lots of mount points. :-) )
CUR ALLOC 20195.....5804M
There are lots of good suggestions about RAID, copying to another drive, etc. But as long as the backed-up data is stored in the same physical location (home) as the original, you are not safe from natural disasters or theft. Maybe this is paranoia but it depends how important your data is to you.
I have isolated the data that changes on a regular basis that I want to back up. Mainly this consists of mail and financial data, in the 10-20MB range. I have 2 boxes so each box backs up data to the other box. But just to be safe I back up to a trusted friend's machine also. It's a large ftp transfer but not bad at all with cable modem.
For other data that I want to save that doesn't change regularly (photos, mp3s, etc) I use CDRs.
What am I supposed to do, VIDEOTAPE all of the stuff as I open it up in Windows? Like I'd want to type all that code back in by hand.
Im sorry i cant believe those prices? Now okay possibly you can get some cheaper drives on eBay but even still i was just recently looking at drive prices for a client and here's a comparison:
HP DLT 80Gig (40 native) = AU$4000
DLT Tapes = AU$150 / EACH
(For comparison)
HP DDS4 (20G Native) = AU$2300
DDS4 Tapes = $60 / Each..
Even after conversion to US$ ( AU$1 = US$0.50) thats way way out of im sure just about anyone's home backup solution price range!
But i completely agree with you, DLT is an Industrial strength backup solution, one I frequently recommend, but even DDS is far out of the price range for almost all home users..
Then when your hard drive fails, there's a certain probability that it might not have failed, and just back up from a different universe.
That would have to be one hell of a scratch to damage it. Surface scratches can be repaired, you can buy the stuff to fill in the scratches at computer/music stores. The scratch would have to be deep enough to actually damage the metal within; unless your intern regularly plays with belt sanders in the office I wouldn't worry.
Additionally, it's a lot easier to protect against scratches; i.e., if the disk is out of physical harm's way (you know, like in a CD case!), it won't get scratched. It's a lot harder to defend against magnetic damage (e.g., you would need specially shielded cases or whatnot).
Liberty in your lifetime
One of the biggest problems with any other sort of backup scheme is that you have to do it and you will usually have something better to do. With a dedicated machine running only this sort of thing you are much more likely to get it done than find the time to burn 20 DVD-RW disks (which will set you back $700 btw) or load 3 or 4 tapes or... The only downside on this sort of thing is that in the corporate setting, this is not a satisfactory "off site" solution.
Drives are getting cheap enough that you should
consider going RAID. Either get two identical drives
(well, partitions) and do RAID-0, or get multiple identical
partitions and do raid 5.
Linux (red hat at least) includes all the software RAID tools
you need...
Can't remeber who makes them, but at comdex this year I saw an external hard drive housing (probably IDE inside) that connected to the PC via firewire. This should allow you to mount a 100GB hard drive in the external housing and plug it in as needed. Also allows for taking one off site. Get two hard drives and housings and just swap them on whatever cycle suits you.
ekiM
On a somewhat-related topic, has anyone out there found any good tools for CD-RW (I like my CDR tools fine) under Linux? What about just being able to get at my old backups on that Win95 Adaptec EZ CD-creator CD-RW I have? (I installed RH 5.2 or some such over Win95 when my roomate got my Win95 partition to eat itself. After all, I had all my important stuff backed up. It just turns out that I can't read my backups.) Is it a standard format, or does Adaptec use their own format?
Copyright Violation:"theft, piracy"::Anti-Trust Violation:"thermonuclear price terrorism"<-Overly dramatic language.
Publish everything to the web and let Google cache do it for you...
That is all.
Most HD manufacturers use 1000^3 as there GB... makes the drive sound larger if you are used to using 1024^3 as your base of measurement.
Then, it is 68,509 disks
What the world needs is this... a virtual filesystem which aggregates storage on multiple machines, storing the data redundantly, with different levels of quality-of-service (so that video is stored on faster storage and archives on slower), encryption by chunks so that no complete file is stored on any one machine, capable of handling both write-once and read-write media (while still allowing any file to be changed), built-in version control, etc. If some machines in this network go down, the data (or most of it) should still be accessible.
Yes I know there are lots of incomplete implementations of such an idea (AFS, some peer-to-peer projects, another FS whose name has slipped my mind at the moment, etc), but I think there is still not one good implementation which has all the right features, can be mounted on multiple OS's, reliable enough, etc. If this were done right, backups would be obsolete.
3ware 6800 Escalade IDE RAID is working great, and it's only $350 or so.
It's a PCI card with *8* IDE drive slots, which you can configure in a RAID 5 array for huge, failsafe backups.
I've got 8 60-gig IDE drives on it, in a RAID 5 array. Gives aboout 420 gigs. Shows up as a SCSI device in OpenBSD. Works great with Linux.
Churning away wonderfully.
I've backed-up 200 gigs of files on it so far.
420 gigs of RAID5 storage = $1100 USD. ($300 for the card. $800 for 8 60-gig drives.)
Here's my post on the OpenBSD list about it
I actually just finished tackling the same problem for my own 20GB mp3 collection. I wrote a utility that basicly automates the process of spliting the directory tree into CD-sized chunks, since the thought of manually spliiting out 30 CDs worth of tars gave me the willies.
I posted it at http://danky.com/cd-cat-0.1.tgz if you would like to check it out.
Ok so i know its not on the market yet buuuut,
,some figures I saw for the true life of a CDR is 2yrs or less if accessed too often !!!
@150gig per disc and going much higher with blue laser it could be the answer to our prayers in the near future. which is why I invested in this small company CDDD.
oh and yep I would go with 2nd or third ( one offsite ) hardrive for cost,speed, and reliability
{ Pillar candles great for when the power fails and you cant see the keyboard..
I have a dedicated CD-Burning machine. It has an IDE removable caddy in it. I am setting it up so that my server, late at night, sends a wake-on-lan signal to the burner, powering it up, and then dumps a backup to the CD-burner. The CD burner then dumps the backup to the removable hdd. I have 2 hdd's in caddies, and plan to rotate them offsite. That way, there is always one backup offsite. There's no need to use a CD-burner for the backup machine; I just find it convienient to have a machine used for nothing but burning and backups.
i trade some usa food aid packages for DLT2000XT Internet is GREAT!
Internet is Great!!! junis
Since when is a harddrive not a semipermanent media that can be easily taken off site? I'm surprised this comment got modded up so high. And since when are tapes such a reliable media compared to a hard disk? So burn-in the drive for a few days before using it for backups. And use a S.M.A.R.T. utility to diagnose the drive before each backup to reduce the chance that something is getting ready to fail.
Your best option is to put all data on a 2-disk mirrored RAID and use another drive as a removable for an off-site or fire-safe backup. The probability of 3 hard disks failing simultaneously, one not in use, is so incredibly small it's laughable. And for that non-zero chance, if it happens, you can pay to have the spindle of one of the failed drives transferred to a new drive in a clean room.
Advantages:
- Standardized TCP-based protocol, uses authentication.
- Lots of free "backup software" available - e.g. pan, newspost, tin...
- Centralized backups over the network! Only one server directory to back up.
- Backups are automatically dated by the server, and can be signed and even encrypted if you integrate pgp into your posting script.
Disadvantages:It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
I've been backing up my data for the last 4 years. I mean literally, I'm not done yet.
I've been looking at the raw data and writing down the 1's and 0's in the correct order in notebooks.
it's long, painful, and expensive; but... but... well I know I started doing it for a reason...
"when life gets complicated, I like to take a nap in a tree and wait for dinner" - Hobbes.
since they only have a 40% duty cycle
better to use SCSI
even still...
DLT IS SO BETTER!!!!!!
Some day all your original files will disappear,
and then your backup script will dutifully erase
all of your backup files.
sure, if you are just backing up your own crap and it's under 20,000 files, i'm sure everyone here has that under control. everyone knows how to get a second drive and back up their stuff.
the problem is when you have hundreds of THOUSANDS OF FILES or you are responsible for backing up hundreds of people's stuff...without tape rotations your dead meat. becoming aware of WHEN things are deleted can take months.
case in point: i had a researcher delete 10 critical files for a 5 million dollar grant. The lady has in the vicinity of 100,000 files. she did not know she was missing them for 8 months.
i pulled out my archived monthly DLT, and restored her files.
the last 6 months tapes did not have her files on them...but the 7th did.
Looking towards the future, we will have 140GB plasma CD-RW drives and "burners", which should also read/write DVD/CD-RW since the shape and size of the discs will be exactly the same. Only instead of using pits for bits, it'll use a beam that can read and arrange the plasma within the disc. Saw it on zdnet.
Personally, I use Norton Ghost. You can use it to backup to CDR, it compresses the data, and you can make the CD bootable. Put my backup CD in, boot, reload my entire system in about 2 minutes.
It's easy to stand out when the general level of competence is so low.
My Suggestion to you is to just plain wait. Since its your home, data backup should not be critical, yet.
I am willing to bet that 18months down the line u can pick up a DVD-RW for $300 and 8.5gig Double Sided DVD's for about $5/piece. I would say that unless the data is absolutely critical, wait. However, your cheapest and easiest bet (i think) would be a large firewire 200GIG Raid array (yes they make them. check MacMall (they should work for PC's too).
Good Luck, and if u read this let me know what you use and send me an email, i'd be interested in this myself a few months down the line. AJ
-------
artlu.net
Even worse is you get perfectly good copies of the original drive but at some point a few files become corrupt and you don't notice until both (or all three) of your backups have the same data. This is why tape backs while expensive are a better solution. You can do multiple backups of the data cheaper and you can do incremental or differential backups so you have shorter backup times.
Product break down for affordable tape backups by PC World here. I just had a hard drive fail. Ontrack couldn't retrieve any data. I'd go with tape.
We're talking about home backup here, so you don't lose your collection of downloaded mp3's if your primary hard drive crashes.
Storage in an offsite data vault on media that will last 10 years is just a wee bit more than COMPLETE AND TOTAL OVERKILL! FURRFU!
I looked into this a while ago wanting a high capacity backup that could go off site that would run under linux I didn't mind having a dedicated machine but needed it not to require a reboot each time I wanted to remove/insert a drive.
The problem I found is that regardless of the caddy/drive/controller noone could show me a way to unmount, spin down and safely unpower a drive prior to removal (using IDE drived). Also if some of the bays weren't filled when the machine _does_ need a reboot, once the bios has decided there's nothing there you cant add that drive later.
I trawled the source to hdparam finding nothing much. I found that SCA compatable SCSI cards normally have the ability to safely spin down and power off a drive, but I didn't find any way to use this in the general case let alone in a specific linux enviroment.
Loop, twist and loop again.
rm -r large_unnecessary_files_and_folders
I have used CDR's in the past but due to recent studies showing true life span to be a lot shorter than the manafactuers claims ie for 25 yrs read 2-5yrs ! so I switched to 2nd hardrive back up.
{ Pillar candles great for when the power fails and you cant see the keyboard..
A little over a year ago I decided to build my own home server to do some simple tasks like e-mail, web server, storage space for MP3 files and the like. When I started the project I had two main goals in mind:
In order to keep the costs reasonable I purchased the cheapest motherboard, CPU, RAM that I could find. I used an AMD K5 333 with a no-name board and a micro-sized case. Basically, the goals here were to keep it cheap, cheap, cheap for the basic hardware. Other then RAM, my experience is that most servers don't really need much processing power; I figured that for what I wanted just about any low-end processor would suffice. Having sufficient RAM however *IS* important, particularly if you're going to run a large number of processes like e-mail, web, etc. The good news was, and still is for that matter, that RAM prices had fallen significantly, making my 256MB minimum requirement affordable. But again, in the basic computer hardware department the goal was to keep it cheap. I'll get to why in just a minute.
Anyway, I was fortunate to have an old CD ROM and floppy drive laying around, these items helped round out the basic system at a cost of approx $300 for everything, sans hard drives and UPS that is. For the UPS I purchased an APC model, the Desktop 110 as I recall. Since my biggest concern wasn't to keep the machine running through a power failure, I could reasonably use a lower-end model. Even the basic model UPS systems will do line conditioning and surge protection so as long as the unit has enough battery storage to do an immediate and graceful shutdown of the server I felt that it met my needs and, I suspect, would probably meet the needs of other people in similar situations. The UPS ran me another $150.
Finally, onto the primary issue, the hard disks and backup. This is where I knew the costs would be high, so I tried to hold onto my cash till I got to this point. What I really wanted to do was to go with a tape-based solution, but two factors convinced me otherwise. The first, and most obvious issue was cost. Good tape backup systems typically run into the thousands of dollars, putting them much to far out of my budget. Secondly, my personal experience with tape systems is that while the tape costs are cheap the units do require periodic maintenance, again at a significant cost. You also need to invest a significant amount of time doing tape rotation and checking the backup logs. Tape drives are much to complex to really do the maintenance yourself (and this from a guy who used to work at a tape backup repair shop) and once you pass the warranty period expect to pay several hundred dollars and spend a week without the unit for the required maintenance. And no, you really shouldn't skimp on that maintenance, reading the logs, etc. Remember, this is your backup solution; you need to be able to rely on it. In a business where these kinds of periodic costs and tasks aren't such an issue and the necessity of having an of having an ultrareliable backup method is critical it makes these issues less burdensome and it makes more sense to use tape. For my basement server the drawbacks and the costs just don't add up. So, what is the best way to go?
After looking at various other backup solutions like the OnTrack unit, CD-Rs, DVD writables and the like, I finally setteled on doing hardware RAID with a Promise card and two 20GB IDE disks. Cost, approx. $500 for the card and the disks. Why did I choose this method of backup? Well, it met all the criteria and then some. I used a pair of fairly inexpensive disks; this again kept the costs down. Since I really didn't care about disk speed (no swapping, all that RAM remember) I went with 5400RPM models to keep it quiet and low-cost. The Promise card has drivers for both Windows and Linux and was easy to install and use. And there are no tapes to swap out, no rotation schedules to maintain, backup is automatic and trouble-free. I know some people will point out that tape rotation is a GOOD THING and that I'm doing myself a disservice by not doing it but to those folks I would point out that for my needs, a home server, tape rotation and multiple backup copies really aren't necessary.
My basement server has run pretty well so far, with nary a glitch. The RAID backup actually DID come in handy about 2 months ago, when one of the drives in the array failed. I admit that I was a little woried but I shouldn't have been. I replaced the dead drive with a similar model and, low and behold, it's still up and running without losing a single bit. So, all in all, I'd say RAID is a decent way to go and well worth consideration.
The issue you raise of backing up raises the questions of what do you want to accomplish? Do you just want to restore if the drive crashes? If so, RAID will give you the peace you seek. Do you want back up because of electrical problems? Floods, fire, vandalism, tornado, Acts of God or other things that go crash on the disk? Back up to a medium that will allow you to take the storage medium off site to a more secure location. ;) Good luck. BTW, I use tape. Daily incremental and weekly full, monthly full and store at work.
If the machine is in the basement, taking the disks or tapes upstairs doesn't help if there's a fire. Keep them with a family member or friend located far enough away that they won't be lost if there's a flood in the area or forest fire or tornado. You need some space diversity (separation) to preclude the same incident from destroying both the orignals and the backups. The limiting factor is how rapidly you need to access the backups. The further away you put them the longer or more expensive to access them in a timely manner.
The length of the operation to back up is relevant. This is determined by the amount to back up, the capacity of the medium (in case you have to use more than one) and the speed at which the media is written to (usually includes time to verify the data). During this process you don't want to be accessing the data or using the machine. This sort of operation is usually conducted at night or early morning to insure unobstructed access to the files and exclusive use of the machine during the backup process. During this time you would normally lock out users to be able to lock the data files while you copy them. Meaning the machine should be left alone while you back up the data.
What do you back up? Unless you spend an endless amount of time customizing the installation, stick to backing up the data. Programs can be reinstalled off the media they came on. You already have copies of them. Your user files and configuration files can be specifically included as part of the back up process if they are that involved. I set up my partitions so that my data is in separate partitions to simplify my backups. I also set up my programs to support that same architecture. This simplifies the process. It does you no good if you go to bed at 11, get up at 7, and the back up doesn't finish up until 9. Especially if you had other processes you needed to run at night, like maybe some rendering.
What will be your schedule? Daily full backups? Initial full back up with daily incrementals, weekly and monthly full backups? How often will you swap out the media and rotate them off site? One way is to do a full initial back up and move it off site. Then every day you do an incremental back up of only those files that have been added or changed and at the end of the week, you do another full. Then you take that tape off site. Keep the initial one as a base line reference. With 3 tapes, you have the initial one and then one full week worth of back up and the incrementals up to the point of failure. If you lose the machine and current tape, you only lose back to the last weeks backup. With more tapes you get more depth. This can be good if you discover that you have been infected with something and the date goes back several weeks. Like the various Code Red variants did.
What software solution is there that appeals to you? Does the software package you want to use support the medium you want to use? If not, consider just writing a script to write to the medium you want. 'Course then you need to figure out how to mark the files you want to backup. That's why they make software packages that handle those sort of issues.
It can be complicated, it can be less so. Figure out what you really want to and can afford to protect against, what you really want to back up and how important it is. From there you can realistically determine the rest. There is much more, like how long you want them to last, whether the system you want has Linux support, and more. This should help you define the problem so that you can solve it. This won't answer all your questions, but there will be addendums to this comment which will point out some of the other short comings.
You must be the change you wish to see in the world - Ghandi
Another idea might be to engage in so-called "Dot Com auctions" where you may find dime-on-the-dollar bargains. If you use compression, you should be able to get 100GB onto a single AIT-2 tape, and you may be able to get it onto a DLT tape (depends on the type). I mention this because there is no need for the newer technology (LTO, Ultrium, AIT-3, etc.) if the goal is to handle 100GB.
The downsides to the Dot Com auctions are three-fold. First, you get equipment "as is". And that really means "as is". So you should preview what you intend to bid on. Second, it can be boring as hell waiting on the item you want to acquire. Third, there is generally a 10% to 13% auctioneer's fee. Hence, if you purchase something for $1000, you are really paying $1000 PLUS the fee PLUS any applicable taxes.
All that said, you can come across some truly amazing bargains...
If you really need to back up >20GB or so I find that DLT is the only reasonable option. You can find used DLTIIIXT and IV drives for less than $500 or so. The media is not too costly, maybe $15-20 per tape. You can fit 15-30 or 20-40 on each of these tapes, so for 100G you probably should figure on your annual 0-level dumps taking about 3-6 tapes, and you should have no problem (with your usage pattern) fitting weekly level 1 backups on a single tape each. So a pack of 10 tapes ought to provide you with a month's backups or more. Total cost around $400-500 if you shop carefully. Maybe less now...
"if you could afford 100GB" - oh come on, disk is pitifully cheap these days, Maxtor 120GB drives can be bought new for under 200 UK Pounds (including our sales tax). Quite a lot of people are getting "interactive digital devices" like PDAs (consider the 5GB disk you can use in an iPAQ expansion slot), digital cameras (consider a 2GB CompactFlash microdrive holding thousands of photos), mp3 players (how big is the iPod?) and dv stuff (Lucas will have everyone equipped with a hard disk based camcorder one of these days ;), with such devices likely to continue to grow in size as well as market penetration, everyone is going to need more storage. Problem is that backup storage has been very stagnant. DVDr goes some way to solve this, but it's not quite big enough (swapping media half way through a backup is not cool).
Chris "Ng" Jones
cmsj@tenshu.net
www.tenshu.net
If you do an initial backup to an extra hardrive somewhere and then just do incremental backups to CDRs chances are that you arent changing or adding 600 megs of data every day.
If you do a full backup to another hard drive every other week and you might go through 30 CDRs a month and worst case scenario for a restore would be one copy from a hard drive and up to 14 CDs.
It might be a little annoying but its cheap and how often do hard drives really crash anyways.
..i just happen to have one here on my pinboard...mhm..its a start!
About three months ago I lost a couple dozen files when Windows 2000 (go ahead and laugh) managed to fuck up it's own file system during or after what appeared to be a perfectly clean shutdown and reboot. That and seeing all the e-mail worms coming from my boss, I figured I needed a real offline backup system.
I bought a used DDS-2 tape drive from eBay for like $40. I added a decent (but old and slow) SCSI controller for about $20. I can get used DDS-2 tapes (4gig) on eBay for about $3-5 each and DDS-1 tapes (2gig) are $1-$3. They claim double the size with compression, but even with data I thought should compress well, I'm only getting about 30% more space with compression.
So what do I use after all that? CD-Rs. It's just too damn convenient and cheap. I burn 2 backup CDs every couple of weeks, one for my main work and one for my personal stuff, including e-mail. The CDs go into an on-site safe. In between those backups, I e-mail things to co-workers, ftp them to my off-site web server, upload them to Yahoo! Briefcase, or just cross my dang fingers.
Akamai probably has terabytes of extra capacity these days. You could store your files there. If Akamai wanted a better revenue stream, they might even market such a feature.
After having been in the business of supporting small business clients for many years, I have been troubled by the situation I see developing. I usually recommend some kind of tape backup as it provides by far the greatest level of redundancy. However, as time goes on, technology changes. Many little shops don't see the need to change such expensive items as backup devices every couple of years but the actual products disappear. There are lots of small companies and individuals out there who have religiously saved off-site backup tapes. If they have a fire, they will still have the tapes but the drive or the software to read them has vanished. This will come back to haunt us. I have seen a few companies go through a near-death experience that was only saved by a good back up. I don't have a good solution to this.
...neighbor A discovers neighbor B's cache of hard-core pr0n is being backed up on his disk... or more prosiacally, their TurboTax files... ;-)
"Biped! Good cranial development. Evidently considerable human ancestry."
After reading this discussion i've come to the conclusion that someone needs to bring up a bit more of the difference bwteen backuping data and archiving. As Many people have pointed out(quite validly) raid is great. As many other people have pointed out, raid sucks. Raid was designed to keep a server running and remove the single point of faliure problem, not as a true backup solution.
With a real archiving solution, all files that have been on the computer at some point should be stored. When you find that you deleted a file two months ago that you really could use right now, a mirrored hard drive will be worth as much as an aol CD in helping you restore. A proper backup must have removable redundant media.
A proper backup should leave a trail of compleete backups in some form that isn't ever going to get deleted. With tapes, this involved keeping yearly/monthly/daily tapes seperatly. With DCD/CD, media is cheap enough to not reuse, luckily. With removable hard drives, on the other hand, for a proper auditable backup trail(When did the hacker break in and steal the cc numbers anyway?) it will take 23 hard drives by my count to cover the year.
This admitedly is a bit more through than the orginial question, but it seems like the discussion has progressed pased the orginal question.
My personal system (once it's finished) will be to reformat all workstations with a 750 megabye user data portion. Everything else should replacable, or backed up by whoever wants it. Mp3's are replacable... Documents/financial information/code is not.
This of course leaves out anyone who actualy generates large amounts of data. If you truly generate large amounts of data, you shouldn't be reading slashdot for how to archive it.
Yes, it could "easily" be replaced. But have fun for the next 3 months changing CDs to re-rip them all to mp3s.
So, in answer to your question... Saving that much time is worth the cost of a new hard disk. Easily.
"We all know was warez the second you said 320 gigs. "
We all know you were archiving warez the second you mentioned 320 gigs.
I never proofread on this site when I post anon, it usually just stays at 0 and never gets read anyway.
Just get like 200 e-mail addresses and back up your stuff to some online backup service that gives you a free 200 megs of free diskspace or whatever. Sure, you'll saturate your internet connection, but what the hey, it's your data, right? 8)
I've had CDs WITH FOIL PEELING OFF still work without a hitch.
Anyhow, I haven't used them in a while, what with my only system with a CD burner being dead for the past year. Since they've dropped in price, have CDRs dropped in quality, too? (Same thing happened with floppy disks.)
though not implemented yet, is to use more hard drives. It's by far the cheapest way to go, and much more flexible than other media.
I had a look and realized I'm going to have close to a terabyte of data in the next 12 months or so... and no way to really back it up.
An extra storage box, separate from everything else, where backup archival copies are kept. One full backup + many incrementals per physical drive in the backup unit.
This will be a unit that is only accessed via backup jobs.. it's not going to be 'used' for anything else. Storage is cheap enough nowadays.
There is always the option of paying for them.
"Movie" does not always mean feature film. I'm pretty sure that by "movie", geekoid meant a more general class of files, that is, any file containing moving images synchronized to audio, such as a feature film's trailer, a Flash movie (YATTA!), a music video, etc.
Besides, a feature film may not be available in a given jurisdiction (think region lockout or shipping restrictions). In the United States, copying of such movies may be protected by "no harm, no foul" (17 USC 107) or the "no suitor, no judge" principle of common law.
Will I retire or break 10K?
Yea, but test that with a rm -rf :)
For my money, it's hard to beat the new ATA RAID cards that are out. Most can be had for less than $100.
Couple that with two or four 80GB drives, for less than $150 each, and you've got yourself a pretty nice array that will keep your data safe against all but the most horrendous problems.
Even with this, you're probably wise to have some offline backup solution to go along with it.
What data would you really want back if your house was swallowed by a hole in the ground? In that situation, do you really need access to your 30GB of MP3 files?
If the anwers is that you really only need access to your Quicken files, then arranging to have those backed up online should be pretty cheap and easy.
Summary: cheap ATA RAID for hardware redundancy, online backup for truly life-critical files.
You know, the funny thing about /usr/share is it is supposed to be where you put "data" which can be shared by users, right?
No. /usr/share and /usr/local/share are for files installed by an application and shared between the different binary architectures the application can be compiled for (such as map files for a game). To share data among users, use /home/johndoe/pub, or use groups.
Will I retire or break 10K?
I currently have about 400GB of online storage on my home network. Being a network geek myself I realize that backups are a must. Granted hard drives are cheap, however there is a finite number of hard drives you can up into a computer. I agree that backup is a must, and if you have something that you absolutely can't lose, then by all means back it up. My servers backup to each other. While this seems like a giant waste of space, and rather difficult to deal with... it works. This strategy has saved me several times, so I see no reason not to recommend it to others.
When devising a backup strategy, most people don't consider the threat of Electromagnetic Pulse (EMP).
A quick google will show you that EMP would be a cheap, relatively easy-to-build terrorist weapon, having devastating effect on our electronics-dependent economy.
A strong EMP would wipe any kind of magnetic media -- tape or disk.
I am by no means an expert. Does anyone know how to defend against it? Would placing your backup media inside a heavy metal safe provide sufficient RF shielding to prevent damage?
That that is is that that that that is not is not.
The trouble with using a second hard disc drive starts when you try backing up your File System Brand Y partition, and realize your second drive needs File System Brand Y, too. Then, when you want to access that data 10 years from now, you can't find software to read File System Brand Y. You're screwed. In the short run, you always have to make sure to NEVER make a mistake and overwrite good data on your backup with bad data. Don't count on beating Murphy's Law.
Tapes are different. There are fewer data formats used by tapes, and thus your chances of recovering data later on are better.
Plus, there is no chance that the file system becomes corrupt on the tape. At worst, you dump all data from tape (possible because files are written sequentially) and pick out what you need. Better than trying to recover a toasted experimental file system from the early 2000's.
As for RAID, that is for redundancy, not "backup." A backup is physically separate from your system, and is not "live." Thus, if lightning strikes your house, your entire RAID array may be toast, but your tape cartridges, off site or on site, are likely to survive.
If you use a removable hard drive for backup, you are resigning yourself to just one or two backup sets. Moreover, off-site backup will become more and more inconvenient.
Don't go with the el cheapo tape drives, like Onstream. Choose quality. Either buy a quality new tape drive if you have the cash, or get a used SCSI tape drive, such as Exabyte, DLT, or DDS-3, or whatever, from ebay or wherever.
As for the supposed problem of corruption of tapes, there's an easy solution. Don't write data to old or used tapes. Buy new, brand name tapes. Furthermore, never assume that your data is actually backed up to tape until you have successfully done a "test recovery" of the data.
Remember that the goals of backing up are peace of mind, and, in the long run, data security.
I am not a lawyer. Do not take my words as legal advice. If you need legal advice, consult an attorney.
I've noticed that IDE devices under Windows 2000 are hot-swappable. As long as you're not pulling out your main HD, Win2k seems okay with that.
Here's how I do it:
- Start -> Settings -> Control Panel -> System -> Hardware -> Advanced -> Device Manager
- Disable or remove the IDE device in question
- Pull the power plug from the device
- Pull the data cable
- Remove device
- Reverse procedure for new device
When you put the power plug on the device, Windows will automatically detect the device and load the driver. It's awesome! I was swapping out CD-ROMs and testing them all day without a hitch.I just wish Linux could do that....
Agreed that just backing up to another HD provides the best overall method for creating a complete backup of 100MB of disk storage.
At the risk of being accused of picking flyshit out of pepper, wouldn't 100MB of disk storage fit onto one zip disk?
You're using her as bait, Master!
I know this isn't exactly related, to the topic, but I was wondering about what good tape backup software exists for Linux. Currently I am using kbackup and find it difficult to use. While tar is simple for backing up a directory, it isn't good for doing a full system backup when multiple tapes are required.
Are there any good open-source GUI-based tape backup programs for Linux? I really miss BackAgain/2 for OS/2 when using Linux.
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
I'm not sure if this violates the Freenet code of ethics. But it probably does. And if everyone started doing it, it would probably kill Freenet very very quickly.
Cryptnotic
My other first post is car post.
for home users, this whole issue would be moot if we all had superfly gigabit connections to the Ether-Net. Then, we could just upload all our crap to the network every night and pay someone a few pennies per day to make sure it doesn't get lost. heck, that's what we do at work...
S.
If hard drives are cheaper and larger than backup media, use hard drives for backup.
Old time admins always used to say that "you buy harddrives to back up your tapes", because the tapes were always less reliable than the harddrive. At work I use cheap EIDE drives to back up data on the SCSI drives on the servers. They are great for that.....
If all this should have a reason, we would be the last to know.
not to contratict you or anything, but unless you are going to bring the extra drive back and forth periodically (and often) it still doesn't address the issue of someone accidentally deleting things (in between the time the extra drive is brought back).
I believe sex is highly over rated... unless it involves me
I have to wonder whether (first of all) why in the heck anyone would need to have 100GB of disk space on a home system
.
Everyone I know has at least 100GB in one ( or more ) of their systems. I'm currently sitting between a box with 7 18GB SCSI drives and one with 5 20GB IDE drives. My friend has ~.75TB over at his house. Most of the space on my box is taken up with mp3's, TV eps and other media. They other boxes are archives. 100GB is quite easy to come by now.
I backup my data using the RASSP (tm) system. RASP stands for Redundant Array of Stupid Slashdot Posts. I cleverly pose my question such that my data is mirrored in the comments section. Of course, the details of this method are protected by patents and trademarks.
This is the solution I've been wanting to implement when I get some extra money. Not a complete offline backup, but probably the best bang for the buck.
:)
:)
Remember, this is not for mission critical stuff, but for mp3's, movies, software installation files, jpg's, etc.
Anything critical (financial files, documents,
etc) should go easily onto a cdr or tape backup.
File Server
$230 4 Channel Raid 5 IDE controller Promise
$151* 80 Gig Seagate U6 (get 3 of em) buy.com
$xxx spare computer with 100MB network card
$100 500VA UPS
$783 Total at least
*Substitute whatever drive you wish.
This config will give you 160GB of space. Total space = (N-1)*X, where N is the number of drives you have, and X is the size of the smallest drive.
Here it's (3-1)*80GB. You have to have a minimum of 3 drives. If any drive dies, shut down the system and replace the drive. No need to backup gigs and gigs of stuff to tape!
notes:
Run an operating system with a Journaling filesystem, (NTFS, Ext3, etc). Don't skimp on the power supply. 250 volts should be enough, unless you are using 5 or more drives or something.
Again, this should be fine for non mission critical stuff! (eg: multimedia files) Just realize lightning can strike and burn your house down, flood, or whatever...
Now if only the Raid 5 IDE cards could come down in price by 50%...
There are more than just cd's and dvd-ram drives to consider, magno-optical media in robotic changers makes for great near line storage.
Currently the state of the art is 9.2 gig drives (I think), but 1.3-2.6 gig drives in 20 slot jukeboxes are readily available on ebay for only a few hundred dollars. Jukeboxes are typically available anywhere from 10-500 slots with space for 1-16 drives. Drives can quite often be upgraded with a firmware update (except HP).
Media is also fairly inexpensive on ebay, both in the write once and re-writeable varieties. Re-writable has the same limitations as cd-rw's in that you typically have to erase the entire platter, but write times aren't too bad, definately faster than cdroms. And read times are quite good.
Changer controller software is available for linux, but isn't terribly good, most is designed for use by tape changers which change tape/platters in sequence.
If you're interested in libraries/changes/jukeboxes look on ebay for optical and any of the prior 3 words.
Somehow, another U of M, the University of Michigan, manages to do this for about 40K students. I don't know what your operating budget is, but, It's certainly not outside the scope of a university. Oh, our IFS space is backed up regularily and you can submit requests for files from tape; so, it's not just some mirrored FC cabinet.
fnord.
Looking over all the previous comments I didn't see this suggestion; the way I do it. ;)
1) Follow the thoughts on doing an offline backup; don't bother with RAID since any hacker attempts would also corrupt the RAID version automatically. Perform weekly manual mirroring to your hearts content. Personally, I have 2x20GB and 2x80GB drives. A 20 and 80 are online. The other 20 and 80 come online only on Saturday mornings long enough to do the backups, then they are taken back offline. This assumes no reasonable hacks have been seen during the previous week. I use a local rsync to keep them matched.
2) Find a friend who also has high speed connectivity to the internet - a brother-in-law nerd works too.
3) Setup accounts and space on each other's servers and perform weekly rsync operations to each other's systems. Basically, you both will have 4 copies ot the data. local master, local offline, remote master, remote offline; Seems like enough redundancy to me. Of course, you don't need to copy everything remote, only the critical files or files you aren't willing to lose.
This covers local/close backups for dumb user mistakes and simple hack validations AND off-site backup needs.
Be certain to use a new version of ssh and set the rsync up to use it!
You may have to help purchase a new drive for your friend if they aren't into this solution. Seems like a small price to be able to sleep at nigth.
at pricewatch $128 buys you a cdr/rw drive & 400pack of 700mb media. This should backup 200GB or 2 backup sets of 100GB even with a few coasters.
You could organize jpgs and/or mp3 by year and mail them to relatives as a present(kill 2 birds w/1 stone).
Then make another set to keep near your home incase you need them quickly.
The initial backup will take a while, but then you only backup jpg/mp3 when the next 700mb threshold is met.
For the rest of the data, backup to a cdrw nightly and rotate offsite weekly.
Pros:cheap/small/mailable/rugged
Cons:Slow initial backup
I'm not sure what a really good solution would be. Using another hard drive as backup is just too tempting to wipe and use for more storage.
I went out and bought a SCSI DDS-3 drive. Each tape holds 12GB. I've been using Amanda to back stuff up. The big problem is that Amanda can't handle the situation where a backup image is bigger than a tape. If you have a big archive of music, it'll have to be divided up into chunks. If you have a database-driven frontend for the music, this isn't a problem (this is where I hope to go...), but people who like the traditional hierarchy systems will not appreciate it.
I actually made a directory that has symlinks in it that point to two separate trees of music (A-M, and N-Z). It's all on the same partition, Amanda just backs up the different directories separately.
I haven't had to recover anything yet, so I don't know how well it works. DDS3 drives are slow, only pushing about 1 Megabyte a second, but I can back up 9 gigs in 3 hours. Full tapes would probably take about 4 hours. The way Amanda works, it balances out the amount of tape used per day. Hmm.. Of course, you'd need something like 10 tapes to just do a full backup. Maybe look at a DDS4 drive, though they were still pretty spendy last I checked.
Unfortunately, backup technology has fallen behind drive technology. Even NASA is having trouble with this stuff these days. They just can't move old data to new tape fast enough to ensure that stuff can be saved.
Well, heck, for the cost of what I'm talking about, you could probably buy a few 100GB drives.. it's pretty crazy.
Thats how I backed up my mp3/warez collection. If I loose a few cds, I know several people with copies.
Regardless of which method you choose to go for, don't forget the principles of the Tao of Backup! "Upon hearing this, the master fell silent." dreaver
I currently have about 90 gig of total home storage...soon going to about 130 gig. But, there's only about 40 GB actually being used for data...MP3's, digital photo's, photo editing stuff, old book reports...you know what I mean. A lot of space is OS files because my total file storage is across 5 machines. Screw backing up the programs! For the most part, I'm going to need to dig out the CD's/disks anyway.
I spread the backups around. Run a script to handle all the machines sequentially. 10 gig goes to a machine down the hall....runs across the fast ethernet wire sometime at night and gets compressed at it's destination. Another 15GB comes off of that machine and is dropped two floors below on a Samba share. Gets compressed too. So on and so forth. One log file gets written...PGP'd and SMTP'd to greet me when I get to work at 6:00 AM.
Yup...it's a pain sometimes. But I more efficiently use the storage without dedicating any one unit. I always leave enough space for other work. I increase tolerance so that if a box dies for good I only lose a piece of the backup scheme. The whole shebang runs while I'm snoozing and can afford network traffic and CPU cycles to compress. And they're all full backups to boot.
I've been nailed a couple of times, but not fatally with this setup. Oh yeah...all the boxes are on UPS's. That's important. I've lost more to the power company than to ghosts in the machine.....
Just get a bunch of Compact Flash or Smart Media cards, stack them up, and throw a rubber band around them... instant backups with your newly created 3d memory!
It sounds like most of your data is static:
- you rip an MP3 once, and that file never changes;
- you rip an album once, and that directory never changes again;
- you save your vacation photos once, and that directory never changes again;
- etc...
Sure, you want all that data instantly accessible, but there's only a small portion of your disk being actively modified daily.
When you rip an album, or store your vacation snaps, back them up. CD, DVD, another HDD - doesn't matter.
Also, backup your actively modified data (accounts, home directory, email, etc) to a separate location whenever you feel it appropriate. A multi-session CD-R works well for me - YMMV.
Maintain a manual index (write a label), and perhaps a speadsheet/text index as well in your home directory (so it gets backed up regularly).
Too easy!
With each breath in, a flower somewhere opens; with each breath out, a flower withers away. In between lies beauty.
This may not be the best way to do it, but it works for me...
I have a "backup" hard drive in my server. This drive is always unmounted so that there is no chance of filesystem corruption from the operating system.
I just use a crontab to run a simple script that mounts the drive and coppies whatever specified backup files to it, then unmounts it. The same method slightly modified could be used to back up this same backup disk to another location on the network on regular intervals.
You might want to look into an Iomega Peerless. The disks are pretty small (maybe about 5"x3"x.5") and the disks are 20 or 40 gb a piece. I'm running one a Windows 98 machine, I couldn't tell you about Linux compatability. It connects to a USB hub and has sustained data transfer of 12 mg a second, I think.
Tim ODonnell (trying to be the most
Ok, so about a thousand people have responded that solution X is a-ok until your house burns down.
Ya know what? I'm not gonna give a shit about some freakin' MP3s if my house burns down. Much bigger fish to fry.
-- Spankmeister General
All solutions come in threes:
First, use at least Raid0,1 for ALL your data. IDE RAID controller cards can be pretty expensive, but if you are in the market for a new motherboard it comes as an extra on a decent board.
THEN, buy a tapedrive for your day-to-day critical data. I know, they are slow, and a reasonably priced one can only store up to 10Gb (uncompressed) but you run these things in the dark hours anyway and use it for the stuff that will stuff you immediately if it goes missing (e-mail, documents, records). Cycle a weeks (or more) worth of tapes, and keep one off-site/fire safe.
FINALY on a regular basis, backup your MP3 collection onto CD-ROM; do the same with the family album etc etc.
It is a hassle, but you sleep a lot nicer at night.
rdiff-backup and RAID it, dude...
http://www.stanford.edu/~bescoto/rdiff-backup/
From the web page:
What is it?
rdiff-backup is a script that backs up one directory to another. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, symlinks, special files, permissions, uid/gid ownership (if it is running as root), and modification times. Finally, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted.
Only solution I have for my terabyte+ at home is mirroring to drives that I rotate offsite. Just selective trees, too. Just about 300GB worth is done that way.
You don't have nearly enough time on your hands!
:-)
.1 mm, since thats what your original measurement was in...
What you really need to calculate is how much a 2.5mm disk is compressed by the weight of 69,443 floppies. Apply the same formula to every other floppy in the stack and then calculate the height!
Once you're done that, find out at what pressure data-loss starts to occur, then redo the above calculations and find the max. height of each stack and how many stacks. You could then redo the calculations using a standard story as a a stack height and find out how much square footage would be needed. Then add aisles for browsing the data.
PS - Also, calculate the height to
A couple of tapes for $15/ea...
backup once a month, or at least every couple of days
(or every Monday after you've spent all weekend ripping CD's)
the above is my personal opinion and does not necessarily reflect that of the little voices in my head
Promise IDE RAID controllers saved my live more than once. Three 100GB drives in a RAID 5 config still leaves with you just under 200GB and a safety net.
/usr/local is precisely for this purpose as you mentioned. Any applications that get built usually default to those directories unless you --prefix=/usr them. NFS mounting /usr/local is the way to go.
The poster asked how to back up 100GB of data (music, photos, etc.).... How much of that is the OS? *Maybe* 1GB tops.
Although you make a good point, it's an insignificant amount of data in relation to the data he wishes to back up.
Kool is selling packs of smokes with a free memorex digital databank... maybe a shitload of those will do it.
Smoke storage... hmm...
would prolly need a surgeon general's warning and end up with a bunch of lawsuits if too much data is lost...
If you mean as part of the OS, check out OS X, it has RAID setup built-in. Works with basically any drive you can mount, and while it might be looked at askance by some interested in industrial means, it should be fine for back-ups, etc.
I currently have ~80GB of which 72GB is a single filesystem. I also have ~240GB in a new case and should start moving the rest of the computer into the new home.. Those 80GB will go to one workstation that currently has way too much software (and data) installed on network disks - software that is not needed on other computers.
The diskspaces mentioned are visible amounts. Current setup is system on mirrored 9GB disk and data on 5*18 RAID5. New setup is mirrored stripe of 3*80GB. Hot spare in both configurations.
On the new system I'll probably have some filesystems (total of about 110GB) for specific uses, but most of the data area will be on one filesystem. That is, there will probably be a ~100GB filesystem for data.
Total visible space for my home network is currently about 200GB and will thus be over 400 when I get the server moved to it's new case - however, at that point I'll be back to some 60% utilization from the current "very close to 100%".
I do have a DDS4 DAT drive. Native capacity of a tape should be about 20GB, but seems more like 18GB (when backing up data that will not compress any further). For me this means that a weekly backup doesn't fit on a single tape. And full backup takes pretty long to take, and I'd really like to have full backup at least every now and then.
Additional drives aren't a solution - I don't trust non-redundant drive setups (I've had more than my share of disk crashes already). Reasonable redundant diskspace costs money, and more importantly, a cheap solution is very limited in physical space requirements. IDE won't work because of cable lengths, which means that I'd need a separate ~100GB SCSI RAID setup for backups and still would need to have mostly static data backed up separately.
For some uses I think DVD-R(W) would fit well:
- readable by mostly any computer nowadays (DVD-ROMs are cheap)
- reasonable space (4.7GB) for most single datasets, and with three or four any single dataset I have
- expected lifespan of a DVD-RW is over 10 years
For some other uses I still need tape:
- nightly incremental backup won't fit on a single DVD-RW but will on the DDS4 tape
- DVD-RW media is more expensive than DAT DDS4 tapes
- weekly specific subsystem full backups can, when well planned, be fit on a single tape per subsystem, and it's reasonably fast so I need to change tapes and push buttons only in the morning and again in the evening
Oh yes, that 72GB filesystem was a pain to fsck. However, journaling fs takes THAT pain away.
It occured to me nobody mentioned online solutions such as Streamload or MyPlay (great for mp3 storage)..
:(
Too bad iDrive & Freespace.com went offline
FYI - Optical media lasts longer than cartridge/Bernoulli style media. So, if you really want that server data, it should go on optical media instead of a Zip. Optical media lasts for > 30 years, while Zip is lucky for 10 years of shelf life.
"Backup is for sissies. Real men upload their data to an FTP and have everyone else mirror it"
http://barrapunto.com/ - News for nerds, en español
Unless you're creating 10 - 100 GBs of *new* data between every backup, it might be simpler (and cheaper) to use incremental backups.
Instead of dumping all 100GB of files every time, 95% of which haven't changed since the last backup, use an incremental backup program to write only the 5% that actually changed. After the initial archive, the backup files will be significantly smaller, and could potentially saved on CDs.
ShoutingMan.com
While it's still pricey, it is within the realm of some home users. It's a fantastic drive, rivaling DLT, and it makes a good showing against AIT-2. The tapes will set you back, but backups these days still means tape. DVD in the best case is 9GB. An Ecrix tape can hold 50GB+ and has a higher write speed than DVD. I've dealt with the company before - they are genuinely pro-linux and are all about making a good product at a good price. Ecrix
hehe, I'll save that for "work" tomorrow :) - Though I think a far more interesting question would run like,
"How many AOL 3.5" floppies does it take to cause said stack to collaps upon it's self - taking AOL with it?"
"Sanity is not statistical", George Orwell, "1984"
I have had great results using my vxa tape backup. It works well with linux, macs, and those other microsoft things. see http://www.ecrix.com/
---- perl -e 'print $i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(
-- Linus Torvalds, about his failing hard drive on linux.cs.helsinki.fi
>> The probability of 3 hard disks failing simultaneously, one not in use, is so incredibly small it's laughable.
One word: Thunderstorm.
Zap, Crackle, Pop. There went your 3 hard drives, all at once.
Yes, it's a minute chance, but I wouldn't call it laughable by any means.
Um tar is only really good for once in a while backups. Long term backups need something little more permeable.
HOwever for the home user tar might be fine as a backup utility.... just a pain to use.
Between my PC, my development server and my fileserver I am pushing 150GB total (with 100GB on the fileserver) - like they said, HDs are cheap (of course I am not using it all).
So far I've just been backing up the important (ie not media) stuff onto CDR. I would like a way to back up everything, but tape drives are too expensive. What I've been doing so far, is keeping a couple of spare 20-30 giggers around and catching bad active ones before they die - obviously far from perfect, but I haven't had major disasters yet.
sic transit gloria mundi
So 100GB drives are getting more and more common. So what? Are backups really any more a problem than when 200MB or less drives were common? We didn't have CD-R back then, much less DVD-R... Only smaller capacity tapes and floppies. And, people with 100GB drives... is what you have on that drive really of more substantial value than what you would have been able to store on a 200MB drive? In my opinion, I don't value what I have on my 30GB drive any more than I valued what was on my 200MB drive "way back when." I think the ratio of data I cared about to data I don't care about was about the same then as it is now. And the backup technology available today can hit that percentage (which I think is about 20%) pretty easily (CD-R) with about as much relative difficulty as floppy-swapping could have back then. Everything else can be reinstalled... no big deal.
Not everyone gets thunderstorms all the time. And when they do occur, it is very rare that you'll struck by lightning! Also, why do you think mains plugs have fuses, and that people use surge protectors?
I had a variety of services running from my personal PC including MP3 servers, ect. running for several years now. For a while at least one drive would fail every year. (Thanks Western Digital) So a long time ago I realized I needed a full backup system that could handle over 10G about five years ago and several times that today. The solution was a full HD mirroring with some extra backup copies.
Every drive has an identical one not being used but connected and powered on. About once a week (usually after a weekly reboot) all of the primary drives would have their image copied over to their mirror. This would simply handle all OSes and file systems (Windows and Linux). In addition every few hours a job would copy some improtant data that changed on a daily basis, mostly just mail files.
This method didn't provide a perfect solution but if I ever took a hammer to a drive and took it down, all you have to do is power off, switch some jumper pins and reboot. In 3 mins. I would be back to where I was a few days ago. Then some quick extra copying would get back the extra incremental stuff, like mail.
What I loved about this was if I ever messed up an install, there was always an image of a good system from a few days ago plus the other important data. More than one a program like NetZero or something would trash my Win95 box. But this would give me a quick go back. It is a little more work than doing Raid mirroring, but also allows you go "go back in time".
This doesn't seem to work for everyone but has saved me more than once. Just my two cents
I've got all the hardware, but I can't seem to find a piece of software that's built for incremental backup to non-tape media. The closest I've found is NTI BackupNOW, but after much frustration I discovered that even their software won't support DVD-R(W) for some months to come.
Has anyone actually **SUCCEEDED** in setting up such a backup system? Bonus question - has anyone had to restore data from this kind of setup?
You know, I can't help but smirk as I sit here and read this. Two years ago, I had a linux system working fine for over 100 days using software raid and two 6Gb Western Digital drives. One day, they just plain stopped working. Turns out there was a defect in the drives that caused the circuit board to wig out and stop working after long periods of uptime. I ended up fixing the problem myself by rma'ing the drives and replacing the circuit boards, but here's the rub... how do you guard against the risk of a manufacturing defect or design flaw? Will a RAID-1 configuration guard against that? Not if both disks are made by the same manufacturer. You may laugh and tell me that the odds of this are so incredibly small that it would never happen to anybody. It happened to me.
I built two servers for the school I work for last year and bought two OnStream SC-50 drives to do daily tape backups using Veritas Backup Exec. The drives both croaked 8 months later. I then found out that the company had gone bankrupt and was bought by a company called "OnStream Data". They said they were a completely different company and didn't have to honor the warranty. I had to pay $50.00 ea. to get the drives replaced.....grrrrrr.....
I have since replaced the drives with Seagate Scorpion 40 GB DDS4 drives, and they work great!
-ted
The real problem here is the absolutely insane rate of hard drive size increases. Hard drive capacity has been increasing at greater than 100% per year since IBM's GMR heads came out. You can now buy a 70TB(yes, terabyte) emc drive array (384 * 180GB seagates)! Maxtor is coming out with a 160GB! ATA drive (once they finish addressing the 28 bit sector address limit). I agree, this poses the issue of "do we really need all of this space?", but data needs will always scale.
Therefore, since the fault of the problem lies with hard drives hugely outpacing every other form of recording medium in rate of capacity increase, the only reasonable solution will soon be (if it isn't already) to use hard drives as the backup medium. Yes, I know, hard drives combine the media with the mechanism and that is normally a big no-no, but in this case I think the monetary facts must be faced. In order to get around the media/mechanism issue as well as the off-site storage issue while not emptying our wallets, I think multi-site dual-hard drive-backup is in order.
At any given site, one would use large ATA drives in the backup server (most likely in an external hotswap cage for the corporates) in place of a tape library/cdr/etc. But, in this case, we should make two copies (use two drives!) to hopefully get around the combined media/mechanism issue. Trust me, the cost of doubling up will be far less than an equivalent media based solution. Off-site fire-proof backup companies could start taking hard drives (being considerably more careful than with tapes, of course). If you want, you could encrypt whatever was on the drive. This scheme scales from the smallest home needs(keep the drives in the safety deposit box) to the largest corporates, and makes the most sense monetarily.
I currently have a 120GB raid0 array (made up of 2 60GB drives). I plan on getting a single, large HDD (preferably 120GB) and a hot swappable bay to do backups with. I went through this whole decision making (tape? cd? dvd?) and realized that a HDD is the best solution. I have a fireproof safe that I'll store it in and just backup data once/month.
:)
ps. burning 30GB of mp3z takes a long ass time. I did that before I made this decision. Do not waste as much time/CD-R's that I did.
In my experience, DLT wasn't a very good way to do backups. Maybe it was just the drive brand we had, but we had to send about 8 drives back to the manufacturer for repairs. Many times with one of our backup tapes still stuck in it.
For some reason, the DLT8000 mechanism was horribly flaky in this way for the first few production runs. You just kept RMAing the damn things until you found a solid unit, then you tried very hard not to breathe on it.
Several times I saw the tape catch miss the tape and get pulled inside without the tape. After that it was back to the manufacturer for repairs.
This is a known failure mode for DLT drives, and only an amazingly lazy tech support droid would suggest that you send the drive back for it. Next time it happens, demand that they walk you through the reset procedure -- it'll take five minutes, tops, and requires nothing more than one screwdriver and a bit of patience.
News for Nerds. Stuff that Matters? Like hell.
Ok here is a good hack: use the mindstorm Lego system to make a robot to change your backup media.... first person that gets one done I will personaly buy them a beer
If you believe statistics, someone in New York is struck by lightning every day.
I hate to say it but tape is still great! I have ~120 Gigs across a number of SCSI Drives. I picked up a Seagate NS20 Travan "refurbished" actual open box, as nothing inside had actually ever been opened, and never looked back. Switching tapes isn't the worst thing in the world guys, and for a home system you don't really need nightly backups. Plus this way you can get your tapes out of the house (leave them at work) or put them in a firesafe in your basement, the possibilites are limitless. Tape also archives better than many of the other solutions being offered up. Just an idea but worth looking into if you need to backup a lot of data.
-OctaneZ
PS. I wouldn't go with anything smaller than a 10/20 compressed drive; I had an 8 Gig Seagate and never backed everything up as it was just too much, but 6 tapes is not bad at all.
so you got a couple 40 GB drives and buy another one to get a RAID 5 (which isn't backup, just a way to stay up). Now what? If a drive fails you are now an idiot cuz you forgot to buy a 4th drive in case one of them fails. So really your price instead of being 150$ is now 300$ and you have to find a store that sells 40GB at 3am...
>Zap, Crackle, Pop. There went your 3 hard drives, all at once.
I'm pretty sure that if the one drive is in a saftey deposite box, the chances of it being struck by lightning at the same time as your other drives is tiny. If that happens to you, I suggest going and buying a lotto ticket.
--Josh
There are exactly 42,935,718 letter sized sheets in a square mile.
This has been my choice for low-cost backup solutions for a couple of years now.
The drives support three different flavours of media - 12GB, 20GB and 33GB, and come with IDE, SCSI, or Firewire interfaces. The IDE is cheapest, at $699, with the media costs being $80, $45, or $35 depending on the capacity.
Is it the absoulte top of the line as far as tapes go? No. But the cost can't be beat. And you get a reasonably fast (3MB/sec) drive with very nice reliability (take a look at the independent testing on their site; e.g. soaking a tape in hot coffee for a minute, rinsing it, drying it and reading the data off.)
They also recently merged with Exabyte, who will be positioning it as their new value solution. Hopefully the Exabyte name will expand the market enough to drive the prices down on these even further.
Matt
This is what I do (man levels of redundancy):
1) Use resierfs -- it'd stable now and recovers better then ext2 in small "incidents"
2) RAID 5 array
3) Run a nightly script that hard links all the files into a . (hidden) directory -- protects against rm -rf''s
4) Run mirror in every directory from cron-- if you lose or mangle one file you cna recontruct it from the contents of the mirror and the other files (works a lot like XOR in raid 5 arrays -- aka a RAID for files).
5) DDS3 incremental back-up; complete backup at regular intervals -- one set of tapes stored off site.
This protects you from multiple levels of failures -- with the catastrophic redundancy being the tapes. You don't always want to rely 100% on the tapes for all your redundancy.
...really appeals to be because I'm a lazy bastard. Being somewhat alergic to physical labor, the thought of swapping anything... tapes, CDs, DVDs, HDs... tends to provoke a fetal-position-slowly-rocking-thumb-sucking response in me. But the prices!? And the storage limits!? Anyone know of a similar service that isn't so stupidly expensive? The post about 802.11b w/your neighbors was compelling... but I live in a human filing cabinet, and my neighbors can barely operate their gas grills, let alone a scary thing like a computer.... But I have to make a decision soon 'cause I just moved and the only computer I own now is this laptop(aside from the POS without a HD I'm using to run LRP). Unless I find something that sounds better, I'm leaning toward a dedicated machine running RAID 5 and biting the bullet and doing [incremental] backups to CDR until (if?) DVD-/+R gets cheaper. If you had a blank slate (no real hardware to speak of and about $3-5k to work with), what kind of networked storage/backup system would you build?
Firstly, I live in Russia. I have a leased-line Internet connection. And I have a verbal agreement that my neighbours pay me some money for Internet after the month if the service is good; I have no obligations except moral ones.
Let us imagine that I have a written contract. This immediately means that I should be registered as an ISP and as the businessperson. Both is a burden that costs much more than my neighbours can pay to me.
And now let us return to the storage problems. If I keep an online backup site I have obligations and am a duly registered businessman and ISP. It's much cheaper for the community to be registered as the "consumers' cooperation" which doesn't produce services and only consumes them, and employ me. Then all the legalities will be the cooperation's President's, not mine.
IIRC, the syntax in /etc/raidtab is: /dev/sdc1
nr-spare-disks n
[...]
device
spare-disk 0
And, if so, can it be done while the system is `live'?
Yes, using the raidhotadd and raidhotremove commands.
Doh! Upon second reading of your post, I realized when you said "mirrorset" you were probably referring to raid 1. If this is the case, then I am probably misleading you. AFAIK, you can only use the spare-disk directive for raid levels 4 and 5.
Enigma
You can get a DLT drive that will do 20-40G per tape for under $200 on Ebay. Add an external case if it doesn't have one, and grab a SCSI card. Cheap, reliable, tested.
Just FTP the files to the SAN at work. I hear they have tape drives and might even do backups!
+++ UGUCAUCGUAUUUCU
Being one of the maintainers of Amanda (www.amanda.org), I'd always been of the opinion that tape backups were the only way to do backups seriously.
/boot on RAID 1 over the 4 disks and / on RAID 1 over 2 of the disks and an alternate root to test upgrades over the other 2, but you get the point). This got me blazingly fast disk access, that tapes would never help me get :-)
The recent explosion in disk capacities and decrease in prices got me to rethink this, just when it came the time for me to set up a home office. When I compared the cost of a reasonably-good tape drive and a number of tapes large enough for me to get at least a month of backups in rotation, and computed how many 60GB disks I could buy with that money, the solution was clear.
I ended up setting up 3 machines with 4x60GB each. They're all on RAID 5, such that if any single disk fails, the machine keeps running (actually, I have
I get all my backup-worthy data rsynced over to the other machines daily or so. I plan to start playing with Inter-Mezzo soon, so that I don't have to remember to run these backups, and so that I don't run these backups on the wrong direction.
But that's not all. With the mind-boggling amount of disk space I could afford, I could (actually, I will, but you get the idea) set up Amanda to backup interesting portions of my home directory to disk, and also replicate this to at least another of my local machines. Such backups can use software compression, such that they don't take as much space as live data. Also, I intend to use another form of compression: instead of backing up CVS trees (I've got loads of check outs), I'm going to back up only local changes to files, so that, in case of disaster, I can still download the original CVS tree and re-apply patches. But this is still a plan, not something I've got running.
Finally, I've got yet another disk on a remote site, to which I rsync not only the interesting portions of my data, but also my backups. I could convince someone else to run this remote backup site for me by offering this person the speed up of RAID 0 over two disks (one of those mine). As for keeping the secrecy of the data on this remote backup site, I'd just get the backup files encrypted, no big deal.
I can strongly recommend this solution: I got pretty much as much data safety as could be expected from a tape-based backup, without any of the hassle of having to switch tapes and moving them off-site and back on-site, and with the bonus of very fast access to local data, unlikely donw-time and fast recovery except in case of total disaster (i.e., having all of my local machines failing, in which case I'd have to either download my backups from the remote site over the net or, more likely, take a replacement machine over to the remote backup site and copy files over a fast local network connection, or from disk to disk.
As for getting 4 IDE disks into a single machine, don't even think of using only the 2 IDE controllers that come on most motherboards these days (for RAID set-ups, you really want one IDE disk per controller). There are a few good motherboards that come with 4 IDE controllers, so that you can even have a CD-ROM and/or a CD-RW in addition to the 4 disks. If you can't find such a motherboard that suits your needs, you can always get one of those PCI cards that adds 2 IDE controllers to your machine.
As for the problem of fitting so many disks in a standard ATX chassis, it can be done. Cooling may be a problem, but a good cooler has been good enough.
All in all, I'm very happy with this arrangement. It was not cheap, but it was not as expensive as a tape-based solution, and it's far more flexible, way faster and it doesn't require any baby-sitting after you get it going. And I can keep far more backup history than I thought it was going to be possible.
That's where all the work is in restoring a system. What do you do about that? It ain't always all in /etc...
Your point is well taken, though. And Debian in particular is organized in a way that it's easier to restore all the configuration stuff.
I ran into this problem earlier this year after a HD failed. I finally settled on backup drives on various machines on the LAN, using rsync to update the backups so that only the changes are copied.
I never had much luck with tapes, and CDR is nice but a pain in the butt when you have 2 gigs to backup.
Brian
Remember Lexington Green!
Go to circuitcity.com and get yourself a few 100 gig hard drive for $120 each (after rebate). Get 2 or 3(you may have to use friends since the rebate is only good for one per household) and then make a RAID configuration with one of them, mirroring your data, then get a firewire kit for the other drive, and plug it into firewire and backup onto it. This way the firewire HDD is not getting its power from the same source, and has a less likely chance of getting fried if your other drives do.
Of course, you could just save a ton of money and use the old "cross my fingers" trick and spend the money on a keg of miller lite, but, hey, its your money.
I think my principles are reachin' an all time low
An external FireWire or USB2 Drive would probably be your best bet. You might have to get install a card to be able to use them, but other than that they would be quick, and if you need real security they would be easy to move to an offsite location. They also don't cost that much when compared to tape drives that would be big enough to be worth your while. Good luck finding what you need.
Greg
What you need to do first is plan on what gets backed up when. Few of my MP3's are backed up as I still have the CD's they were converted from, my digital pix get burned to CD whenever I get a new CD's worth, no software is backed up because I have the install disks, but data files are backed up daily to zip drive.
Moral: Not everything has to be backed up at the same time to the same media, but you do need a backup plan and schedule that accounts for all files *AND* that you follow.
The chances aren't as bad as you make them sound...
Let's just say you have an internal RAID system with, oooo, 4 drives, along with a removable drive to backup everything. The problem lies in the whole trusted/shared medium concept. If there is a surge passed along the case, the SCSI/IDE cable, or through the power-supply cabling, not only will ALL of your drives get toasted, but if you have the backup-harddrive connected to the system (actively archiving your data, finished archiving and waiting for you to remove it, or just because of a BAD practice of never actually removing the removable hard drive) you will loose your backup hard drive as well.
While RAID is a good thing, multiple hard drives are still at the mersey of everything they are connected to. Using such a system as your only backup is a bad idea that happens too often. Having a removable hard drive is an option, but the work involved really makes other solutions much more viable, especially on a large-scale (and on a small-scale, people are lazy!).
I propose a network-based automatic backup system for most people. You simply have your main system automatically backup it's data over the network to another system (systems with low number-crunching capabilities can be put back to work here). Of course, you would want to maintain at least 2 concurrent backups in-case the main system dies during the said backup. The benefit of network backup are that human intervention is not required (the user and administrator don't need to do much of anything after initial setup) and off-site backups can happen transparently (just send the data to the other office down the street, across town, whatever.
Speed of the network may appear to be a problem, but 100 GigaBytes (UNCOMPRESSED) can be transfered in 2.22... hours over 100Base-Tx. First of all, it's likely you'll be compressing that data, which on average halves the size, and so the time is halved as well. Secondly, Gigabit over Cat-5 is available at $45 per NIC, making backups take one-tenth that time. And finally, an encrypted SSH, IPSec, PPTP, (etc) tunnel could be established that would ensure the data is kept private. Data security is much more difficult when you have multiple copies of it unencrypted in a conviently sized package. You are just saying 'steal me'.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
I'm a bit of a linux newbie, but better late than never. I noticed that linux supports RAID 5 in software. For a dedicated file server, this should be no problem.
So that saves the cost of a $250 Hardware IDE card controller. Yay!!!
Now, something that occurred to me regarding the management of "archive" files, such as mp3's, ogg's, and the like. Each time you rip a CD and encode the wav's thereof, you've reduced your storage requirements by nine (roughly). Why not hardlink your archive files to an unorganized ../discXXXX/
directory. Link enough files in each directory
to roughly equal the storage media you're using.
Then, when you have a full "disc", burn it to CD.
We're not talking about an organized disc, remember.
Just faithful backup of new data. Do all of your
heirarchical data management in a set of different
directories, or use a database of some kind. Essentially, you can take out
much of the load that would be handed off to
Amanda or other archiving schemes.
assert(expired(knowledge));
This is slightly OT but since there are so many people posting about RAID right now i might as well throw it out:
I have a scsi 34GB raid 0 array on a DPT i2o card that im borrowing from a friend, and I recently tried to add an IDE drive to my system to wean myself off of the raid array, since i have to give it back. But when I installed the IDE drive, the raid array became unchained or something! i know i know "shoulda used raid-1" but i didn't really have a choice at the time, I needed it for DV editing and couldnt' live with just 17 gigs (*sigh* oh for the days when a 40MB hard drive was huge)
So basically what I'm looking for is an affirmative answer to one of the following questions:
Can I ghost the split drives, and just create a new chain in the card's bios? will the new chain erase all the data on the hd's or if i make sure the config is the same will my data magically appear?
OR, is there a way to take the ghosted images, splice them together somehow and make a new 34GB partition on my new (80gb) ide drive?
Paizurishitetai desu ka?
I was backing up my music collection using DDS2 tapes recently. I had picked up a drive off of eBay for $50 so I thought it would be a perfect match. Media was "cheap" so I thought I was set.
/etc and the like...
My boss was kind enough to give me some old used tapes we had after upgrading to DDS3 tapes a while back. I ended up with 10 or so tapes.
Using NTBackup, I set about backing up 20 GB of MP3s, with hardware compression turned on. Each tape took about 3 hours each, and of course I had to be there to change the tape and then click OK/continue. Well I ended getting a bit over 3.5 GB per tape so I needed to use 6 tapes. All in all it took well over a day to complete the backup, what with switching tapes and the like.
My recommendation is that if you opt for tape backups to invest in a large capacity tape drive. You can get DDS3 drives for $300 or so, and 40GB DLTs for $500. I'm looking into getting a larger tape drive because my time is worth something to me... I don't want to sit around changing tapes all day.
However, the DDS2 drive is perfect for backing up my kernel and
Here's guide I found on Quantum's site on setting up your Linux system to use your brand-spankin'-new DLT drive. It tells you what apps you'll need, and what you'll need setup in the kernel. I took a quick glance at it and it seemed like a good starting point:
LINUX_DLT_Config.pdf@quantum.com (from this page)
There's also an interesting document on changing the "SCSI Inquiry Banner":
DLT_Inquiry_Banner.pdf@quantum.com
This is why for backups you only should take tarballS - just TAR, no gzip, bzip or compress option. Tar is designed to jump bad blocks, this is why you still have the blockish design of the data stream (see manpage). So if you do compression all data behind that spot will be corrupt.
.Z or .bz2) for transfer.
Short: *.tar for backup, *.tar.gz (or similar
My friend and i run a server with 20ish gigs of mp3s, and many pictures as well.
We bought 2 spindels of 50cdRs and went threw and labeled each cd according to how we had stored the mp3s. A cd labeled "full ablums #-Au" would have covered anything that started with a number and Au in the full ablums dir. and anything that wasnot in the system before backing up we just make a cd that says update then the date of the update.
Works well and makes it easy to get to anything specific if just part of the drive goes bad. or great for that mobile mp3 playing cdplayer too... so it could serve multiple uses... backups and to listen to from the backups.
we need a dual scsi controller, raid-1 mirrored machine with hourly^H^H^H^H^H^Hnightly offsite backup (to an alternative location via longrange 802.11 wi-fi networking to our automated tape backup library perhaps?) enclosed in a reinforced fireproof safe with appropriate heat exchange and ventalation systems, power conditioning and backup via dual redundant ups's.
... i forgot ... the mp3's only take up 20% of hdd space, the rest is pr0n =)
just how important are your mp3 collections again?
oh sorry
**AA: a bunch of mindless jerks who'll be the first against the wall when the revolution comes
Data loss. I can buy another computer, but I cannot buy the old emails, savegames, highscores, logfiles, private keys, etc.
Time loss. If, due to hardware malfunction, software glitch or pilot error, recoverable data is lost, it takes quite some time to set up my system again. I don't have that time -- or prefer spending it otherwise.
Data modification -- think virus or being cracked into. You might want to restore/compare the data.
Capturing info if you have a partial accessible HD (Software problems), so you can get images for post-mortem analysis or have more than one repair attempt.
Funny, I was about to post a similar question...
I used to backup my computer systems (Linux on various architectures, ca. 10 GB only in total) to DDS-1 (4mm DAT, 2 GB native, 4 GB with compression) using Amanda, and I liked it. I had a set of 16 tapes, and took backup every night. All I had to do was insert the right tape, and Amanda took care of the rest. The only `problem' was that I didn't dare to buy an additional hard drive, because I wouldn't be able to back it up (disk partition sizes are limited by what fits on a tape if you use Amanda), but I didn't need the disk space anyway.
Unfortunately my tape drive died a few months ago, so I was looking for a new backup solution.
When I bought the DDS-1 (almost 5 years ago), it was not cheap (ca. 750 EUR for the drive), but reasonable, expecially since the low media cost (less than 5 EUR per tape). CD-writers were only marginally cheaper, but the media was much more expensive. The same for cheap Travan drives.
These days large hard drives are very cheap, but tape price/technology hasn't followed. So a new (larger) tape drive with room for future expansion would be (IMHO) too expensive, and I decided to buy a CD-writer instead. Blank CD-Rs are very cheap, and this turned out to be the cheapest solution (price per MB), for reasonable backup sizes.
So now I'm looking for a good automated system to backup my computer systems over the network. Something like Amanda, but for CD-R(W).
Amanda is great, but it has two limitations:
- Partitions have to fit on one tape.
- You can back up to tape only.
Combined, it doesn't make much sense to write a CD-R backend to Amanda, since your partitions can't be larger than 650 MB (a bit more if you use gzip compression). Unless you write a backend that treats multiple CD-Rs like one virtual tape.
Any suggestions?
What I'd recommend is a combination of RAID and vbackup - it gives you intelligent incremental backups on e2fs and similar filesystems. You get a complete copy of your original filesystem for every time you ran vbackup, with unchanged files hardlinked together to save space. This is way more convenient then tape, because:
- It's all automated - no changing tapes.
- I have a full snapshot of what my home directory looked like 3,4,5, however many days ago.
- Restoring is easy - cp -a
- Restoring individual files is easy too. - cp
Depending on what fraction of your filesystem changes on a day-to-day basis, you don't need very much more space. So, you get incremental-type backups with the speed and convenience of hard disks.Unmounting/ remounting the filesystem read-only when it's not in use minimises OS failure, and speeds up backups. So that's nearly all of the above failure modes apart from disasters. And backing up over a network can help minimise that too.
In all seriousness, an advantage of open source software is that there's less need of backups. The few scripts/ programs I find really useful I've also made open source and uploaded to a bunch of web servers I have accounts on - if my computer is stolen/ house blown up, I can just download them again. (There's a Linus quote in here somwhere as well, IIRC).
An EMP of the magnitude required to erase a disk would loose the 'magic blue smoke' from any semiconductor-based electronics devices in the near area as well as in an unbelieably large not-so-near area. As with the burned down house - after a nuclear holocost I think I would have other worries than that my fried computer couldn't play the erased files on my still smouldering hard drive.
I saw somewhere you can get 9.4GB double sided DVD-Rs for $5. I think it was http://www.dvdwriters.co.uk With disks this cheap you can stick your tape backups and stuff .. remember he only wants to backup MP3s not important medical data.
Also having them on DVD-Rs allows you to take them to a m8s house and well you know the rest ;)
Does a solution exist for using another machine for backup that allows the user to "roll-back" to an older version. A common request from users is, "Can you retrieve the copy of this document from March that I modified in August, and was corrupted irrepairibly?" I assume there must be some sort of system that stores just the diff'ed data from Rsync, stores a base image, and then allows you to pick to what date the diffs should be applied to. Kind of like CVS for backups. This would make a second box a truly useful solution for backup without having to store a backup hard drive from yesturday, last week, last month, six months ago, last year, 2 years ago, 4 years ago, and 7 years ago (for FDA).
Why do you need to *backup* your MP3's? they are ALREADY backed up on your ORIGINAL recording CD, aren't they? ;)
Therefore your only worry is your pictures, I'm sure tape or CDR would be fine for this - even if you have several gigs it should be fine.
just do it
Where do you want to be, What are you doing to get there.
The problem is that this is only really guarding against hardware failure. The reason I use tape drives at work is that most of the restores you need to do are because someone deleted something important & then a couple of weeks later someone else discovers the loss. With a single media system you don't get to keep a history. The individual tapes are cheap enough that you can keep history stretching back months.
Well VCR tapes are pretty cheap about .25 cents each at yard sales, and they hold about 100 meg or so at $2.50 per gig youd be looking at... ah the heck with it just commit it all to wetware memory.
I define "backup" as including at a minimum daily copying of NEW data, plus whatever other arrangements are needed to esnsure that you can get your needed applications and data up and running in a new system within the required time. "Archiving" is making long-lasting copies of data that isn't going to change often.
Home computers usually don't have much data that changes frequently, nor do you have to spend a lot of money to ensure that you can be back up quickly -- that is, re-installing all your applications is acceptable in the rare case of a hard drive crash. Maybe only your checkbook records need back-ups. (The ancient DOS shareware program I still use for the checkbook fits nicely on one diskette along with 13 years of data, so I'm all set there.) But your photographs and tax records definitely need archived. The only truly long-term solutions for archiving are rather impractical for most people: printing to acid-free paper with permanent inks, which gets mighty bulky in the long run and might not preserve colors, or etched in stone or metal for _really_ permanent records. The best practical solution I know of for your family jpeg photos is to buy good CD-R's (100-year lifetime claimed), and make two copies on two different brands. Keep one set in your safe deposit box or something, so if your house burns down you still have it. Every 3 years, review the archives -- are the disks holding up, does it look like compatible drives will remain on the market, and will the data and disk formats remain comprehensible to new software? Every so often, you are going to have to copy to new formats.
Note that there are other not-so-standard optical media that are technically better. The trouble with relying on one of those is that at some point you will have good disks and be unable to obtain a drive to read them. It's going to be a very long time until that happens to CD's. Write-once DVD's are worth checking into; I don't yet trust their data stability or longevity on the market, but unless the copy protection @#$%^& screws it up, they are going to be a better archival medium than CD-R's.
Businesses need both archives and frequent backups. Tapes are NOT a great solution for backups. Our e-mail server here had a hard-drive failure recently. (Very much against my own opinions, this used microsoft software and tape backup.) It took four days to install all the software on a new hard drive and restore all recoverable data from several tapes. The most recent backup tape was unreadable, so two day's e-mail was lost. This was not a big problem, but 4 days without e-mail was a very big problem. And what I hear is that this is pretty typical: 20% of tapes from very expensive backup systems don't read back, and it takes far too long to restore from them when they do work.
So for a business to recover from a hard drive crash before it becomes a major problem, you need to mirror the hard drive. (Or if you need hundreds of gigs, use RAID 5.) At the cost of hard drives nowadays, there is no excuse not to.
This protects you against HD failures, but not catastrophes like someone stealing the servers, or crashing an airliner into the building. Many of the WTC businesses actually were ready for that, almost -- they were backing up their servers to servers in Minnesota. The service also included some desks, computers, and phone lines so if the offices in NY were destroyed, the workers could get right back to work as soon as they arrived in Minnesota. The only thing not planned for was planes to be grounded at the same time, so the people had to drive to MN. But I can't think of a better plan for a company that is concentrated in just one city.
If a business has offices in multiple cities, then you can backup the servers to other company offices...
Finally, mirror drives give you instant recovery from many catastrophes, but are no good against the most common causes of data loss: corrupted files and viruses. If you are mirroring, the corruption spreads to the backup within seconds. So you need something besides the mirror drives. Periodically taking off an archive copy of the data may be sufficient. Rotating full and incremental tape backups give some protection, since the chances of bad file + bad tape at the same time are rather slim. Or for the really paranoid, have a whole chain of back-up (not mirror) hard drives -- each night you copy the nth drive to the (n+1)th, so if it takes two days to notice that a file was corrupted, you've still got a good copy on drive 3. Add to this a good off-line archiving system for files that aren't used frequently (so corrupt copies could spread through all the backup HD's before anyone notices), and you are pretty well covered.
Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;)
I'll make sure I tell my boss that we have that as an option when we next discuss backups.
I'm just trying to figure out which one of us would have to explain to the board how our clients' customers' credit card numbers ended up all over the 'net.
On second thought, I'll stick to tape.
They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.-Franklin
This stuff is hard to replace. You know how hard it is to get good porn off of some site, stupid popups and voting systems.
and all it takes is one to fail...
This system gave some ideas about backups...
SmartDAX
[Dr. Mutex LARTs himself]
Ok. I read that backwards. Nevermind.
I suppose you are most interested in backing up MP3 and pr0n?
Cheap, easy and a huge ammount of capacity.
Nuff said
Seriously, what I'd like to know is....does anyone who posts actually READ the previous posts? How many times is the same suggestion (RAID/Removable HD/OnStream/rsync) going to be repeated as if NOONE has suggested it yet?
Is it me? There is no simple, cookie-cutter way to backup that much data that is cheap/easy/idiot-proof. There are alot of good suggestions on this topic, but YOU are going to have to decide which will work for you. You are gonna have to comprimise on something, only you will know what is acceptable.
Good Luck!
WTF? Over?
Safety deposit boxes tend to be made of metal
in a cube shape. I'm sure you've heard of a
"Faraday Cage" - thunderstorms aint a problem!
Multimedia.
People need to get over "home system" meaning low processing/storage needs. Home users store music and movies, play games, etc. A typical home user needs more CPU & storage than a small or medium business.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
Since you mentioned the shared medium problem, remember that the network is a shared medium as well. Earlier this year, I nearly lost all of the machines I have at home due to a lightning strike. I saw a flash across my ceiling and had all of my systems go down hard. (yes, each system has a UPS and anything not on a UPS is on a good quality surge supressor.) The boxes wouldn't even power on after the storm had passed. After a day or two of troubleshooting, I found that the switch and all network cards had been fried. The network cards were keeping the system from powering on (ATX motherboard).
No, it hasn't. In the mid 1990s there was a point where backup was affordable and convenient. High end hard disks were 2 Gigabytes, and $15 DDS2 tapes held 4 Gigabytes (native!) and worked in a $600 tape drive.
It was wonderful. Everything fit and it didn't cost thousands of dollars. I think a lot of people (e.g. me) got spoiled during those few years. ;-)
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
By having the system in the basement, are you protected from a flood? Whatever backup storage solution you come up with, make sure it takes water damage into account. A spare hard drive in a fireproof safe next to the server won't help much if the basement floods.
Burst pipe, water damage from putting a fire out, storm of the century induced flooding...
-LokiFoo
Always keep offsite backups
Still wrong. You see, there's actually 3 different interpretations of MB:
:)
1. 1024*1024 bytes. This is correctly called MiB or MibiByte, but is often mistaken for MB or MegaByte, especially when buying RAM.
2. 1000*1000 bytes. This is the real, SI definition of MegaByte, and is what hard drive manufacturers use.
3. 1024*1000 bytes. This doesn't have a real name, but it's what they use to calculate the size of floppies. Don't ask me why.
So the number of floppies required is:
(100*1000*1000*1000)/(1.44*1024*1000)
Which comes out to 67,817 disks.. Or if you format your disks at 1.68MB then you can reduce that to a mere 58,129 floppies...
Ahh, how fun it is to waste time
-Tom
I agree that all of your ethernet cards are at risk... That still doesn't significantly affect the security or integrity of your data. Even if you have a lightening strike while you are backing up your data, (which kills the main system, and ruins the backup) you should have at least one or more backups available to use.
If you are a large company, or part thereof, you likely hace fiberoptic lines between your main system and the backup. Which is not to mention that Ethernet surge protectors are getting more popular all the time.
BTW, why did you have Ethernet outside where it could be struck by lightning in the first place? Just curious.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
Pffft, try fsck'ing a 384GB filesystem :-) Now that takes a while. (And yes, I have a very good reason for such a large partition (and no, it's not on a home system)).
Well, just to place my hat in the ring... I have a whole network backup method. I use Norton Ghost to backup all my clients to the main file server (this allows for very quick recovery) and then I do a hard drive based (removable hard drive) backup... this hard drive based backup is coupled with a ghost backup onto CD-ROM of the Linux system that I use for the file server... the Hard drive is unmounted and removed and swaped with another drive the next night.... this way I have a very fast recovery time even on my file server..
u're setting yourself up for a huge lawsuit
Do you american have to do everything with a lawsuit. First thing that happens if you do not like you neighbours is
A They ask you to remove the backup. (or more techinacllay they start to send an empty backup for a week, and than a BIG BIG file fileld with random)
B they cut the line. (or put 220 V on it.)
If you turn up with a gun (it is texas?) then you go to court else one of the neighbours moves.
Having the spindles transfered is waaaaaaaaaaay out of the price range of most home users, even many businesses. Plus, you're counting on the platters not being damaged...
Go buy a DVD-R burner for about 500$. At 4.7 billion bytes per disc, burning 2.6mb/sec, it's both cheaper, faster and more versatile than any low-end tape backup system. Of course they work in most DVD-Rom drives, so you can carry your discs anywhere. As a bonus, prices will surely drop within the next year as new models surface.
-Billco, Fnarg.com
True. If it gets struck then definatly buy a lotto ticket.
There are exactly 42,935,718 letter sized sheets in a square mile.