Slashdot Mirror
Affordable Home Backups for 10-100G Systems?
MichaelJames asks: "Ok, I have my MP3's streaming, all our digital pictures up, and a file server running on one machine in the basement. What would be the best way to do simple backups of the system and data? Get a tape drive Get a CDRW or DVDRW to backup the MP3 and pics, but use the old Zip drive for the file server data?" With drives in the 10-20 gig range only getting smaller and less expensive, what are we to do for backups, that have yet to scale well in the same range. For home systems with up to 100G of storage, what do you use to back up that much data, with a solution that's affordable to the average computer user? Have DVD writers become cheap enough for serious consideration as a backup media?
Given that a 100G hard drive is cheaper than any removable media solution, why not just buy another hard drive and install it in a removable (not hot-swappable, just removable) rack?
Racks are $20 at my local Fry's, and inserts for other hard drives are $10.
DVD media is about $6 per 4.7GB disk now, but do you really want to use 20 pieces of media to back up a 100 GB disk?
One thing some people do is back up their HD to a second HD.
Zip disks seem practically useless these days--recordable CD is just too cheap and universal by comparison.
Tape drives are the high-end solution, but expensive.
Just get a lot (A LOT) of 1.44MB floppy disks...
----
WWJD...For a Klondike Bar?
Hey, I've got all these 5 1/4" floppy disks sitting in boxes in a back closet. I bet if I added them all up, they would amount to close to 100 GB.
I don't really think CDs and DVDs really aren't big enough for regular backup of large sets of files. It's just too inconvenient to have to setup a bunch of different 5GB backups, one per DVD (or swap DVDs). The only convenient solutions are to do what the first poster said: use a second harddrive, they're relatively cheap. Or buy a tape drive to store the backups.
Personally, I backup to a second harddrive.
CD-R/CD-RW are too slow and too small, plan on spending a day or so swapping disks. You can always mirror to another hard drive, get a basic RAID card or just use a Ghost-like program to do manual backups. But tape is still cheaper per megabyte and more reliable. Sure, you can damage a tape, but it's harder to do than with a hard drive. SCSI tape drives are more expensive than another drive, but fast enough, and allow you to keep multiple versions or copies of your backup. Try that with hard drives and you need arrays. Tape starts looking REAL cheap then.
Ignorance is the root of all evil.
I have a 100BaseT network, and a server computer that resides in a different room from the rest of my systems. I rotate backups using those aluminum drive caddies. A pair of 60G drives turned out to be MUCH cheaper than the equivalent size tape backup. Every day, I rotate out the drive at the end of the day, and swap with the other. The spare I keep in a fireproof safe. Just tarball the appropriate directories. Done. Poof. Much faster than the average DDS3 tape drive too. Runs at night and I don't even notice it.
I have to say that this is coming from someone with a total of around 280gig at the house, but...
Out of 100gig, how much do you really NEED to back up?
The vast majority of my space is taken up by MP3s (where I converted my CD collection), but that could easily be replaced. To tell you the truth, of the things that I would need (documents, pictures, etc), I could easily fit it all onto a CDR. Well, maybe two. (I take lots of pictures)
Basically it boils down to, do you really need to shell out the money for that extra drive?
:^)
'Life is like a spoonful of Drain-O, it feels good on the way down but leaves you feeling hollow inside'
Actually, what I do is make the new (largest one I own) drive the backup drive, put the old backup drive into use as the primary drive, and retire the smallest one. Just make sure the new drive is as large as the others added together.
CD-R's are OK, but why bother with the hassle? Just run a cron job to copy the files every evening/hour/whatever.
If all this should have a reason, we would be the last to know.
Onstream 30 or 50 GB ADR Tape backup.
Pros:
Can be found for under $100
Linux Support!
Cons:
Tapes are expensive
I got an external usb drive for $248 for an 80G unit. It's the cheapest, fastest, least hassle way of backing up your data. Yeah, of course, you can buy internal drives at a much lower cost but I also needed a way of carting the data to other machines easily. Even though I don't have a USB2 interface, backing up my 40G or so was complete overnight. I'm sure if I had usb2, it would have only taken 2 or 3 hours. Sure as hell beats tape. Tape is a thing of the past. It's NOT cheap.
Ultrium tapes can backup 100GB Native but the price tag is way out of line for small buisness or home use (5000$+ a 100GB drive, ouch).. same goes with any dataloading systems... The only cheap tape backup I've found that was giving the best storage/price (aside from buying those used DSS 4/8 gig drives) is those 33GB Native VXA drives.
--- Metamoderating abusive downgraders since my 300th post.
First, on a typical system, not all data is really worth backing up; the OS and all applications can be reinstalled in the event of a crash (for Linux, it might even be slightly beneficial, as you'll reinstall newer versions and get rid of various cruft you've forgotten you have). Some data has been saved just because it's convenient or simply less bother than having to actively remove it (for me, I tend to collect old logs, various mails I never will look at again and documentation that's several revisions old). A lot of mp3:s and movies may already be burned onto CD:s. That filled 40Gb drive may actually 'only' contain 4-5 Gb of data that actually needs to be backed up.
The data I actually need to back up I manage by having the important stuff an specified directories, then mirroring them over the net to my machine at work. By doing it incrementally, there is little time or bandwith wasted.
/Janne
Trust the Computer. The Computer is your friend.
I once worked at a place where we had a lightning storm. Within a week, about half of the hard drives had failed, out of about a dozen. RAID won't save you then. And how fast can you get replacement hard drives installed, anyway?
All the affected machines were plugged into good UPSes, too.
Moral of the story: Always use offline backups.
Another solution might be to pair down what you backup. It isn't strictly necessary to backup everything on your hard drive. OSes and programs can be reinstalled, but the data that you create with them is the more precious commodity. Of course, figuring out exactly what you need to backup is the problem, and you still lose some information that isn't easily backed up or restored (like settings for programs in the Windows registry). It's taken several reinstalls of Windows before I figured out exactly what I needed to save...
To minimize your storage costs, try using LZip: Lossy Compression. Sure, you won't be able to restore your system to EXACTLY the same state, but you can compress your files to as little as 0% of their original size!
Why are you bothering to back up your data?
/` or a virus)? If so, then a RAID solution is useless.
That may seem like a stupid question, but you need to consider the reasons you want to have a backup before you settle on a method.
Are you afraid of your drive failing? If so, then using a RAID solution should cover you.
Are you afraid of losing your whole system (perhaps due to lightning or theft)? If so, then your backup must be kept physically isolated from your system.
Are you afraid of accidentally deleting files (such as `rm -rf
Are you afraid of having your system down for an hour or two while you replace a drive? If so, then regardless of other issues, you need a RAID setup.
Do you want to use your MP3s with some other device? If so, you probably want CD-R copies.
Of course, there are other considerations that I haven't mentioned or thought of.
The cheapest backup media now is more hard drives. Really!
So I got 4 x 60 GB Maxtors (cheapest MB/$ ratio when I purchased), two Promise Ultra TX2-100 controllers, and set up a 180 GB software RAID-5 under Linux. I'm running ReiserFS on it, so I don't have to fsck 180 GB if I crash the system running an experimental kernel, and I keep the whole thing on a big UPS. Total cost for hardware was about $500.
I mirror data onto the RAID using rsync from my other computers, (or just drag-and-drop to the Samba server from the Win2K box). I think this is cheaper and more effective than a tape backup system for 180 GB of data.
The Linux software RAID gives me the reliability - I've inadvertently tested it when a power connector popped loose from one of the hard drives - I didn't even notice until I read the kernel log (for a different reason). After powerering down, I plugged the drive back in, restarted, did the "raidhotadd" command, and away I went again. No data loss, no hassle.
For stuff that I really, really want backed up beyond the reach of thieves and fire, I use CDR's and a safety deposit box. Luckily there is not too much stuff that falls into this category - source code and documents for projects I've worked on, my email, stuff like that - it still all fits on one CDR.
If you don't use Linux, I think Win2K can do software RAID too. Never checked though.
Torrey Hoffman (Azog)
"HTML needs a rant tag" - Alan Cox
Clearly the answer, for easy backups of a 100G drive, is 21 iPods.
sulli
RTFJ.
There are several programs allowing to use your DV camcorder for backups. For example:
http://dvbackup.sourceforge.net
The 100GB Western Digital Special Edition hard drive with 8MB of cache.
:)
Supposedly, it performs just like a SCSI drive, but it's IDE. A couple of these in the aforementioned rack in a mirrored RAID combo would make a perfect backup.
I'm definitely swapping out my current configuration for two of these once I can afford the $600.
Agreed that just backing up to another HD provides the best overall method for creating a complete backup of 100MB of disk storage.
However, I would suspect that most users don't change a huge percentage of their HD's content on a daily basis, unless you are routinely d/l'ing or ripping MP3s and MPGs on a daily basis (and I note that when I do generate that kind of traffic, it is usually because I am making a compilation CD, and while this does generate a few GB of "new" files on my HD that day, that data doesn't need to be backup up because I've got the original CDs anyway).
As a result, it seems to me that a reasonable solution is to create a "baseline" backup, say to a CD or DVD, at system install time, when there is (relatively) little on the disk, and then each day (or week, depending on needs), do an incremental backup of changed data only to another CD.
This approach is obviously quite inefficient if you have a complete HD failure, in that you have to recreate a new drive by starting with the first backup CD and then restore EACH ONE thereafter until the final CD restores the disk to it's last backed-up state, but for a more common problem of losing or corrupting an individual file, since that is more likely to happen with a recently modified than a remotely modified file, you are likely to be able to restore a last good version within only a few CD's of the most recent incremental backups.
A lot of people have mentioned that disk to disk backup seems to be the best way to go.
I agree.
What hasn't been mentioned is rsync, which makes disk to (local or remote) disk backups fast and easy.
It is trival to set up a second disk that is a "stale" mirror of your primary disk(s) that backs up nightly, and will boot off a floppy. This captures some of the advantage of RAID (quick recovery) while being an actual backup, not just fault tolerance.
Rsync can use ssh as a transport, so you can securely back up remote disks as well.
-Peter
Raid doesn't address one of the other common modes of data loss: catastrophic failure/natural disaster. If your raid gets set on fire, shocked with 1MV lightning, or doused in water, it will probably be completely gone. Offline backups (such as removable, or offsite backups) are much more reliable (it isn't likely your house is going to burn down at the same time your bank is subsumed by a tidal wave).
Another thing that some of us are looking towards is finding a trustworthy friend to share capacity with. If you each buy the extra hard drive (or have space to spare), and rsync nightly with each other, you can get reasonable coverage, and offsite backup. Just pick one reasonably geographically far from you so your data doesn't get sucked up in the same tornado.
Or if you don't trust the "friend", use an encrypted filesystem, or crypt the files first.
I already have 2 30GB drives, and after hearing horror stories about hard drives crashing recently, I've decided my next step is to get another 30GB drive and run RAID5 across them all. Linux can do this in software by the way, and this way you can be assured that your data will stay intact if one of the hard drives crash or not, plus you won't lose half of your drives to backup.
If you are concerned with recovering deleted files, simply use tar or something similar to backup to either a separate directory, or create a separate partition on the RAID array. Another advantage is that you can always increase your storage by slapping in an additional drive, partitioning it the way you want, and then adding it to the current array.
This is really only a short term backup. Since the storage media and the reader are one unit, if either fails the back up is toast. If you have two drives, you might as well setup mirrored RAID.
Real backup is done on semipermanent media (>10 year storage) in a format that can be taken off site easily.
No work at all if you go removable.
(Removable drive rack review)
All of the removable IDE (or SCSI) racks are pretty much the same as the gadget in the review, with minor variations on the theme.
As a bonus, they make futzing around with other operating systems and/or distros (benchmarking, porting, fooling around) a piece of cake, and are a great way to "use up" those old Other uses - sneakernet with 20G removable media. If you live in an apartment and can't h4x0r j00r w4llz with cable runs, it makes loading content onto your "MP3/DiVX jukebox" computer a snap.
I've got two on my "main" machine (one to boot from, one to use as a backup / "gigabyte floppy drive"), and one on each of my "media playback" machines.
Go onto ebay... you can pick up a DLT (15/30GB) drive for around $150. scsi card $50. media are a little expensive (~$25 ea), but for around $275 and a little opensource backup software, you can get reliable backups.
RAID offers good protection for some things: hardware failure (ie: HD crash) and uptime. That aren't the only woes, however... You can loose data in a lot of ways:
Disaster (fire, quake, flood, nuff said)
Hardware failure (disk, controller, ...)
OS failure (FS corruption, ...)
Application failure (User space applications malbehaving, virii, ...)
User failure (accidental deletes, experimental children - trust me on this one ;-)
:)
...) are a plus, but more cumbersome.
RAID will protect you from the second, but will happily add nothing in case of any of the other failures. Backing up to another media is a necessity.
Adding an extra disk (or two, or three), and some tar/cpio cronjobs will add basic protection. (No disaster recovery for you, unless it's off-site
Removable harddrives (firewire, frames,
Tape is considered a more 'trustworthy' backup medium because the mechanism and data storage are separated (ie: tape drive / tape), while in a HD it's in one single package, and it's not as easy to replace the logic board/stepper motor if this flunks. With tape it's easier: just get a new tapedrive.
Anyhow: don't rely on RAID to save your data - it won't.
Okay... I'll do the stupid things first, then you shy people follow.
[Zappa]
RAID won't protect against those types of accidents. Indeed, RAID will happily mirror your screwup for you, automatically. Real backups, on removable media, are the only way to keep a working copy of your system. A separate, permanent hard drive would work too (although it doesn't provide the mental separation between "secure backup" and "live system" that DVDs or tapes would). The important thing is that you perform your backup manually, at a point when your system is in a "known good" configuration.
It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
That's why it's a REMOVABLE hard drive. Kinda like the hot-swappable ones that servers have. Just pull it out of the machine and put it someplace safe when not needed - you know, someplace the cat can easily rub against it!
RAID (even RAID1) is not backup, it is fault-tolerance.
The difference becomes clear when you say ">" when you mean ">>"!
-Peter
What do you do when the building burns down?
Is there something you are trying to keep secure?
Why do you want to keep your data safe?
Is an encryption device utilized with a harddrive or an application?
Where did you obtain all of your software?
Are you looking to copy to a device that has the ability to encrypt files?
If you are looking for a portable back-up device, why do you need it to be portable?
Do you travel extensively?
When you do travel, do you primarily travel by air?
Do you have a digital camera?
Do you have a mobile phone?
Have you ever encrypted an email message?
Have you ever deleted an email message?
If so, have you had data rewrite over the sector(s) containing such message?
What was the title of the last book you purchased?
"There ought to be limits to freedom"
Of course, data file should not be installed on a local drive, so that you can implement some sort of a disk imaging solution for the base installation. The disk image should be of the main drive with the core installation folders or mounts.
This way if someone screws up the system, you can blow out the main drive, replace it with a known good config, and then add the two or three apps you need, with the datafiles safely someplace else. This could even be done from a bootable CD, if needed.
"It is a greater offense to steal men's labor, than their clothes"
I'm sharing my cable modem via 802.11 with all the neighbors and since I am the local "neighborhood helpdesk technician", they often come to me for advice. Recently, one of them wanted to know how to go about backing things up properly. It dawned on me that hard drive space is abundant and most people are buying much more than they need (the person in question has an 80 gig at about 20% capacity). So I worked out a deal so that everyone is backing up to each other's PC at night on a weekly basis. The 802.11b connection keeps drive thrashing to a minimum yet provides enough speed for complete backup on an overnight basis.
I should start charging for these ideas... Can't wait for the proliferation of freenet!
Life is the leading cause of death in America.
Magnetic storage : the data is sensitive to electromagnetic fields, heat, and the magnetism in the ferrite particules decays over time (i.e. the data has a reasonable chance to get corrupted after 5 to 10 years, even if the media is properly stored). Also, reading a magnetic medium can wear out the medium, but that's not an issue if you just want backups. The bit density on these media is good, and the price per megabyte is excellent.
Optical storage : there is no decay in theory, but I read somewhere that pressed CDs actually have a lifespan of 20 to 30 years even when properly stored, and CDR(W) even much less, especially when stored exposed to sunlight. I'm not certain there is much real-life data on the durability of DVDs. The bit density is good, but not as good as magnetic storage, and the price per megabyte, I guess, is comparable, but mastering optical media requires more effort.
I'm not considering the fact that magnetic storage media are re-writable, since you're talking about doing backups. So I guess, the question really boils down to whether or not you want to have backups that are more reliable over time, and whether or not you want to spend the time and effort to create CDs or DVDs of your data.
Then of course, there is the question of obsolescence : CDs have been around for years, and I don't think they're going to disappear any time soon. However, tapes for example can become unreadable because nobody makes readers anymore (ever tried to re-read a 80Mb Colorado tape recently ?). Same goes for hard-drives, although I'll admit IDE and SCSI interfaces will be around for quite a while longer.
Then of course, if you really badly want your data stored safely for the longest time, you can get yourself an old punchcard deck, a very large box of punchcards, and there is a fair chance that some archeologist in the year 4000 will find them in a dig just above where you house used to be ;-)
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
For all those moderating the posts about setting up a RAID system as 'Redundant', please think first if such moderation is not 'Redundant' itself, since RAID is obviously 'Redundant'. Hmm. OK, it could be RAID-0, but noone is talking about stripping. :)
Looks like the cheapest Kingston/StorCase offering (which supports ATA/100, bonus!) costs around $80 mail-order.
Are there any other reputable manufacturers that sell a cheaper solution for IDE?
o/~ Join us now and share the software
There are two ways you can go relatively cheaply, and IMHO a far better solution than CD-R or CD-RW.
Pick up a DLT2000XT (15gb native) off ebay for about $200. Tapes are dirt cheap, about $5/ea and the media is extremely durable, nearly indestructible.
Pick up a DLT1 (40gb native) off ebay, about $500. Tapes are moderately expensive at around $20/ea, but again the media is extremely durable.
DLT is industrial strength backup, the drives are built like tanks and the tapes can take incredible abuse.
Its all standard SCSI and works great with linux, no problems whatsoever.
I considered buying hard drives for backups, but they are far too fragile for long term backup and off-site storage. Most drives arent designed to be spun up and down lots of times either.
Last thing you need is for your backup harddisk to go splat when youre trying to power it up to restore your main system from a data loss.
With DLT, this isnt likely to happen.
For my home network, I use removable internal hard drives. There are several manuafacturers that make units that will turn an IDE or SCSI drive into a removable unit. I've got a dataport IV which has a component that fits in one of the 5" bays in my PC and connects to the motherboard via an IDE cable. A second component opens to hold a standard 3" IDE drive and plugs into the first component. I've got several of those. Back-ups are straight-forward: 1) Shut down the machine, pop in a drive module. 2) Boot 3) Do back-ups either locally, or across the network. 4) Shutdown, pop-out drive module and place it in a drawer.
It would be nice if I could hot-plug the drives instead of having to reboot. If I was more thorough (or paranoid), I'd take the drive off-site and put it in a safe deposit box. Dataports are made by CRU-INC, but there are other similar products.
[Insert pithy quote here]
I know this is a windows solution however I'm running two w2k advanced servers and have one on site one off site at my home.
They are clustered. First created with a 100mb crossover link. Then moved one offsite with a modem link between them (direct modem to modem). AS the files don't change much there is not a lot of traffic. (1 file in 3 days maybe).
There is also software link doubletake and surviveIT for the corporate world.
The MyTh - I am a figment of the Imagination - [Im Probably even not here]
Bringing up the system is less of a problem with newer OSes, since you can usually, at minimum, get to your data. Configuring the database, webserver, and firewalling depends on how good you are with the OS. However, when I worked at a former company there was no real plan to get a working system back in place. We were using Novell with Arcserve -- unfortunately, you couldn't get to the data without a working system.
Next I usually try to segregate rapidly changing stuff versus things that are pretty much static. E.g, my mp3 collection is relatively static. I occasionally buy a fresh CD and rip it, but I'm pretty much satisfied with my collection as it is. I put these on CDROM. It takes a while to create them, but it's cheap and safe. If you want to keep everything up to date, you can run a script to save only files not included on the CDROM.
Finally, I back up my constantly changing stuff such as CVS, MySQL database, etc. to 4MM tape. It's cheap (hardware and tape) and most drives are pretty well supported.
but sumthin like this was just posted on ask slashdot recently...
actually, most of the points made in the earlier story apply here.
and as i'm sure others will/have jumped in to say, there's a big gap between failover/redundancy afforded by a raid setup or external hard disk and tape backups. it, like so much else, depends on what you need.
i'm pretty discouraged personally. i don't really see an affordable way to do real backups of a couple hundred gigs of data. it's probably going to have to be a mixed setup. most of the data is static: flac & mp3's. so maybe that just needs one offsite backup that's done to a couple hard disks--basically a mirror. then get some kind of tape system for the other 40gigs or so of slowly or quickly changing data that i also want to keep a couple snapshots of for historical purposes, as well as rotate offsite. still not sure what the latter solution should be.
i have an old dds2 drive. but that's 8gb max compressed. dlt or ultrium is way out of budget. maybe an onstream. but i don't know anyone who's actually got one--any firsthand reports on these? and how likely is whatever onstream uses likely to exist/be supported in say 5 years or beyond?
Even with grip, ripping, categorizing, encoding more than a few cd's is a pain in the butt.
Jesus was all right but his disciples were thick and ordinary. -John Lennon
If you need to protect against fault tolerance then just get another hard drive. IDE RAID on the cheap can be done via software or another drive controller.
However, if you need to backup data with the ability to rollback changes or deletions then you are most definitely looking at a tape system.
Keep the Classic Slashdot.
Currently we've got a storeroom full of boxes and boxes of CDRs. I want to make an archival backup of them.
Negatives will last for a hundred years if they're stored properly. Just the other day we made reprints of one of the first sessions our studio did, more than 20 years ago. However, CDRs do not last this long. Assuming nothing catastrophic (fire, CDs breaking) happens; CDRs are only made to last a few years at most. (Rough estimates put the shelf life of CDRs at 10 years or so.)
What form of storage is the most archival?
Consider that we've got probably a thousand CDRs sitting in boxes. Some way of doing a batch backup (CD towers?) would be great.
Username taken, please choose another one.
I'm classifiable as an audio addict, having taken my entire personal
/boot
/home
/pchome
/pub
/pub/mp3
/scratch
/pub/mp3_2
/pub/software
/etc/cron.hourly/rsync_with_fumus script:
/pub
/pub
/pub
:-)
collection of CD's and ripped them to MP3's at 320 bit, and wanted to
have them stored in a central place, accessible from any machine in my
home. Currently this collection is at approximately 620 full CD's of
music, and I'm pushing right at, or just above the 80 gigabyte limit.
Now when you factor in personal files, financial records, games,
downloaded material, installation software you don't want to lose,
etc...etc... Well, see for yourself. Here's my space breakdown for the
partitions on my main file server Fumus (Smoke, in Latin):
fumus:/pub/mp3 # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda3 3.0G 2.1G 804M 72% /
/dev/hda1 129M 6.8M 115M 6%
/dev/hda5 9.8G 1.8M 9.3G 1%
/dev/hda6 20G 13G 6.3G 67%
/dev/hda8 40G 22G 17G 57%
/dev/hdb1 75G 38G 33G 53%
/dev/hda7 1.9G 20k 1.8G 1%
/dev/hdc1 74G 34G 40G 46%
/dev/hdd1 74G 36G 37G 49%
So, here's what I looked at:
Tape: For the size I'd need: Way WAY too expensive. When I brought
the media down into the range I'd afford, I'd be swapping tapes all week
to get a backup done. Not time effective.
CD-R: Faster, yes, but at 650 megabytes per media, same problem as
tape, only you've traded magne tic for optical.
Extra hard drives in the same machine: Originally, this is exactly what
I had done with a single file server running Reiser file systems in the
more experimental days. I got the scare (and lesson) of my life when
Reiser went a bit nuts, and started corrupting some of my data. I only
lost about one percent, but I vowed, never never NEVER again would I
backup data on a critical machine on live media in the same machine.
Okay, so here's what I finally DID select as my solution: A second
machine called Ignis (Fire in Latin) that uses the absolutely identical
configuration, right down to the types and number of drives, partition
sizes, everything. They both connect into my 100Mb network switch, and
Ignis rsync's from Fumus every hour on the hour thanks to scripts in
/etc/cron.hourly
In fact, here's Ignis'
rsync -arul --one-file-system --quiet fumus:/pub/mp3_2
rsync -arul --one-file-system --quiet fumus:/pub/mp3
rsync -azrul --one-file-system --quiet --delete --force fumus:/pub/software
rsync -azrul --one-file-system --quiet --delete --force fumus:/pub /
rsync -azrul --one-file-system --quiet --delete --force fumus:/pchome /
Is this a bit extreme? Yes. But... if, gods forbid, Fumus really does
let out its magic smoke, or Ignis does catch on fire, and the physical
media were actually damaged, hopefully the damage would be limited to
*one* case, and wouldn't end up taking both machines out. Then I really
would be crying the blues.
Oh yes, and each machine is on their own 900VA UPS. I'm not playing
THAT game.
Another principle of good backups is to have another copy in another location, since having an extra hard drive won't help you if your house burns down.
OK, so buy more than one back up HD for each application and don't leave it in the box. If you want, take the extra copy home or put it in a safe deposit box. His point is that the extra HD is still cheaper than removable media for most people right now.
This is all lost on me right now. The largest projects I have easily fit on a CD, source executable and data files uncompressed. My photos are largely segregated by where and when they were taken and are archived that way. The one or two home movies I've made are no more than 10 minutes long and of such pathetic quality that they are less than 100MB each. This may change in time, but right now the 10 cent CD and a central FTP server are more than adequate. Thingy goes to FTP, then gets CDed several times and removed.
I doubt I'll ever pay to have someone else paw through^H^H^H^H^H^H^H^H^H store my data.
DMCA, Hollings, Palladium. What might have sounded like paranoia is now common sense.
Not to distro-bait, but Debian in particular shines here because apt makes it so damn easy to bring a system back to the state you wanted. For myself I have created a meta-package (.deb) which does nothing but depend on the applications I want installed on every desktop system: galeon, gnucash, xchat, gaim, xmms, vim-gtk, and a handful of others. Then I back up my meta-package, all of 10k including a few shell scripts I wrote for myself. Install my meta-package on a new system, and voilá, apt fetches and installs every app, that I need to continue working, dependencies included.
It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
I don't think you even need a new directory. I believe tar can do incremental backups. I know there are some Win** utilities to do similar.
Also look at this comment which offers very similar advice.
Jesus was all right but his disciples were thick and ordinary. -John Lennon
Backup has always been expensive. You have two choices:
1) Go cheap (i.e. Zip drive, MO drive, CD-RW, etc.) and only backup the files you NEED (i.e. home directory, "My Documents" folder, etc.)
2) Shell out for tape. This way you will be able to make multiple backups, keep them offsite, maintain them long-term, and back up most of your system. I find that the sweet-spot is usually to go 1-2 generations old (i.e. right now I'm using a DAT24 drive), that way you only have to pay about $400-600 instead of $1500+ for the mechanism.
Don't fall for the "just use hard drives" trick. Hard drives have a number of problems.
1) Mechanism + media in one unit. I've seen hard drives whose heads STUCK to the platters, rendering them useless, after only sitting around for a couple of months. Oops! Data gone! They are also sensitive to static and to environmental changes -- if your backup drive gets zapped accidentally, you can't just plug the media into a new mechanism! Data gone!
2) Backups limited. If you backup a 100GB system onto a 100GB hard drive, you're limited to one physical backup. This creates all kinds of problems... Many's the time I've had to go back four or five generations in backups to find an old file that I didn't realize I'd deleted months ago. Not to mention that you are limited to one backup stored in one location -- no redundancy.
3) Even RAID has problems -- I've had bad SCSI cables that filled a RAID filesystem with corruption before we were able to track down the problem and switch the cable. If that happens when you're just depending on RAID to preserve your important data... oops! There's a reason why many RAID-enabled datacenters also maintain backups of critical data in a second medium!
So that's my personal take. I think either tape, maintaining at least 5-10 backup generations at a time, or MO/CD-RW just for keeping your critical files. Or both, even.
STOP . AMERICA . NOW
Right now I _encrypt_ my files and put them on a CDR. So if I loose the CDR, I can be sure no one is readking my journal :-)
NOw, I always wanted a
- reliable
- cheap
- plenty of
- unix friendly
online backup. I tinkered with myspace.com and virtual hard drive and they are not worth while
- for the amount of spam you get
- the tiny space (10Meg, yeah right!)
- most of them want to install clients (win xx only)
Does anyone know any good alternatives? I think this is a lucrative business. Can not understand that no good service is available!
thanks
LInuxLover
Most of you are missing this problem (fire/theft). What are your various solutions to account for and protect against this?
h ol d=1&commentsort=0&mode=thread&cid=2689576
I use a removable drive that can be taken to another building or put in a fire safe. Any other options out there? I'm sure we're creative enough to have some decent options.
More info on mine (don't want to re-type)...go to
http://slashdot.org/comments.pl?sid=24768&thres
(sorry, you have to cut and paste cause I'm lazy)
For instance someone recommends OnStream, the 30GB or 50GB models.
Well, those models are really 15GB and 25GB, the 30GB and 50GB numbers are based on expected compression.
Of course you can use the same compression to fit more on ANY backup media!!!
Why not call DVD 4.7GB media as 9.5GB media?
Or CDR media as 1.4GB?
Remember JPG, MP3, AVI, MPG, ZIP, RAR, GZ and so on will not compress at all. So you begin to see what a farce tape capacity is.
Given the REAL capacity, how much do you pay for a Onstream tapes per real GB?
15GB for $30
25GB for $40
So the best price is $1.6 per GB.
Then there is the drive cost, figure $300 for the 50GB model. So let's say you buy 12 tapes, and a drive, 300GB of backup storage for $780.
That's a real cost of $2.6 per GB, for data that can only be read on another tape drive, and is not random access.
Harddrives are cheaper per GB, faster, and can be plugged into any computer. The price keeps falling everyday. So only buy what you need now, and pay much less in a year.
Uncomfortable with IDE? Go Firewire, it's hot plug and play (no reboot), about $3 per GB (and falling).
First you must give consideration to what you really need to backup. I only do my home directory. I have a tar/gzip of /etc there. I burn my mp3's to cdrom as necessary. How much stuff do you really need to backup after that?
RAID 5 is not foolproof. I just had a server lose one disk which caused a second disk to go offline which toasted the whole array. I restored from tape.
Extra disks are not foolproof. I tried to boot SCO unix the other day and the harddrive no longer functions.
Tape has been the most reliable as long as it was written with a tape program that still exists. I have read 5 year old BRU tapes successfully.
I still use a 4MM DAT. It's small, so I don't backup much, only the necessities. But I know I can get it back!
PK
dvbackup is a utility that lets you store up to 13gb of data on commodity miniDV tapes. With the use of a Digital Camcorder.
:(.
This is not exactly a dirt cheap solution, but if you have/want a digital camcorder anyway, there's only the cost of extra tapes.
Make sure the camcorder works with dvbackup before buying one though. It doesn't work with any JVC's that I know of, or at least not mine
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-U
for i in `cat rsync.list| egrep -v "^#"` /vol/backup/$HOSTNAME/$DATE
/vol/backup/$HOSTNAME/$DATE
do
HOSTNAME=`echo $i| awk -F: '{print $1;}'`
DIRECTORY=`echo $i| awk -F: '{print $2;}'`
DATE=`date +%A`
install -d
rsync --numeric-ids --compress --rsh=/usr/bin/ssh --recursive --archive --relative --sparse --one-file-system --compare-dest=/vol/backup/$HOSTNAME/current $HOSTNAME:$DIRECTORY
done
Then once a week we run a similar script that updates the 'current' directories and uses --delete
(rsync.list contains entries like "hostname:/some/mounted/partition")
Get high-bandwidth Internet access, buy two more hard drives, find two other people who do the same and a few kilometers away from you (each one in the opposite direction). Start exchanging encrypted backups (for example, tar files postprocessed using GnuPG).
To backup a 100GB drive, you require...
- 6 DVD+RW (18 GB) discs, or
- 20 DVD-RAM (5.2 GB) discs, or
- 158 CD-R discs, or
- 72,818 HD 3.5" floppy discs
My car gets 40 rods to the hogshead, and that's the way I likes it!
I have over 100 GB of hard storage, however most of that is mp3s, videos, and other things that do not change. I have burned on CD all of my mp3s and other things that don't change, but I don't want to live without if my hard drive goes down. Then every night I have a script that does an incremental backup and zips it, PGPs it, and then ftps it to my friends hard drive, both at college with a fast connection and a 100 miles apart. Also once a week I do a full backup. I figure the chance of both of are computers getting struck by lighting, bruning in a fire, or failing at the same time is pretty rare.
Its not the best of all solutions, but I don't have anything I would die without and it was really cheap (free).
This is one segment of the computer market where the industry has failed to provide a solution.
And so - I will continue to hear stories from co-workers, friends, and relatives about the x months of data that were lost when their computer crashed.
Some hotshot venture capitalist with some geek buddies ought to jump at this opportunity.
Manufacture a removable storage - external device - writing to cheaply manufactured slower hard drives (why spend top dollar on 10,000 RPM drives when you're doing backup as a batch process overnight? - oh that's right, because the industry doesn't make slow, high-capacity drives anymore - they assume that all applications are high-end).
Bundle the box with some cross-platform data management software. Users could just plug in the box and set it off once a week. The rest of the time, keep the box in a fireproof safe, unplugged, or something like that.
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
Didn't they go out of business awhile back?
RAID is to provide either additional speed and/or hotswappable capability. RAID really stinks as a backup, since RAID doesn't care when some program deletes most of the hard drive, when some user removes too many files, or when the OS barfs. Sure, RAID will save your DATA if one HDD fails, as long as whatever caused it to fail didn't affect the other drive, but for the reasons already listed, this doesn't mean RAID is a valid method of backup.
However, a HDD in an external enclosure could be considered a valid backup, however, for true redundancy, you better have two drives you swap, and you better be doing surface tests regularly. A drive, properly treated, should last many, many years. Also, you could combine a drive with monthly or quad-yearly backups to CD-R, just make sure you do your research on the inks used in CD-R disks, some don't last as long as others.
Just my $.02
What causes an EMP? Nuclear explosions. With all the terrorists and their suitcase nukes around, you're magnetic data's not safe! Go Optical. Better yet, break out the punch-cards!
</fearmongering>
Seriously, if your backup media is magnetic, be it a tape, mirrored harddrive, or a vast pile of old AOL 3.5" floppy disks, you've got to watch out.
Case in point: A company I know of stores off-site tape backups at a reputable insurance firm's lock-up. There are all sorts of gurantees against fire, flood, tornados, etc...
What there is not a guarantee for is Larry, the night watchman, who brings his ancient portable television up to work with him every night. He sits it directly on top of the tapes he's supposed to file that night.
The previous didn't happen, but it *could*. There are all sorts of accidents that can corrupt magnetic media that wouldn't harm CD or DVD media.CD-R's are especially cheap, and you can reasonably back up everything short of massive databases or large AV projects on one. If you regularly make massive databases and/or large AV projects, you can probably afford DVD-RW.
Tape is getting obsolete and DVD's are getting cheaper all the time.
The next Slashdot story will be ready soon, but subscribers can beat the rush and slashdot the links early!
What happens when your intern scratches your delicate DVD?
I forget...are we at war with Eurasia or East Asia?
That worked really well for backing up our 80MB drives onto stacks of 1.44MB floppies, since you would really only need to insert about 5-10 floppies during your weekly backup, just to get the files that changed.
So why not just do incremental backup onto CD-Rs? Even with 100GB of archives, most of those are static. You probably won't need to use more than one CDR per week (maybe two) to track the changes. It's cheap, relatively painless if you've got the right software (and it wouldn't be hard to throw together incremental backup/recovery scripts in Perl if you're into that sort of thing.) and you've probably already got a CD burner.
If less than 650MB of files change in a week, the rest of the CDR can be filled up with files that were on earlier CDRs (this way your backup set can remain finite and you can throw out the earlier CDRs as they become obsolete. Or if you keep them all, you can reconstruct that state of your hard drive at *any* time, not just at the last backup.) This seems ideal to me--why is everyone else talking about expensive solutions like tape drives, DVD-RWs, and second hard drives?
I have a positive modifier on Troll. When I mod someone Troll their karma should go UP!
I have to wonder whether (first of all) why in the heck anyone would need to have 100GB of disk space on a home system. But then I have five systems networked together and have more storage than I would have thought sane a few years a go though I have a bit of a ways to go before I will run into the poster's backup problem. It wasn't too long ago that, if you could afford 100GB, you could probably afford a SCSI array controller that would let you do a lot of RAID, hot swapping, automatic drive replacement, etc. With today's cheap disk prices you don't have to be wealthy to have an ocean of disk space. (I can remember the days when we thought having 900MB on a MicroVAX II was extravagant.)
You could always do it the traditional way and get some tape drives. Unfortunately, they're much more expensive than you might think when you have to backup that much disk space. You certainly wouldn't want to go cheap and be feeding 90m DAT cartridges into a drive all night (it'll start feeling like you're backing up to floppies before long). A good high capacity tape drive can get, what, 20GB onto a single cartridge? Not bad. And I think that at this point in time, tape is more cost effective than DVD-R. (Something tells me that the MPAA, and maybe the RIAA, will try to keep it that way too.)
Mirroring disks can be helpful. Hard disks are getting cheaper and cheaper. Heck it's almost scary mow much disk space you get in a typical PC sold at Best Buy nowadays (and without a backup device; it's almost criminal). If you're running mirrored disks you'll forestall the inevitable disk crash that takes all your data with it. Question for the Linux folks using the `md' driver: Does it allow adding a third member to a mirrorset? And, if so, can it be done while the system is `live'? (The third member gets removed and taken offsite in case there's a disaster.)
One final thought: The poster wasn't actually running a 100GB filesystem were they? I'm thinking that a power glitch could cause a world record to be set for the longest fsck-on-reboot run. Plus I'd think that backing up such a beast would be a challenge. I tend to keep my filesystem sizes no larger than what I can fit on a single tape cartridge... just to make life simple. (I'm used to having to pipe `df' commands through `more' at work so I don't mind lots of mount points. :-) )
CUR ALLOC 20195.....5804M
There are lots of good suggestions about RAID, copying to another drive, etc. But as long as the backed-up data is stored in the same physical location (home) as the original, you are not safe from natural disasters or theft. Maybe this is paranoia but it depends how important your data is to you.
I have isolated the data that changes on a regular basis that I want to back up. Mainly this consists of mail and financial data, in the 10-20MB range. I have 2 boxes so each box backs up data to the other box. But just to be safe I back up to a trusted friend's machine also. It's a large ftp transfer but not bad at all with cable modem.
For other data that I want to save that doesn't change regularly (photos, mp3s, etc) I use CDRs.
What am I supposed to do, VIDEOTAPE all of the stuff as I open it up in Windows? Like I'd want to type all that code back in by hand.
That would have to be one hell of a scratch to damage it. Surface scratches can be repaired, you can buy the stuff to fill in the scratches at computer/music stores. The scratch would have to be deep enough to actually damage the metal within; unless your intern regularly plays with belt sanders in the office I wouldn't worry.
Additionally, it's a lot easier to protect against scratches; i.e., if the disk is out of physical harm's way (you know, like in a CD case!), it won't get scratched. It's a lot harder to defend against magnetic damage (e.g., you would need specially shielded cases or whatnot).
Liberty in your lifetime
One of the biggest problems with any other sort of backup scheme is that you have to do it and you will usually have something better to do. With a dedicated machine running only this sort of thing you are much more likely to get it done than find the time to burn 20 DVD-RW disks (which will set you back $700 btw) or load 3 or 4 tapes or... The only downside on this sort of thing is that in the corporate setting, this is not a satisfactory "off site" solution.
Publish everything to the web and let Google cache do it for you...
That is all.
3ware 6800 Escalade IDE RAID is working great, and it's only $350 or so.
It's a PCI card with *8* IDE drive slots, which you can configure in a RAID 5 array for huge, failsafe backups.
I've got 8 60-gig IDE drives on it, in a RAID 5 array. Gives aboout 420 gigs. Shows up as a SCSI device in OpenBSD. Works great with Linux.
Churning away wonderfully.
I've backed-up 200 gigs of files on it so far.
420 gigs of RAID5 storage = $1100 USD. ($300 for the card. $800 for 8 60-gig drives.)
Here's my post on the OpenBSD list about it
i trade some usa food aid packages for DLT2000XT Internet is GREAT!
Internet is Great!!! junis
Since when is a harddrive not a semipermanent media that can be easily taken off site? I'm surprised this comment got modded up so high. And since when are tapes such a reliable media compared to a hard disk? So burn-in the drive for a few days before using it for backups. And use a S.M.A.R.T. utility to diagnose the drive before each backup to reduce the chance that something is getting ready to fail.
Your best option is to put all data on a 2-disk mirrored RAID and use another drive as a removable for an off-site or fire-safe backup. The probability of 3 hard disks failing simultaneously, one not in use, is so incredibly small it's laughable. And for that non-zero chance, if it happens, you can pay to have the spindle of one of the failed drives transferred to a new drive in a clean room.
Advantages:
- Standardized TCP-based protocol, uses authentication.
- Lots of free "backup software" available - e.g. pan, newspost, tin...
- Centralized backups over the network! Only one server directory to back up.
- Backups are automatically dated by the server, and can be signed and even encrypted if you integrate pgp into your posting script.
Disadvantages:It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
I've been backing up my data for the last 4 years. I mean literally, I'm not done yet.
I've been looking at the raw data and writing down the 1's and 0's in the correct order in notebooks.
it's long, painful, and expensive; but... but... well I know I started doing it for a reason...
"when life gets complicated, I like to take a nap in a tree and wait for dinner" - Hobbes.
If you were using Adeptec's DirectCD thing for backups which I know a few people do, the CDs are indeed burned in their own format designed to work well with the DirectCD packet writing. Only DirectCD can even read the format let alone write it. AFAIK there isn't a mount extension for the DCD format but you might look around for it.
I'm a loner Dottie, a Rebel.
The Eindhoven folks are still there, and shipping new product, and it also looks like they've started a division in San Diego to deal with the tape manufacturing in Tijuana. This should help the quality of the tape.
ADR is not bad as a cheap solution. I once wrote an MP3 jukebox that used an OnStream drive as the main storage medium, and it worked quite well.
A well-crafted lie appears unquestionable - Dama Mahaleo
If you really need to back up >20GB or so I find that DLT is the only reasonable option. You can find used DLTIIIXT and IV drives for less than $500 or so. The media is not too costly, maybe $15-20 per tape. You can fit 15-30 or 20-40 on each of these tapes, so for 100G you probably should figure on your annual 0-level dumps taking about 3-6 tapes, and you should have no problem (with your usage pattern) fitting weekly level 1 backups on a single tape each. So a pack of 10 tapes ought to provide you with a month's backups or more. Total cost around $400-500 if you shop carefully. Maybe less now...
About three months ago I lost a couple dozen files when Windows 2000 (go ahead and laugh) managed to fuck up it's own file system during or after what appeared to be a perfectly clean shutdown and reboot. That and seeing all the e-mail worms coming from my boss, I figured I needed a real offline backup system.
I bought a used DDS-2 tape drive from eBay for like $40. I added a decent (but old and slow) SCSI controller for about $20. I can get used DDS-2 tapes (4gig) on eBay for about $3-5 each and DDS-1 tapes (2gig) are $1-$3. They claim double the size with compression, but even with data I thought should compress well, I'm only getting about 30% more space with compression.
So what do I use after all that? CD-Rs. It's just too damn convenient and cheap. I burn 2 backup CDs every couple of weeks, one for my main work and one for my personal stuff, including e-mail. The CDs go into an on-site safe. In between those backups, I e-mail things to co-workers, ftp them to my off-site web server, upload them to Yahoo! Briefcase, or just cross my dang fingers.
...neighbor A discovers neighbor B's cache of hard-core pr0n is being backed up on his disk... or more prosiacally, their TurboTax files... ;-)
"Biped! Good cranial development. Evidently considerable human ancestry."
though not implemented yet, is to use more hard drives. It's by far the cheapest way to go, and much more flexible than other media.
I had a look and realized I'm going to have close to a terabyte of data in the next 12 months or so... and no way to really back it up.
An extra storage box, separate from everything else, where backup archival copies are kept. One full backup + many incrementals per physical drive in the backup unit.
This will be a unit that is only accessed via backup jobs.. it's not going to be 'used' for anything else. Storage is cheap enough nowadays.
Yea, but test that with a rm -rf :)
DLT8000 (the 40 native) is the "current" last of the original DLT line, which probably accounts for its cost.
DLT3000 and 4000 (15 and 20G native) are yesterday's news and can't meet the capacity needs of most organizations anymore, which is why you see so many of them on ebay for $200 or $500.
The carts are still kind of spendy. DLT4000 needs the DLTTAPE IV which is still nearly $85. The DLT TAPE IIIXT tapes for the DLT3000 drives are cheaper at $45, but that's still kinda spendy.
For that kind of money, it might make more sense to buy a DVD-RAM drive (supported under XP, MacOS and maybe others) from ebay for $150 and a bunch of carts and do one full and then incremental backups as needed. Still only barely adequate for storage needs above 20 gig.
For my money, it's hard to beat the new ATA RAID cards that are out. Most can be had for less than $100.
Couple that with two or four 80GB drives, for less than $150 each, and you've got yourself a pretty nice array that will keep your data safe against all but the most horrendous problems.
Even with this, you're probably wise to have some offline backup solution to go along with it.
What data would you really want back if your house was swallowed by a hole in the ground? In that situation, do you really need access to your 30GB of MP3 files?
If the anwers is that you really only need access to your Quicken files, then arranging to have those backed up online should be pretty cheap and easy.
Summary: cheap ATA RAID for hardware redundancy, online backup for truly life-critical files.
You know, the funny thing about /usr/share is it is supposed to be where you put "data" which can be shared by users, right?
No. /usr/share and /usr/local/share are for files installed by an application and shared between the different binary architectures the application can be compiled for (such as map files for a game). To share data among users, use /home/johndoe/pub, or use groups.
Will I retire or break 10K?
When devising a backup strategy, most people don't consider the threat of Electromagnetic Pulse (EMP).
A quick google will show you that EMP would be a cheap, relatively easy-to-build terrorist weapon, having devastating effect on our electronics-dependent economy.
A strong EMP would wipe any kind of magnetic media -- tape or disk.
I am by no means an expert. Does anyone know how to defend against it? Would placing your backup media inside a heavy metal safe provide sufficient RF shielding to prevent damage?
That that is is that that that that is not is not.
Agreed that just backing up to another HD provides the best overall method for creating a complete backup of 100MB of disk storage.
At the risk of being accused of picking flyshit out of pepper, wouldn't 100MB of disk storage fit onto one zip disk?
You're using her as bait, Master!
I know this isn't exactly related, to the topic, but I was wondering about what good tape backup software exists for Linux. Currently I am using kbackup and find it difficult to use. While tar is simple for backing up a directory, it isn't good for doing a full system backup when multiple tapes are required.
Are there any good open-source GUI-based tape backup programs for Linux? I really miss BackAgain/2 for OS/2 when using Linux.
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
I'm not sure if this violates the Freenet code of ethics. But it probably does. And if everyone started doing it, it would probably kill Freenet very very quickly.
Cryptnotic
My other first post is car post.
If all this should have a reason, we would be the last to know.
I'm not sure what a really good solution would be. Using another hard drive as backup is just too tempting to wipe and use for more storage.
I went out and bought a SCSI DDS-3 drive. Each tape holds 12GB. I've been using Amanda to back stuff up. The big problem is that Amanda can't handle the situation where a backup image is bigger than a tape. If you have a big archive of music, it'll have to be divided up into chunks. If you have a database-driven frontend for the music, this isn't a problem (this is where I hope to go...), but people who like the traditional hierarchy systems will not appreciate it.
I actually made a directory that has symlinks in it that point to two separate trees of music (A-M, and N-Z). It's all on the same partition, Amanda just backs up the different directories separately.
I haven't had to recover anything yet, so I don't know how well it works. DDS3 drives are slow, only pushing about 1 Megabyte a second, but I can back up 9 gigs in 3 hours. Full tapes would probably take about 4 hours. The way Amanda works, it balances out the amount of tape used per day. Hmm.. Of course, you'd need something like 10 tapes to just do a full backup. Maybe look at a DDS4 drive, though they were still pretty spendy last I checked.
Unfortunately, backup technology has fallen behind drive technology. Even NASA is having trouble with this stuff these days. They just can't move old data to new tape fast enough to ensure that stuff can be saved.
Well, heck, for the cost of what I'm talking about, you could probably buy a few 100GB drives.. it's pretty crazy.
I currently have about 90 gig of total home storage...soon going to about 130 gig. But, there's only about 40 GB actually being used for data...MP3's, digital photo's, photo editing stuff, old book reports...you know what I mean. A lot of space is OS files because my total file storage is across 5 machines. Screw backing up the programs! For the most part, I'm going to need to dig out the CD's/disks anyway.
I spread the backups around. Run a script to handle all the machines sequentially. 10 gig goes to a machine down the hall....runs across the fast ethernet wire sometime at night and gets compressed at it's destination. Another 15GB comes off of that machine and is dropped two floors below on a Samba share. Gets compressed too. So on and so forth. One log file gets written...PGP'd and SMTP'd to greet me when I get to work at 6:00 AM.
Yup...it's a pain sometimes. But I more efficiently use the storage without dedicating any one unit. I always leave enough space for other work. I increase tolerance so that if a box dies for good I only lose a piece of the backup scheme. The whole shebang runs while I'm snoozing and can afford network traffic and CPU cycles to compress. And they're all full backups to boot.
I've been nailed a couple of times, but not fatally with this setup. Oh yeah...all the boxes are on UPS's. That's important. I've lost more to the power company than to ghosts in the machine.....
This may not be the best way to do it, but it works for me...
I have a "backup" hard drive in my server. This drive is always unmounted so that there is no chance of filesystem corruption from the operating system.
I just use a crontab to run a simple script that mounts the drive and coppies whatever specified backup files to it, then unmounts it. The same method slightly modified could be used to back up this same backup disk to another location on the network on regular intervals.
You might want to look into an Iomega Peerless. The disks are pretty small (maybe about 5"x3"x.5") and the disks are 20 or 40 gb a piece. I'm running one a Windows 98 machine, I couldn't tell you about Linux compatability. It connects to a USB hub and has sustained data transfer of 12 mg a second, I think.
Tim ODonnell (trying to be the most
It occured to me nobody mentioned online solutions such as Streamload or MyPlay (great for mp3 storage)..
:(
Too bad iDrive & Freespace.com went offline
Unless you're creating 10 - 100 GBs of *new* data between every backup, it might be simpler (and cheaper) to use incremental backups.
Instead of dumping all 100GB of files every time, 95% of which haven't changed since the last backup, use an incremental backup program to write only the 5% that actually changed. After the initial archive, the backup files will be significantly smaller, and could potentially saved on CDs.
ShoutingMan.com
-- Linus Torvalds, about his failing hard drive on linux.cs.helsinki.fi
So 100GB drives are getting more and more common. So what? Are backups really any more a problem than when 200MB or less drives were common? We didn't have CD-R back then, much less DVD-R... Only smaller capacity tapes and floppies. And, people with 100GB drives... is what you have on that drive really of more substantial value than what you would have been able to store on a 200MB drive? In my opinion, I don't value what I have on my 30GB drive any more than I valued what was on my 200MB drive "way back when." I think the ratio of data I cared about to data I don't care about was about the same then as it is now. And the backup technology available today can hit that percentage (which I think is about 20%) pretty easily (CD-R) with about as much relative difficulty as floppy-swapping could have back then. Everything else can be reinstalled... no big deal.
I've got all the hardware, but I can't seem to find a piece of software that's built for incremental backup to non-tape media. The closest I've found is NTI BackupNOW, but after much frustration I discovered that even their software won't support DVD-R(W) for some months to come.
Has anyone actually **SUCCEEDED** in setting up such a backup system? Bonus question - has anyone had to restore data from this kind of setup?
The real problem here is the absolutely insane rate of hard drive size increases. Hard drive capacity has been increasing at greater than 100% per year since IBM's GMR heads came out. You can now buy a 70TB(yes, terabyte) emc drive array (384 * 180GB seagates)! Maxtor is coming out with a 160GB! ATA drive (once they finish addressing the 28 bit sector address limit). I agree, this poses the issue of "do we really need all of this space?", but data needs will always scale.
Therefore, since the fault of the problem lies with hard drives hugely outpacing every other form of recording medium in rate of capacity increase, the only reasonable solution will soon be (if it isn't already) to use hard drives as the backup medium. Yes, I know, hard drives combine the media with the mechanism and that is normally a big no-no, but in this case I think the monetary facts must be faced. In order to get around the media/mechanism issue as well as the off-site storage issue while not emptying our wallets, I think multi-site dual-hard drive-backup is in order.
At any given site, one would use large ATA drives in the backup server (most likely in an external hotswap cage for the corporates) in place of a tape library/cdr/etc. But, in this case, we should make two copies (use two drives!) to hopefully get around the combined media/mechanism issue. Trust me, the cost of doubling up will be far less than an equivalent media based solution. Off-site fire-proof backup companies could start taking hard drives (being considerably more careful than with tapes, of course). If you want, you could encrypt whatever was on the drive. This scheme scales from the smallest home needs(keep the drives in the safety deposit box) to the largest corporates, and makes the most sense monetarily.
I hate to say it but tape is still great! I have ~120 Gigs across a number of SCSI Drives. I picked up a Seagate NS20 Travan "refurbished" actual open box, as nothing inside had actually ever been opened, and never looked back. Switching tapes isn't the worst thing in the world guys, and for a home system you don't really need nightly backups. Plus this way you can get your tapes out of the house (leave them at work) or put them in a firesafe in your basement, the possibilites are limitless. Tape also archives better than many of the other solutions being offered up. Just an idea but worth looking into if you need to backup a lot of data.
-OctaneZ
PS. I wouldn't go with anything smaller than a 10/20 compressed drive; I had an 8 Gig Seagate and never backed everything up as it was just too much, but 6 tapes is not bad at all.
This has been my choice for low-cost backup solutions for a couple of years now.
The drives support three different flavours of media - 12GB, 20GB and 33GB, and come with IDE, SCSI, or Firewire interfaces. The IDE is cheapest, at $699, with the media costs being $80, $45, or $35 depending on the capacity.
Is it the absoulte top of the line as far as tapes go? No. But the cost can't be beat. And you get a reasonably fast (3MB/sec) drive with very nice reliability (take a look at the independent testing on their site; e.g. soaking a tape in hot coffee for a minute, rinsing it, drying it and reading the data off.)
They also recently merged with Exabyte, who will be positioning it as their new value solution. Hopefully the Exabyte name will expand the market enough to drive the prices down on these even further.
Matt
This is what I do (man levels of redundancy):
1) Use resierfs -- it'd stable now and recovers better then ext2 in small "incidents"
2) RAID 5 array
3) Run a nightly script that hard links all the files into a . (hidden) directory -- protects against rm -rf''s
4) Run mirror in every directory from cron-- if you lose or mangle one file you cna recontruct it from the contents of the mirror and the other files (works a lot like XOR in raid 5 arrays -- aka a RAID for files).
5) DDS3 incremental back-up; complete backup at regular intervals -- one set of tapes stored off site.
This protects you from multiple levels of failures -- with the catastrophic redundancy being the tapes. You don't always want to rely 100% on the tapes for all your redundancy.
IIRC, the syntax in /etc/raidtab is: /dev/sdc1
nr-spare-disks n
[...]
device
spare-disk 0
And, if so, can it be done while the system is `live'?
Yes, using the raidhotadd and raidhotremove commands.
Doh! Upon second reading of your post, I realized when you said "mirrorset" you were probably referring to raid 1. If this is the case, then I am probably misleading you. AFAIK, you can only use the spare-disk directive for raid levels 4 and 5.
Enigma
You can get a DLT drive that will do 20-40G per tape for under $200 on Ebay. Add an external case if it doesn't have one, and grab a SCSI card. Cheap, reliable, tested.
Just FTP the files to the SAN at work. I hear they have tape drives and might even do backups!
+++ UGUCAUCGUAUUUCU
Being one of the maintainers of Amanda (www.amanda.org), I'd always been of the opinion that tape backups were the only way to do backups seriously.
/boot on RAID 1 over the 4 disks and / on RAID 1 over 2 of the disks and an alternate root to test upgrades over the other 2, but you get the point). This got me blazingly fast disk access, that tapes would never help me get :-)
The recent explosion in disk capacities and decrease in prices got me to rethink this, just when it came the time for me to set up a home office. When I compared the cost of a reasonably-good tape drive and a number of tapes large enough for me to get at least a month of backups in rotation, and computed how many 60GB disks I could buy with that money, the solution was clear.
I ended up setting up 3 machines with 4x60GB each. They're all on RAID 5, such that if any single disk fails, the machine keeps running (actually, I have
I get all my backup-worthy data rsynced over to the other machines daily or so. I plan to start playing with Inter-Mezzo soon, so that I don't have to remember to run these backups, and so that I don't run these backups on the wrong direction.
But that's not all. With the mind-boggling amount of disk space I could afford, I could (actually, I will, but you get the idea) set up Amanda to backup interesting portions of my home directory to disk, and also replicate this to at least another of my local machines. Such backups can use software compression, such that they don't take as much space as live data. Also, I intend to use another form of compression: instead of backing up CVS trees (I've got loads of check outs), I'm going to back up only local changes to files, so that, in case of disaster, I can still download the original CVS tree and re-apply patches. But this is still a plan, not something I've got running.
Finally, I've got yet another disk on a remote site, to which I rsync not only the interesting portions of my data, but also my backups. I could convince someone else to run this remote backup site for me by offering this person the speed up of RAID 0 over two disks (one of those mine). As for keeping the secrecy of the data on this remote backup site, I'd just get the backup files encrypted, no big deal.
I can strongly recommend this solution: I got pretty much as much data safety as could be expected from a tape-based backup, without any of the hassle of having to switch tapes and moving them off-site and back on-site, and with the bonus of very fast access to local data, unlikely donw-time and fast recovery except in case of total disaster (i.e., having all of my local machines failing, in which case I'd have to either download my backups from the remote site over the net or, more likely, take a replacement machine over to the remote backup site and copy files over a fast local network connection, or from disk to disk.
As for getting 4 IDE disks into a single machine, don't even think of using only the 2 IDE controllers that come on most motherboards these days (for RAID set-ups, you really want one IDE disk per controller). There are a few good motherboards that come with 4 IDE controllers, so that you can even have a CD-ROM and/or a CD-RW in addition to the 4 disks. If you can't find such a motherboard that suits your needs, you can always get one of those PCI cards that adds 2 IDE controllers to your machine.
As for the problem of fitting so many disks in a standard ATX chassis, it can be done. Cooling may be a problem, but a good cooler has been good enough.
All in all, I'm very happy with this arrangement. It was not cheap, but it was not as expensive as a tape-based solution, and it's far more flexible, way faster and it doesn't require any baby-sitting after you get it going. And I can keep far more backup history than I thought it was going to be possible.
What you need to do first is plan on what gets backed up when. Few of my MP3's are backed up as I still have the CD's they were converted from, my digital pix get burned to CD whenever I get a new CD's worth, no software is backed up because I have the install disks, but data files are backed up daily to zip drive.
Moral: Not everything has to be backed up at the same time to the same media, but you do need a backup plan and schedule that accounts for all files *AND* that you follow.
The chances aren't as bad as you make them sound...
Let's just say you have an internal RAID system with, oooo, 4 drives, along with a removable drive to backup everything. The problem lies in the whole trusted/shared medium concept. If there is a surge passed along the case, the SCSI/IDE cable, or through the power-supply cabling, not only will ALL of your drives get toasted, but if you have the backup-harddrive connected to the system (actively archiving your data, finished archiving and waiting for you to remove it, or just because of a BAD practice of never actually removing the removable hard drive) you will loose your backup hard drive as well.
While RAID is a good thing, multiple hard drives are still at the mersey of everything they are connected to. Using such a system as your only backup is a bad idea that happens too often. Having a removable hard drive is an option, but the work involved really makes other solutions much more viable, especially on a large-scale (and on a small-scale, people are lazy!).
I propose a network-based automatic backup system for most people. You simply have your main system automatically backup it's data over the network to another system (systems with low number-crunching capabilities can be put back to work here). Of course, you would want to maintain at least 2 concurrent backups in-case the main system dies during the said backup. The benefit of network backup are that human intervention is not required (the user and administrator don't need to do much of anything after initial setup) and off-site backups can happen transparently (just send the data to the other office down the street, across town, whatever.
Speed of the network may appear to be a problem, but 100 GigaBytes (UNCOMPRESSED) can be transfered in 2.22... hours over 100Base-Tx. First of all, it's likely you'll be compressing that data, which on average halves the size, and so the time is halved as well. Secondly, Gigabit over Cat-5 is available at $45 per NIC, making backups take one-tenth that time. And finally, an encrypted SSH, IPSec, PPTP, (etc) tunnel could be established that would ensure the data is kept private. Data security is much more difficult when you have multiple copies of it unencrypted in a conviently sized package. You are just saying 'steal me'.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
Now, something that occurred to me regarding the management of "archive" files, such as mp3's, ogg's, and the like. Each time you rip a CD and encode the wav's thereof, you've reduced your storage requirements by nine (roughly). Why not hardlink your archive files to an unorganized ../discXXXX/
directory. Link enough files in each directory
to roughly equal the storage media you're using.
Then, when you have a full "disc", burn it to CD.
We're not talking about an organized disc, remember.
Just faithful backup of new data. Do all of your
heirarchical data management in a set of different
directories, or use a database of some kind. Essentially, you can take out
much of the load that would be handed off to
Amanda or other archiving schemes.
assert(expired(knowledge));
I was backing up my music collection using DDS2 tapes recently. I had picked up a drive off of eBay for $50 so I thought it would be a perfect match. Media was "cheap" so I thought I was set.
/etc and the like...
My boss was kind enough to give me some old used tapes we had after upgrading to DDS3 tapes a while back. I ended up with 10 or so tapes.
Using NTBackup, I set about backing up 20 GB of MP3s, with hardware compression turned on. Each tape took about 3 hours each, and of course I had to be there to change the tape and then click OK/continue. Well I ended getting a bit over 3.5 GB per tape so I needed to use 6 tapes. All in all it took well over a day to complete the backup, what with switching tapes and the like.
My recommendation is that if you opt for tape backups to invest in a large capacity tape drive. You can get DDS3 drives for $300 or so, and 40GB DLTs for $500. I'm looking into getting a larger tape drive because my time is worth something to me... I don't want to sit around changing tapes all day.
However, the DDS2 drive is perfect for backing up my kernel and
Don't listen to him. Tapes always let you down!
Some of the problems with tape are:---
Granularity -- i.e. you accidentally deleted "xmascard.lst" how do you know which tape it was on and how long will it take to scan several tapes to find your 2K file?
How can you be sure the tape is readable. In my experience something like 5% of backup tapes cannot be read by any drive except the one that wrote the tape.
No standard format, no backward compatabilty, short market life for thevarioustecnoligies. You may be very smug that you have an "acme" format backup stored in a safe deposit box, but after your house is flooded (mentioning the f*r* word upsets some people) you discover "acme" drives are no longer sold anymore, you just have to hope someone is trying to unload a working drive on e-bay.
Expense -- the drives, the tapes etc. will all cost several times as much as a couple of external hard drives.
Old COBOL programmers never die. They just code in C.
It provides good redundancy against hardware failure. Together with using snapshotting you can have better than nothing protection against both hardware and software errors. The presumption here is that it is a *home* network. In other words, super-high availability is not important. If a drive goes bad, I can afford to shut down that system until I have a replacement hard drive. Sure, a hot spare is nice and really keeps you covered in a drive failure, but isn't that critical when you can power down the system in event of drive failure without consequence.
Backup is cost prohibitive to a lot of home users. I back up most important stuff to CD-R, but the other 30 gigs or so I just have to stick it on the RAID-5 and cross my fingers.
XML is like violence. If it doesn't solve the problem, use more.
The cheaper and motherboard IDE-RAID controllers are useless and more expensive to W2k and Linux users. Why? Because they require about the same or worse CPU load than the native software RAID solutions. The cheaper IDE-RAID cards are mostly smoke and mirrors. First, a BIOS-Trick to at least make it look like an array at boot. Then, the OS drivers take over with RAID operations done almost entirely in software, not hardware. If you don't run the vendor's or some other special driver for the controller, you do not see a RAID array, but the drives that should be in the RAID array as independent ide devices.
XML is like violence. If it doesn't solve the problem, use more.
I define "backup" as including at a minimum daily copying of NEW data, plus whatever other arrangements are needed to esnsure that you can get your needed applications and data up and running in a new system within the required time. "Archiving" is making long-lasting copies of data that isn't going to change often.
Home computers usually don't have much data that changes frequently, nor do you have to spend a lot of money to ensure that you can be back up quickly -- that is, re-installing all your applications is acceptable in the rare case of a hard drive crash. Maybe only your checkbook records need back-ups. (The ancient DOS shareware program I still use for the checkbook fits nicely on one diskette along with 13 years of data, so I'm all set there.) But your photographs and tax records definitely need archived. The only truly long-term solutions for archiving are rather impractical for most people: printing to acid-free paper with permanent inks, which gets mighty bulky in the long run and might not preserve colors, or etched in stone or metal for _really_ permanent records. The best practical solution I know of for your family jpeg photos is to buy good CD-R's (100-year lifetime claimed), and make two copies on two different brands. Keep one set in your safe deposit box or something, so if your house burns down you still have it. Every 3 years, review the archives -- are the disks holding up, does it look like compatible drives will remain on the market, and will the data and disk formats remain comprehensible to new software? Every so often, you are going to have to copy to new formats.
Note that there are other not-so-standard optical media that are technically better. The trouble with relying on one of those is that at some point you will have good disks and be unable to obtain a drive to read them. It's going to be a very long time until that happens to CD's. Write-once DVD's are worth checking into; I don't yet trust their data stability or longevity on the market, but unless the copy protection @#$%^& screws it up, they are going to be a better archival medium than CD-R's.
Businesses need both archives and frequent backups. Tapes are NOT a great solution for backups. Our e-mail server here had a hard-drive failure recently. (Very much against my own opinions, this used microsoft software and tape backup.) It took four days to install all the software on a new hard drive and restore all recoverable data from several tapes. The most recent backup tape was unreadable, so two day's e-mail was lost. This was not a big problem, but 4 days without e-mail was a very big problem. And what I hear is that this is pretty typical: 20% of tapes from very expensive backup systems don't read back, and it takes far too long to restore from them when they do work.
So for a business to recover from a hard drive crash before it becomes a major problem, you need to mirror the hard drive. (Or if you need hundreds of gigs, use RAID 5.) At the cost of hard drives nowadays, there is no excuse not to.
This protects you against HD failures, but not catastrophes like someone stealing the servers, or crashing an airliner into the building. Many of the WTC businesses actually were ready for that, almost -- they were backing up their servers to servers in Minnesota. The service also included some desks, computers, and phone lines so if the offices in NY were destroyed, the workers could get right back to work as soon as they arrived in Minnesota. The only thing not planned for was planes to be grounded at the same time, so the people had to drive to MN. But I can't think of a better plan for a company that is concentrated in just one city.
If a business has offices in multiple cities, then you can backup the servers to other company offices...
Finally, mirror drives give you instant recovery from many catastrophes, but are no good against the most common causes of data loss: corrupted files and viruses. If you are mirroring, the corruption spreads to the backup within seconds. So you need something besides the mirror drives. Periodically taking off an archive copy of the data may be sufficient. Rotating full and incremental tape backups give some protection, since the chances of bad file + bad tape at the same time are rather slim. Or for the really paranoid, have a whole chain of back-up (not mirror) hard drives -- each night you copy the nth drive to the (n+1)th, so if it takes two days to notice that a file was corrupted, you've still got a good copy on drive 3. Add to this a good off-line archiving system for files that aren't used frequently (so corrupt copies could spread through all the backup HD's before anyone notices), and you are pretty well covered.
No, it hasn't. In the mid 1990s there was a point where backup was affordable and convenient. High end hard disks were 2 Gigabytes, and $15 DDS2 tapes held 4 Gigabytes (native!) and worked in a $600 tape drive.
It was wonderful. Everything fit and it didn't cost thousands of dollars. I think a lot of people (e.g. me) got spoiled during those few years. ;-)
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
I agree that all of your ethernet cards are at risk... That still doesn't significantly affect the security or integrity of your data. Even if you have a lightening strike while you are backing up your data, (which kills the main system, and ruins the backup) you should have at least one or more backups available to use.
If you are a large company, or part thereof, you likely hace fiberoptic lines between your main system and the backup. Which is not to mention that Ethernet surge protectors are getting more popular all the time.
BTW, why did you have Ethernet outside where it could be struck by lightning in the first place? Just curious.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant