Slashdot Mirror
A Storage Solution for Lots of Digital Photos?
Duizendstra asks: "I've been asked to explore the digital storage possibilities for a professional photographer. One of the characteristics is the rapid growth of the amount, and size of pictures. At the moment, one photo session produces about 2 GB of raw data. He has an Apple - Power Mac G5, and he currently uses DVD as his storage medium. However, he has lost quite a few photos because of DVDs that can't be read anymore. I would like to know if any Slashdot readers have any experience in creating a solution for such a problem? Any help/idea(s) would be greatly appreciated!"
Would be to buy a few external hard drives. With the storage capacity of hard drives these days you can't go wrong. I bought an enclosure for one of my internal drives and now I don't know how I could live without it. Having a portable drive like this is an amazing convience, especially with a capacity of 160GB.
One approach is to burn three copies and then you can recover the data by averaging the signal between them. This requires multiple drives but it is better than having to give up on archived data. Manufacturers suggest: "Store your recordable DVDs vertically, protected from sunlight, in a room that avoids wide variability in temperature and humidity."
http://tinyurl.com/4ny52
He should get an Xserve RAID, of course.
It'll just work, it's well integrated with his G5, and it's cost effective.
Try one of these
2GB/session isn't really enough information to design a storage solution but I'll dump out some generic big, reliable and cheep storage suggestions.
S hareShow.asp?ID=1764600
For large scale reliable storage I dislike both optical and tape. They both quickly become more work to manage than it's worth and have serious reliability issues. Hard drive based is the way to go and since hard drives do fail and that is a bad thing, it's best to use RAID. It's especially a good idea since RAID is getting easier, since hard drives are getting cheaper per unit and since SerialATA is making it easy to hook them up right.
Heres a basic design that I'm actually working on for a home server for myself:
http://secure.newegg.com/NewVersion/WishList/Wish
It's a 3U rack mountable 2TB storage server. Put a Linux distro on it with some small RAID1 boot partitions and a software RAID5 storage partition, throw samba and some email-home config to notify of drive failures and you've got a decent place to store up to 1000 of those 2GB sessions. Zip up the old ones if needed for more space. If rack-mounting isn't desirable there are cheaper desktop cases that would probably be appropriate.
If this is overkill a 4 drive RAID5 array or even a 2 drive RAID1 array is much much easier to accomplish. Standard case, motherboard, power supply and drives with a Linux distro and you're done. Hardware RAID is also an option but since software RAID's high CPU usage wouldn't be an issue here I'd go that route.
set softtabstop=4 shiftwidth=4 expandtab nocp worlddomination
This is a system for a professional photographer, storing the digital photos that are the lifeblood of the business. (Note: most professional photographers charge a nominal fee for a session, but then make the real money by selling prints. No negatives or no files = no $.) This is most emphatically NOT the place to try to do things on the cheap. It's an absolutely necessary and tax deductible business expense.
I work for Apple, and while I'd prefer that this place purchases an Apple-based solution, I am not wedded to a particular OS or brand of hardware. However, you get what you pay for -- either through hiring a skilled professional building an open-source based storage system or by paying for a commercial solution (such as Apple's XServe RAID unit). Be sure to include the necessary system maintenance in the budget for such a complex setup, including off-site backups, on-call support, and making sure that it stays up and running during successive system updates and upgrades.
Given that the photographer is already using an Apple G5, I suspect an XServe RAID solution will suit the situation quite well. One unit can provide 7TB of storage, which at 2GB/session works out to about 3500 sessions at current resolutions (also allowing plenty of headroom for growth as resolutions increase). Apple offers professional services, on-call support, and training for server administrators. In addition, if you're looking for an Apple consultant with the necessary skills in your area, check the Apple Consultants Network.
--Paul
This is definitely a good recommendation. ATA over Ethernet has comparable performance to a similar SCSI configuration. Additionally the price tag is not quite as hefty. Coraid's prices range from $2,000 for 2 Terabytes - $4,000 for 7.5 Terabytes. That does not include the price of a NAS server. Like the parent said you should be able to homebrew this. You could possibly set it up on the Mac G5 you currently use. Making the assumption that you can easily set up an NFS server in Mac OSX.
If storing all that data was important to my livelihood, it would be hard for me to not justify this investment.
Other options, are optical storage, which is not working out for you currently. Of course, you could drop a couple thousand on a good tape-drive, and spend several hours backing up, and restoring. Then you have to deal with worn tapes, and eventually a worn out tape drive. Of course, you may outgrow your tape solution and have to invest a few more thousand in a larger capacity tape-drive.
On the other hand, the biggest problem in a fault-tolerant RAID is taking a few seconds to unplug a dead drive, and replace it.
/^([Ss]ame [Bb]at (time, |channel.)){2}$/
Your friend could probably get about 6:1 compression I'd guess
I *highly* doubt that. It's unlikely he'd get any significant compression, and very possible that compressing the files would actually increase their size.
We're talking about photos here, which are already compressed. Even RAW photos are compressed heavily (though losslessly). For example, a Canon EOS-1Ds Mk II takes RAW photos at a resolution of 4992 x 3328 with 36 bit per pixel. An uncompressed image would be 4992 x 3328 x 36 / 8 bytes, which is about 71MiB. The image files produced by the camera, however, are 14.6MiB, a compression ratio of nearly 5:1. The file system compression isn't going to get much more. On RAW files from my camera, bzip2 -9 only averages about 0.1% reduction in file size, and bzip2 -9 is very good -- and very slow -- compression.
Disk drives are the best way to safely store large volumes of data, especially when you add some redundancy, but don't expect to get any help from compression of already-compressed data.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
I'm a photographer and I'm looking at over a terabyte of external firewire drives piled up on my desk and spindles of DVDs backing them up. Right now the options for reliable long term reasonably prices storage pretty much suck if you generate around six gig of files a day. Lots of hard drives is fine as far as price goes, but they aren't an answer for long term storage. If anyone has an idea for storage in the 30 year range, I'd like to hear about it. My experience with tape back in the DC250 days was pretty dismal, and I don't ever see any tape systems touted for long term storage, but I'd love a recommendation for a system that I could trust for more than 10 years. I assume that a product to address this market will show eventually but I'd like to have something now
S ection=pressreleases&department=maxellusa_pr&Line= datapr&Open=datapr410 608/105586/?ST=english
For what it's worth I'm looking forward to these when they come out.
http://www.maxell-usa.com/Content/Pages/Page.asp?
or these
http://techon.nikkeibp.co.jp/english/NEWS_EN/2005
buy an external 300 GB firewire drive every couple months, label the drive by the time period. If you are really worried, have 2 every couple of months, they are pretty cheap. Drives don't fail sitting on a shelf, at least not like dvds do. The last thing you wan't is a raid system thats active every day, it only increases the likelyhood of failure. raid is great for data you must access now, but a waste if you are just using it to back up data you only need once every so often.
If it is really important, use tape backup, make redundant copies, and send one off to a data storage place. As others noted, a backup solution should be part of the cost of the job, and is not really that expensive when divided over the different projects.
-I just work here... how am I supposed to know?
WiebeTech makes a desktop RAID enclosure that looks interesting. It's actually two 5-bay RAIDs in a single unit. This is essentially a desktop equivalent to Apple's XServe RAID. WiebeTech has plenty of good high-capacity disk solutions.
Nobody proposed the "a gmail account per session" yet?
Two phrases I don't like seeing anywhere near each other.
Aim more for "redundant" and "widely tested" for starters.
I'm Rick James with mod points biatch!
Most everyone is recommending hard drives and I'm definitely part of that crowd. Most everyone says "RAID" as well. I'll tell you what I do at work, not for photographs necessarily, but for all our data. I have two servers for user data, one on each side of campus. One is the "active" server with RAID drives, the other is a backup. Each night (I have the luxury of a quiet network at night), I run a network backup to toss stuff over to the second server with RAIDed drives. This prevents the accidental "rm -rf" users that just 1 server with RAID wouldn't prevent against (of course other types of attacks WOULD kill two server solution, which is why important data goes to an external firewire drive from the backup storage server).
Granted, not everyone has the cash to blow on all this, but my stepfather, who is also a professional photographer, has finally taken that all important step toward moving to digital. He's been backing up to CD and he usually gets away with a session on one or two CDs, not counting any editing he does in Photoshop (he still prefers to touch up photos by hand). Anyway, he has been watching his bookshelf fill up with CDs, much like all his file cabinets that store all his old hard copy negatives and select prints. Any long time photographer probably deals with the same stuff, which was a problem before digital ever came around.
What I've been working with him on is what will most likely be a big storage server... even at 2GB a session, you could shoot every day of the year and only use 700GB, which will cost you about $300-400 in a non-RAID solution nowadays (based on me just purchasing 4x 300GB drives at $110 each and my friend buying a 400GB for $200). A small server with a few drives will be all the online backup one should need, plus to be extra safe, either that backup server, or just a few external drives.
If you backup to the external drive once a week or so, this should save anyone from the accidental rm -rf (my stepfather once deleted all the pictures on his laptop by accidentally dragging the wrong folder to the Recycle Bin -- naturally all his photos were too big for the trash and were instantly deleted, luckily he had all his CDs to restore from). Plus, as long as you're backing up regularly, it should be obviously that the hard drive is working or not working. If you start hearing clicking, or feel something funny -- get it replaced.
I guess my summary of all this is to have two backups. If one is your "online," primary storage, it should be obvious if it's failing or not failing, and assuming you're backing up to your second backup regularly, there shouldn't be any danger of you not realizing it's failing, because you are using it all the time. with DVDs and CDs and other media of that type, it's because you set it on a shelf and forget about it for years that is where the danger is caused.
Sorry this was long.
Sig!
If the imperative is protecting his data, then he should do what professionals do with any other kind of critical data: put it on tape. Sure, you probably could put together an optical solution, but the tape technology is compact, proven, convenient, stable and scalable. I'd avoid anything proprietary aimed at the consumer level (if anybody still IS aiming tape technology at consumers), and look at technologies such as DLT which are popular for critical applications among professionals. Support for technologies adopted this way is measured in decades.
Given that this is his life work, he really should invest a few thousand dollars and put together a strategy that will protect him from media and system failure, localized disasters such as fires, and possibly even regional disasters. With a little thought, while it is not going to be cheap, it will be a bargain.
Supposing he's willing to put four or five thousand dollars into this. He can get a SDLT tape drive with a 160GB native capacity (don't count on compression for photos), and 16Mb/s native transfer rate. That day's photo session takes two minutes to back up. am deacj tape stores possibly up to half a year of work. He'll have enough money to buy a good number of tapes, so with a a little thought he'll have a good system for archiving his old stuff, one that is not vulnerable to single tape failures and has an offsite (important!!!) component too. And he may have enough money left over to buy a fire resistant media safe that could buy his data at least a couple of hours of time. Depending on the economic value of his work, he could also send backups to an offiste media storage facility that provides a very high degree of security against regional disasters as well.
I'll tell you a story I tell all my clients when the cost and inconvenience of a well designed backup program comes up.
Years ago I had a client who drove up with what looked like a huge piece of burnt toast in the back of his nice Mercedes sedan. He was was a CPA, and this was three weeks before tax day; the burnt toast was a minicomputer that had all his client's tax work on it. He'd been doing backups daily to tape, but contrary to our advice he had stopped bothering to take them off site. Under the circumstances, if he'd had an offsite backup, we'd have lent him everything he needed, even the office space if necessary. He'd have been back on track with maybe two days down time on the outside. When tax season was over he could have moved to a new office, bought new equipment from the insurance settlement, and his biggest worry would be decorating. But all this depended on the offsite backup he didn't have.
There's a small chance that some of his data mightbe retrieved nowadays, by firms specializing in this sort of thing. But they didn't exist in the early 80s, an in any case I wouldn't want to bet on it. The computer had obviously taken major heat; the interior wiring and connectors weren't just smoke damaged, they were brittle from cooking. We did the best we could, removing the drives, stripping and swapping the electronics on them, cleaning all the connectors on the drive with tetracholoride and so forth. After a few hours of work it was clearly futile, but we spent another day on it trying pointless and hopeless things, just to make him feel like we'd done everything possible. None of this would have been necessary, but for want of a simple step he was fully equipped to take, but seemed like a bit too much bother at the time.
The lesson is that while people comprehend small disasters like misplacing a file, large disasters are sometimes so horrible to contemplate that they discount them altogether. If your client is lucky, he'll be irritated with being saddled with having to swap tapes every morning and perhaps rotate them offsite every few days. Maybe labelling the tapes will be a chore. If he's unlucky, you'll be a hero.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
No one does, because, frankly, digital photography hasn't been around that long--so there are no solutions that were around in the 70s, that are now still working, to demonstrate that they are reliable over that kind of time period.
I'm unsure if you think this is a digital photography problem, or if you just believe digital photography is the only possible reason someone would need massive, long-term, reliable storage.
Either way, it's an "ask slashdot" at least twice a year, for all sorts of reasons. It's a general computing problem.
The only possible solution as I see it, is to quit sissying around with firewire drives... they're nice when you need a little extra storage, and don't want to dick around with opening the case. Get a real fibre channel card for $50. Get a fibre channel enclosure for $500. And another $5000 or so will get you decent, lowspeed FC drives, with a few spares. Over the course of 20 to 30 years, you'd have to constantly rebuild it... a 140gig drive from today won't be replaceable in 16 years unless you have a spare (even if you did, would it be reliable itself?). It could easily cost alot, but then, maybe you're already spending quite a bit?
Also, if I hear SATA one more time, I think I'll puke. This guy is asking about 50-ton dump trucks, and people are talking about riced out Honda Civics.
You are talking out of your ass. I'm seeing your butt cheeks move in sync with the words.
Big ass SATA RAID
SATA? Do you also act like you're on the same level as F1 race mechanics, because you checked the oil in your Ford Pinto?
using a mix of software and hardware RAIDs
Yes, that's a strategy! In something very nuanced, complicated, and with potentially disasterous consequences, let's mix the two together for even more complexity. Maybe this is worth considering in some cases, but without an expert there to come to that conclusion, and certainly without any stated reasons for this, THIS IS A BAD IDEA. If this is what you eventually decide to use, do not hire parent poster to do the job. Oh, and since you'll want someone who knows what they're talking about to do it, it's going to cost more than his estimate.
A cheap, sane alternative would be to compress your photos. JPEG really is good enough
Just when I thought you couldn't be any dumber. "Yes, for long term storage of your incredibly hi-res pics meant for professional photography and graphics, where every single lost bit seems to count, why not print them out on acid-saturated paper with my crusty inkjet printer that's out of yellow?" I mean, my god. It's an ask slashdot, people are supposed to be stupid. It can't be helped, but damn. There are sea urchins with more advanced cognition.
I don't ever see any tape systems touted for long term storage, but I'd love a recommendation for a system that I could trust for more than 10 years.
SuperDLT. It's what we use for SARBOX data retention compliance.
Unfortunately, that's "enterprise" tech, which means Big Bucks.
"I don't know, therefore Aliens" Wafflebox1
It works. Reliably. In my previous job, we pretty much depended on it. A single faulty tape could cost us from $50k and up. And we didn't do backups of data on it... The tape drives were used continuosly 24/7.
If you can afford it, is an entirely different question. I think it's about $30k...
Film. No, seriously, there are labs that will project your digital files onto slide film. Usually you can find them if you look for presentation preperation companies. They usually advertise this service for converting computerized (read: PowerPoint) presentations to slides for showing in venues that do not have video or (SX-/X-/S-)VGA projectors. Good film has proven that it lasts for a long time. Go check your 30 and 40 year old slides.
Video Production Support
Well, (and this is me just talking out of my ass here), you could maybe invest in a film recorder (we called it the slide-shooter). Think of it as a digital projector that projects onto film for later development. We used to use one in my lab to tranfer presentations from PowerPoint to slides (for scientific conference presentations), but I imagine that, as a last resort backup solution, it might work well for photos. The slides are definitely off-line, and I presume could be used to recover the photos in the event that all else fails.
The backup scenario I imagine would be:
Now I imagine that this is not an ideal solution, but it does provide you with the 30-plus year proven backup (we've all seen 30-year-old slide shows). The downsides are that shooting the slides is a time-intensive process, and it adds in the overhead of purchasing and developing the film. I also imagine that you lose some quality going from digital to slide and back, but, you do have the photos in the event of hard-disk failure, and slides are fairly condition-tolerant and physicially small as a backup. Just make sure you don't drop the box.