Slashdot Mirror
Large-Scale Video Archiving?
BondHeadGuy asks: "Ok, say you have 1000+ cameras emitting 30 frames/second worth of 640x480 grayscale video...and you have to store it indefinitely. What do you do? This is a real question, believe it or not. 30 frames/s * 300 KB/frame = 9 MB/s per camera. 100:1 video compression brings that down to ~90 KB/s. But 90 KB/s * 1000 cameras = 90 MB/s, or ~8 terabytes/day. Retrieval, though, can be essentially arbitrarily slow. Reliability should be good enough to not be annoying long term. Is there a solution that: has 8 TB/day storage capacity, can handle the 90 MB/s write speed, and lets you save some bucks on the (slow) read side?"
Although you are probably looking for a digital solution, don't overlook the solutions that already exist. Security camera VCR's (available at RadioShack et al.) can put 24 hours (or more) of video on a single VHS tape. Get a few VCR's (at $200 each), and a pallet of VHS tapes at Sam's club, and you could record all the video you want!
simply get 1000+ computers with as many large SCSI hard drives as possible.
:) ]
[1000+ can easily be 10000+ or 100000000+, but lets be realistic
not for nothing, but is this for either
1. a reality based web-tv show
2. some bizarre web porn thing
3. some actual legitamite venture
4. security issue ?
hope you pull it off.
--donabal
Safety First Day?
Check out www.emc.com , if you need MASS storage, they're who you need to talk to.
Yep, I never spell check.
More incorrect spellings can be found he
Like a security camera in a stairwell or something? In that case you can use motion detection to start/stop recording and save well over 100:1. The choice of video codec is going to be important if it's for security (so faces, etc. can be recognised), but if not, you can crank the compression ratio up quite high on most codecs, especially the video codecs that do frame-by-frame motion differencing (i.e. not MJPEG).
300k/sec seems very excessive. You could try converting it all to mpeg4 with a DivX encoder (http://www.divx.com) and that should compress it right down. If you've got sound in there too, strip it out or at least convert to MP3.
You can do all this with a great program called Virtual Dub (http://www186.pair.com/vdub/)
You've got mail. Pattern baldness. - Crow
Gonna be expensive,
How long does the data need to be stored for? Tape is good if indefinate storeage is not a requirement. (Tape degrades fast.. but is reusable)
Terabyte tape libraries are fairly common. Check out any of the major datacenter manufacturers. Sun and HP both have a unit of about 7TB. But you're talking several 100k$ for a fully automated unit.
Cheapest route would be to go back to the dark ages. Buy a bunch of 100GB tape drives and lots of tape (70 tapes a day ain't bad). Hire a few minimun wage tape monkeys to change tapes on command. Setup a LED display or a big monitor for the computer to flash tape change commands on. (Old IBM trick)
Mark
What your really looking at is some kind of Heirarchical Storage Solution. What happens is that once you have predetermined how much data will be saved from the camera each night. You can get some kind of disk array to store it on. That disk array will also be attached to some kind of HSM solutions such as what is provided by StorageTek's SAMFs. That solution will automatically backup the data that is stored on your disk and remove it from your disk so new data can be stored on the costly disks. From now on your OS and applications think that the data is on disk but in reality its on tape. When the data is requested the software will automatically get it from tape and place it back on the disk. This can be rather costly however.
Be pragmatic and only archive 15fps. This cuts your archive media costs by ~50% no matter what solution you choose. 15fps should be adequate, although who knows your exact parameters.
Go with a FC solution - stay away from EMC, as they will try to sell you a massive Symmetrix for your needs. Sounds like you need a building block approach, one block a day. Doesn't need to be TOO fancy, eh?
Here are some options for FC disk storage:
- Sun T3
- EMC Clariion
- Compaq Storageworks
- HP VA7400 -- my fav
Just to warn you, you're looking at something on the order of 20k/day to operate this setup... now, I'm sure the price would go down QUITE a bit if you're purchasing 8-10TB a day, but even still, it's a huge cost.
I looked at a 10TB solution from the above vendors, and the cheapest I got it was $0.0425/MB!
That's alot of simultaneous pr0n recording...
The first solution I would look at is how they do this at the casinos in Vegas (or anywhere else). Everytime I see one of those 'On the Inside' shows about casinos on TLC/Discovery Channel, they are always boasting about their video camera capabilities, and their ability to archive everything that happens. Whatever solution you find there, while not always the most cost effective, has definately been tested in an environment on the scale of 100's, if not 1000's of cameras.
sig--we don't need no goddamn sig
A suitably large DLT library with a fairly large number of drives would probably do this. Couple it with some HSM (Hierarchical Storage Management software) and you're probably all set.
In terms of sizing, assuming you get 6MB/s per DLT drive, you'll need at least 15 drives. Go for 20. This gives you room to do cutovers, and the like. I'd recommend fronting this with a LARGE disk for scratch space (preferably solid state, but if that's not in the budget, a big old SCSI disk'll do.) You'll need a pretty hefty server to handle all this (at least a pair of Sun E450s for redundancy). You'll also chew through at least 200 tapes a day at a native capacity of 40G/tape.
HOWEVER, this is by no means cheap. The virtue of the fact that you're talking about 8 terabytes a day should be a clue to that. The sort of tape archive, tape supply, and tape library you'll need is... vast. You're talking very high-end hardware here. You'll need a good cataloging system, and some serious software to maintain all this. You'll need to keep about 75% of your drives streaming all day every day. Tape costs alone will run to about 10k/day, let alone electricity, storage, maintenance and initial outlay. I'd venture a project like this is probably a $15 million dollar outlay to do it right, with at least 2 full time support staff and budget on the order of $40k/day . But if you've got the money, go for it.
and you have to store it indefinitely.
Retrieval, though, can be essentially arbitrarily slow
Oh, so your looking for a storage medium with infinite space but slow retrieval time?
Easy. Free-Space Medium.
Just use an extremely high gain antenna, a ton of power, and the space around us. Transmit the compressed data stream, aimed at a distant planetary body of your choosing. I would reccomend something in the 100 light year range or so. Now, when the waves hit the body and are reflected back to earth, you will have what is essentially a 100 light year long piece of storage.
And when the waves get back to earth, the technology for terrestrial storage will be extremely inexpensive, and the reception equipment will be too.
I apparently forgot that sig != uptime...
Still, it's fun to read about
Are we presuming that there will be action most of the time in all 1000+ cameras? If not, then why store the images where there is no action? I'm doing a similar digital surveillance thing, albeit much smaller. (7 cameras, 1 fps, only at night [when there's not much action anyway]) My images all go to jpeg's and I wrote a little C program to throw away all the "similar" images. My algorithm is somewhat conveluded, but more-or-less only keeps the images if they differ by more than a certain amount. I'm sure video compression schemes like mpeg would pretty much take care of this if you're storing to video format. Storing to jpegs has the benefit of being able to easily pick out any time stamp, but I can't watch them like a video.
A simpler solution might be to put a motion detector on each camera and have them only record when there is motion. Using your 100:1 image compression, you estimate 8 terabytes/day. I would expect you could get (warning: pulled from thin air) 1/10th of that by ignoring anything that isn't moving.
But then you have a quandry: was there really no motion at camera #469 at 12:30am, or was something just broken?
Are you capturing in digital format? Are you sure that your systems are even capturing at 30fps?
It's unlikely that any digital conversion device(s) would be able to handle the input from 1000+ cameras and then be able to get that data to a central storage location through a local network.... the bandwidth needed for something like that would be incredible (90 MB/s ??), perhaps even requiring it's own seperate gigabit fibre network. Even in a high-end server with several devices connected to it you'd be lucky to capture 20fps.
But if you were able to get it done, storage options would probably consist of some type of RAID array (with a HUGE number of disks to be able to hold 8TB/day).
Storing that much data indefinately would require enormous rooms dedicated to storage devices, which may not be feasible. Storing data for a week or even a month would be a challenge in itself.
Having things in digital format is nice for indexing and fast retrieval, but in this case it may be too costly. Storing data on video tape may not be as fast or convenient, but it's much easier to store 2 twelve-hour tapes per day per camera than it is to set up and maintain 8 Terra-bytes of hard drive space per day.
Why not do what TV stations and big production companies do to store large quantities of footage: make a big library of digital recordings. I know the restrictions are 640x480x16, but why not go with 720x480x32 instead? Because its digital, you can get lossless recall through firewire in any halfway decent video camera, and backups to hard drives for compression, editing and analysis is pretty easy.
Why set up a high-tech solution for a low-tech problem?
"Look at me, I invented the stove!" -- Ben Franklin
100:1 video compression brings that down to ~90 KB/s.
Very interesting problem, with one more very interesting challenge that hasn't been raised yet:
Because the video is streaming in 24/7, you'd have to build a real-time compression system that could handle the 9MB/s and produce a 100:1 ratio. You could perhaps distribute that across multiple machines/CPUs, or build a custom parallel hardware setup to handle the encoding, but at this scale, the overhead of everything might prevent you from reaching the essential criteria of real-time.
Does anyone know what the hardware requirements are for real-time encoding one 640x480 stream? Now, multiply by 1000.
Now, for data recovery as you said it may be as slow as it gets, something
Throw your stuff to /dev/null and it will be very happy to watch your videos.
If things aren't actually moving in most of the shots (ATM or warehouse surveillance cams, for example) then you'll be able to get far better than 100x video compression.
Also, how much a factor is comunication. 1000+ cameras ona LAN or WAN?
Any secondary logging going on here? Any metadata (ATM transactions, notes, etc.) that should be stored along with the media? Do you want to use this data for easier access? Is there any preprocessing (facial recognition)?
You mentioned recall could be arbitrarily slow, but if it's possible to speed it up with only small changes, is it worth it?
Feel free to ignore these questions. Largely I'm just curious about something you probably can't talk about, but then again as a systems engineer, I'd find it difficult to recommend a solution without knowing more factors that could impact on ways I can't think of until I know more factors...
Kevin Fox
Get a thousand TiVO's. Why settle for AVI quality when you can see your terrorists and burglars in stunning MPEG-2?
Now you can make your own decision about helping him out (or not).
I once discussed with some colleagues about the possibility to store data indefinitely by sending them around on the Internet...
:
Freenet would be a good idea : store it "everywhere" but "nowhere"...
Or you may put crypt/uuencode it and have Google cache it for you...
Or post bits in (lots of) public fora...
Or all at once:
because your 8TB data per days just seem to be a lot it is obvious you'd even need 10,000 square inch of these to store one year of your data...
But now, tell us, what is it for ?
Is it to monitor some people ?
Is it legal ?
(Actually do I mind ?)
OK, let's take your problem the opposite way
How much storage can you afford ?
Just decide which of the nb of cameras, pictures resolution, fps, etc. you'd sacrify to fit in what you'll actually get...
Trolling using another account since 2005.
10 machines with SANArray FFx-2 from Mylex - that's up to 9TB per machine... It writes up to 270MB/s...
;)
Set up a Round Robin DNS system - and write a small application to run on the servers that makes the DNS point to the next machine when it's full...
Put a DVDWriter on each machine and viola
Any technology distinguishable from magic, is insufficiently advanced.
Since you did not state a retrieval time or storage/retention needs, I am going to offer to scenarios; one for long term, fast access storage, one for short term and/or slow access storage.
Storing 8TB/day for a long time with quick access would probably require a tape silo, which is essentially a tape library the size of a small house. StorageTek is one of the leaders in silos (And might be the only vendor making them these days.), and they make some pretty nice stuff. Their PowderHorn 9310 is a nice model for bulk storage and quick recovery. A downside to the silos is that they do not often handle DLT tapes, which can make it hard to use tapes outside of the library.
If you do not need fast access to the data, and have time to root through tapes for restores, just get a smaller tape library (Anything in the 50-100 tape range from ATL/Quantum Adic or Qualtstar running SuperDLT drives controlled by Veritas Netbackup would give you an easy way to handle all the data. NetBackup has excellent archiving capabilites (IE record data, wipe data from disk.), works on just about any platform out there, scales well, and keeps files in GNUTar format for easy access. As for storing the tapes themselves, if you have a small retention time just keep around a few hundred tapes to cycle through. If you need to store the data for a long time, get a few thousand tapes and a set of nice shelves to keep them on. If you do not have somewhere to store them, Iron Mountain does a great job storing data, I have worked with them before and toured one of their facilities, and I can vouch that they do a great job storing data.
hitachi has several very large storage arrays that are very competitive with EMC last i checked. again, that is if you need it to be in digial format and need it to be online.
alex
I've looked into almost this exact problem (we had about 100 hours of full color video/day - broadcast quality)
Your going to have to get VERY friendly with your local "Storage Area Network" vendors. What we came up with as a best SHORT term solution was this - Store the video on Video tape or DVD (depending on quality requirements - DVD is NOT broadcast quality), and then use multiple players - things like DVD jukeboxes/tape changers. They can either be manually loaded, or a robot. You then use a cache to store the vidio on a last in/last out basis if you need fast playback (assumption here - the most recently used tapes are most likely to be used again)
Encoding isn't that bad a problem - you just use multiple encoding stations - You say you have 1000 cameras - you're probably going to need better than 1000 encoding stations (don't forget spares) - you batch up 1/2 hour (for example) files and write those out to the SAN when your done - while one station is encoding, the next is recording, and you batch the encoded file up into Near line storage, so you don't NEED real time
Storage is going to take space/money BIG MONEY - your talking about 30 DVDs worth of data/day depending on your robots. Figure 1000s/day
Charlie
-- 73 de KG2V For the Children - RKBA! "You are what you do when it counts" - the Masso
Obviously this is some sort of security system that watches a large amount of space. So we are talking either a Casino or park of some kind. If not, then these are the people to ask.
:)
Also, is keeping all of the footage forever a requirement? Or just some of the footage? I would think you may want to keep the footage for a couple of days or weeks at most. If something requires footage to be kept longterm then you would move that from the harddrive to cdrom or dvd.
This is a job for a cluster of iMac's if I ever saw one
This is not the sig you are looking for...
This is actually a pretty easy question to answer:
Don't Do It.
This is someone either playing a theoretical game (in which case, the answer is "outsource it") or its someone who has no idea what they really want. You have, ultimately, many conflicting specs here.
You may as well ask for a space shuttle that can fly to pluto in two minutes with no fuel.
Any system that is recording a thousand video inputs is unlikely to need 30 fps for 24/7 (I can't think of anything short of national security installations that would even desire to record 30 fps 24/7, and you'd still have trouble justifying 1000 cameras to cover every building in Washington, DC). Not to mention the logistical implications of DELIVERING 1000 full-frame video feeds to a central location -- you could saturate the entire radio spectrum for the eastern seaboard or have to build the largest gigabit LAN ever deployed.
If you have a real question, please ask it, but this is as bad as a pointy-headed boss spouting off insane specs as the "requirements" for a project because he wants to be on the cutting edge.
And BTW, you won't need 300k per frame for a grayscale 640x480 video image (except that you desire insane specs, which point we've already covered). A fine quality image could be stored in 25-50k, even less depending on the real needs (of which this project seems to lack).
Recursive: Adj. See Recursive.
No problem. Just broadcast the video streams into outer space on 1,000 different channels of TV. When you want to fetch that video, just go the appropriate number of light years away from earth and start watching.
-russ
Don't piss off The Angry Economist
Do you work for that new "Homeland Security" agency??
So what HSM means is Hiarchial Storage Management. Basically, when file hits a threshold of time, space or whatever, it will take that file and put it to tape. Then, it will replace the original file with a stub of a file that says 'when this file is needed, it's located here!'
Now, for tape storage, I highly recommend going with LTO as a tape format. You might consider doing SCSI LTO tape drives with a Crossroads 4450 connected to Broccade switches to make a SAN as well. By putting it on a SAN, you'll have the ability to spread around your clusters that you'd be putting in. LTO can spool data at about 10-20 MB/sec. Hence, if you get an STK or IBM storage library with LTO, you can fit around 20 tapes in there, and do 200 MB/sec. Plus, LTO has variable speed when writing to it, so it's better than DLT in that regard. Not to mention LTO's 100 Gig native capacity and a better compression ratio than DLT. (2.0 vs 2.2) Then, it's just a matter of cycling tapes through. If you're honestly talking that high amount of data to keep INDEFINATLY, then you might want to look at STK's Powererhorns, which hold around 2000 tapes. Plus, you can always add another wall of Tapes if you're not getting the throughput you're expecting. Or you could look at some of the larger scale robots out there, but they don't support LTO tape format yet.
By doing the EMC SAN solution to an STK powderhorn, you're looking at an enterprise level solution that will support you for years to come. Course, this comes from someone who's a vendor-neutral consultant with experience with similar technology, so your YMMV. `8r)
Let us know how it goes!
Gonzo Granzeau
"Nothing the god of biomechanics wouldn't let you into heaven for.." -Roy Batty
Except if the video is saved uncompressed, then you're cutting out 50% of the needed data all together. Even compressed, you're compressing half as much to begin with so the savings will be 50% + compressed savings. You'll still be ahead of where you were.
/dev/null ???
Pros: Extremely high write speed
Cons: Hard to get data back out, but since "Retrieval... can be essentially arbitrarily slow" you've can just re-film whatever it was that you missed. With the money you save on the video gear, you should have a nice little production budget, too...
OtakuBooty.com: Smart, funny, sexy nerds.
The Casino industry is probably the most advanced in the business of surveilence... the average Vegas casino probably approaches the scale you're talking about already, however they probably don't archive indefinately.
However, any information I've seen shows them to still be mostly analog capture for any storage, or at least digital-to-analog conversion for storage.
Although they probably won't talk about their security systems, they'd be a great resource.
MadCow.
I used to have a sig, but I set it free and it never came back.
The guy who wrote the post, is working on face-recognition technology. (thanks for researching him, cwhittenburg)
This exactly the type of thing that could be done, but shouldn't. Stay away from this, we can only hope that any competent people will stay away from this and they will never get it to work.
Sleep with one eye open.......
Why 640x480? That's higher resolution than broadcast TV. Do you need that? Broadcast TV is 460x360. Capturing at that resolution will lose you detail, of course, but if it's detail you can lose, your storage requirement just dropped by 40%.
And since you said retrieval can be "arbitrarily" slow, I'd look into using VHS videotapes--even if you store compressed digital on them--as a storage medium. They're slow as hell for rerieval, but the media might be cheap, especially compared to the likes of AIT and such.
That's an awful lot of data. Why exactly are you doing this? What is the application? Who are you working for? If you are working for/in the United States, does this application meet the requirements of the 4th and 1st Amendments to the Constitution?
Just a few minor questions.
sPh
Still compression might be the way to go, except not compression of the single frames, but some way to only store each x'th frame, and between such a store, and the next xth, only store differences, and again compress the entire stream (stored frames +differences).
The efficiency of the differences storing should be improved by a preprocessing (to compression) step to try to reduce small variations in color,
iow to make two surfaces match
in color in the digitized picture, if they are equal in color
If one, while designing the difference-detection
algorithm, is able to differ between background and foreground, one could try to further increase
compression, while maintaining quality by using lossy compression for the
background only, while keeping the foreground (e.g. faces, important with security cams) sharp.
Since some security camera's send home nearly static, or a set of static images recurring after
a certain time (moving cams), this should increase compression.
I can't imagine that something like this is not already available, e.g. as a sideproduct of all
the research that went into the DVD/DivX/MPEG
standards.
Like we're going to tell you Mr. NSA/CIA/FBI/Big Brother. :)
Need Free Juniper/NetScreen Support? JuniperForum
Oh, can I be one of the tape monkeys?
Live to Code, Code to Live!
Only 1000 cameras?
I mean 1000 cameras is only enough to put one camera into each home in a fairly small community. Most of the solutions I'm seeing posted so far don't scale up very well. What if you need multiple cameras per home? And what to do about large cities? Maybe this should be a seperate Ask Slashdot question?
I'll see your senator, and I'll raise you two judges.
(((640 *480) * 24bpp) * 30FPS) * 1000 =
Roughly 2.21184e+11 bits per second, uncompressed.
Hmmm.
Why not just erect a satellite dish and stream the data into space? The cost of having an alien intelligence beam it back to you would probably be less than conventional storage costs.
I would especially favor this plan if it's some kind of face-recognition software or other privacy-deprivation program.
Another plan would be to hire 1000 people to sit and swap tapes on their home VCRs every few hours - just hire the people who currently plan to make a living stuffing envelopes at home.
Yet, something tells me that you're not doing anything that I would want to encourage...
Cheers,
Jim, hopefully out of your Jurisdiction
-- My Weblog.
Careful...I've used NetApp boxen, and (I'm assuming you're talking about NAS) they just don't have the sand to handle that kind of load reliably. Granted those that I worked with are now ~18 months old, but we had an Oracle update stream hitting one with a bandwidth of less than 15 MB/s and it started dropping NFS connections. If you try this, be careful to use high speed connections and to multiplex onto as many interfaces as possible!
A hero is someone who knows when to run away. I am a hero. -Trent the Uncatchable
You need to reduce as early as possible as fast as possible. Choose the minimum resolution/bit depth/frames you can live with. Find a compression device (i.e. mpeg2/4) that can handle n cameras realtime and drop res/bits/frames to your specs. These will be expensive, but you can probably make up for it by reducing your storage and network requirements. Funnel n cameras into the devices using analog cabling, then spit the resulting digital stream out a network to m archiving stations with lots of high bandwidth storage to use as buffer. Of course, you'll need spares for each of these devices.
Hire your tape monkeys to run tapes and/or get a disk silo. I suspect you could get away with much less storage than you think by compressing early.
Even better would be motion detection on the compression or camera side so that only the cameras that are sensing movement would be working.
As for your network, you'll need to partition it to meet your bandwidth requirements.
Good luck!
You could probably use the distributed file system capabilities built into Windows 2000 to help out with the storage and retrieval. It allows for integration with "jukebox" archival products, or can keep track of a "work queue" for manual retrieval of offline media, all very point and click and user friendly.
/dev/tape_retrieval/x123908473 >grep /poopoo/ -s -u -c -k|tar -xvf < make compile|more|less" to retrieve an image.
Of course, nobody around here wants to hear that, so why don't you just go with some Linux thing where the user has to type "cat
with time delay you dont get full frames.
heres a solution :
3 Breece Hill (or SpectraLogic OEMed) 2.75TB AIT-2 autoloaders with 2 sony ait-2 drives (66Mb/s throughput per autoloader) each - $30,000 each x 3 = $90,000
3 x Sun Ultra 2's @ $2000 each = $6,000 (yep they can do 66MB/sec and max out the breece hills and they can handle the compression/video..total throughput will be 66MB/sec x 3 = 198MB/s which should be more than enough to handle the video feed.)
AMANDA tape backup software (free download) and Sun solaris + a shell script.
30 AIT-2 50GB catridges (about the size of a cigarette pack) x 3 = 90 catridges (should fit in a small briefcase) per day for 2.75 x 3 = 8.25TB/day @ $3000/day
So..it can be done for $100,000 and running costs at $3000/day (plus 1 guy 10 minutes to swap the catridges from the autoloaders and cycle em at the end of the day)...which is fairly cheap for this setup. A small standalone sony AIT-2 drive in a PC can read back the tar files of the video if necessary. One large room should be able to hold all the tapes for a couple of months or so. still at $90,000 its not exactly cheap.
No matter how you look at it and what media you use, if you plan on storing this stuff indefinitely you'll have to go for a large (think medium - large shed size) jukebox. Even if you store it on, say, an EMC solution, you'll have to take old data offline at some point for cost and efficiency reasons.
.1TB IDE Hard drives, which would take up less space, and you don't have to worry so much about hardware failure since the drive electronics are switched daily.
That being said, remember that a run-of-the mill 16/10/40 CD-Recorder records at up to 140MB/s, which is well over your goal of 90MB/s. High-capacity CD-Rs can go to 800MB, so you'll end up writing 10,000 per day, and you'll want to use about 100 or more cd-rom drives to do the recording (for reasons of robustness, and to lengthen the average MTBF).
BUT it sets you up for an easy swap in later to switch to DVD-R, or the newest 100GB disc (vapor-hardware) technology, though you can't go much cheaper than CD-R, along with storage reliability of 5 years or so in decent circumstances, 10+ years in climate controlled lockers.
I'd imagine making CD-R cassettes that hold 10,000+ CD-Rs which simply get changed once a day (remember - 100 spindles of 100 CD-Rs is not much space, and can easily be handled by standard warehouse automation equipment). As a bonus, you can store them by day. Need a few hours on Oct 29? Just pull out one cassette. You'd have to trade out a CD-Recorder daily as well, but overall you're going to spend a LOT less than any other storage solution out there.
You can't get much cheaper than this, and the tradoffs may well be worth it to you. But it would likely be a fully custom which would take time to design. It might compare favorably to a cassette of 100
-Adam
I'd recommend finding a solution that works for 200 or so video stations, then split up your cameras/back end systems into chunks of that size.
So find a 200 video solution, and duplicate it five times, situating each site in a geographically convienient manner. A bit of coding glue on the back end might let you have a centralized index.
A single mechanism to handle that much traffic is going to run into many problems and probably won't be satisfactory. bandidth, storage, encoding time, etc for a smaller number of cameras is far more tractable for a smaller number of machines.
This also gives you a way to scale. What happens six months later, when now you have 1300 cameras to deal with? add another site. With a good template, your costs become very predictable, you have a built in mechanism for pilot testing (you will learn ALOT from building the first site), and your project time to scale up becomes easier to estimate.
The original question is basically meaningless without a description of the real project requirements.
You've put some big numbers up there, which all the hardware-heads have been happy to answer for you. But without real information, this is all just big-clock-speed hardware masturbation.
The real question is: what kind of single system/application would produce 24,000 hours of unedited high-quality video per day and storing it until the end of time?
Most respondents seem to assume that you're running a network of security cameras. If so, other posters have indicated that your quality or recording time requirements are probably 1-2 orders of magnitude too high.
If you're producing something where this video is actually going to be watched by people who care about beautiful full-color full-frame-rate production quality, where is your 200 million dollar production staff that will be watching and cataloging this data? And if you can afford that, surely you can afford at least one knowledgable systems engineer who knows how to design a storage system!
My absolute favorite bit of your post: "Reliability should be good enough to not be annoying long term". So how much lossage is "annoying"? Will you save space by randomly culling videos from the previous 7 days? If the whole system breaks down once a month, will that be annoying? If you lose one minute out of each hour, will that be annoying?
This is the sort of problem that can be designed and solved if you've got the need and budget for it. But it's not a turnkey solution. That's why you hire engineers.
Comment removed based on user account deletion
Using FPGA hardware, Rob McCready, the original poster achieved around an 88% face detection rate in real-time from black and white pictures with a 30 frames per second video camera.h tm
This explains his crazy requirements : looks like he is busy thinking about large scale applications of his work.
http://www.nce.gc.ca/en/success/9920/micronet3_e.
That'd be a storage nightmare.
I don't think so.
Let's assume one camera per VCR, full 30 fps. That's 3 8-hour tapes per day per camera, 3000 tapes a day from 1000 VCRs. 1000 VCR's should cost you $100,000 and take up one
medium sized room (power and AC will need to be enhanced). 3000 tapes per day shouldn't cost more than $3000, or $1 million per year.
You'll only need a few tape monkeys at any given instant, because they'll be around one tape needing changing every 28 seconds. A days's worth of VCR tapes (assuming we pack them in boxes with NO room to spare and stack the boxes in blocks) will take up about 1.5 cubic meters or 50 cubic feet (based on 1x4x8 inches per tape, my rough estimate). That means for a year's worth of tape you need 550 cubic meters or 20,000 cubic feet, which is 3300 square feet if piled six feet high. 3300 square feet is about the floor-size of one big house.
Question to original: Are you still sure you want to do this? If so you might be best off "spreadking the load around". IE: Don't do it all in one place. There are a million convenience store camera's and vcr systems in the world, but they're not all in one place.
Off-hand I can only think of one thing that would handle 3,000 terrabytes per year, and that's if the half million people using Morpheus donated 6 Gigabytes of space each year to your cause.
Okay, mod me as redundant, but I'd like to know what the possible real world application is...
First off, there are commercial solutions that tape 10:1 onto VHS tape... maybe you could take one of these a step further (custom ASIC dev) and do 100:1 for grayscale.
Now, assuming analog is not the way to go (sounds like you want to stay digital), first you'd probably want to do motion detection. Only capture the stream from a camera that is reading changes. Depending on the application, that can cut your storage space down significantly, especially in off-peak hours.
Second, I'd probably rethink 30 fps. 15 fps is nearly as good, and takes half the storage space. ATM cameras can range from 5 fps to 1/5 fps... and give good enough results in most circumstances.
That combination should take you down to between 1 TB and 100 GB per day (maybe less depending on usage)... which is definitely more manageable. At that point, an HSM is the answer... look at any of the other excellent posts regarding this.
I worked at one point for a major US corporation with similar needs, and they have a data backup center... it's basically a huge warehouse with tape drives, robots to change tapes and cart them around the warehouse, and tons of archive space for the said tapes. Perhaps this would be a good solution for you? As I recall, they spent about $5 MM for the hardware, and about the same for the actual installation (which is terrorist resistant, etc.)
Hope that helps!
I am disrespectful to dirt! Can you see that I am serious?!
Might I ask why your requirements include arbitrary retrieval time? I would put it to you that if you have arbitrary retrieval time, you DON'T have to save everything. Are you saying that if I built you a system that saved everything at your highest bandwidth constraints, but took a year to do a retrieval, that is acceptable? I seriously doubt it.
If you are saving that much and care that little about it, you don't need to save that much.
Patrick Doyle
I mod down every jackass who puts his moderation policy in his sig. Oh, wait a sec....
Just pay everyone on slashdot $10 to run a special version of morpheus. Then just send everyone feed from one camera. When you need to access it, just hope they're online.
Username taken, please choose another one.
I was posed a similar question at my place of employment, and the only thing I could think of that was remotely feasable was to use DVCR's or security vcrs. Thus the bulk of the information, could stay on tape, and we'd only transfer onto a computer, what we actually needed.
Free Techno/Jazz/DNB/MI Music by guys obsessed with monkeys!
Ok, here's what you do: Take all of your signals (don't even bother compressing them... as other people have pointed out, it looks like 9 GB/sec might be hard to compress) and beam them straight out into outer space. This solves both problems - the massive amount of data, and the high bandwidth.
Now, current technlogy offers few options for playback, but since you don't mind waiting for the playback (you did say arbitrarily slow), I'm sure solutions developed in the future will be both feasible and cheap. (if not, just wait a little longer). Here are a few possibilities:
Develop ultra-focused and ultra-sensitive receivers to detect the signal bouncing off of a distant object.
Teleport a big mirror ahead of the signal to bounce it back. Listen for it to return.
Develop a faster-than-light vehicle to pass the signal, and then use future technology to record it on some futuristic ultra-dense medium. (Note: this can be done obeying Einstein's theory of relativity. Space isn't quite a vacuum, so the radio signal is travelling at a speed less than the speed of light in a vacuum. You could evacuate an area of space for your vehicle to travel so that it is less dense than the space the radio signal is travelling through, and thus has a higher speed of light. You'll be able to catch up without exceeding the speed of light in your path, but exceeding the speed of light in the radio signals path).
Wait for either the universe to collapse in on itself, or a black hole to devour both the earth and the radio signal. This should, with any luck, warp space such that the signal hits the earth again.
Direct the signal at a black hole such that it orbits it. The signal will be trapped in orbit. You could send a vehicle to this area to read the signal, and then retransmit it with more energy to escape the black hole. Of course, you'll only get a brief snippet of data before the vehicle is sucked in, so many may be required. (I'm not too good at black hole theory; hope it doesn't show!)
Anyway, there are lots of possibilites!
-- morcheeba
founding member, literalist society.
HIV Crosses Species Barrier... into Muppets
I have dim memories from those lame Discovery channel 'Inside Las Vegas' of shelves of neatly arranged VCR like devices, with a couple of spare tapes next to each one.
They probably use custom long record hardware and rotate the tapes, keeping them only as long as necessary.
Bleh!
sPh
Recently at a trade conference (Crestech) I saw a demo of a system some grad students had put together which paired a panoramic low-rez camera with a moveable hi-res camera. The panoramic would observe an area, and software would recognize areas of activity and focus the other camera on it. But this isn't really what the Ask Slashdot'er was looking for, obviously.
Freedom: "I won't!"
$5 / month hosted VPS on linux = awesome!
Rob McCready, an electrical and computer engineering grad student at the University of Toronto has developed the first face-detection program that uses programmable hardware - which is much faster and more accurate at discerning faces than any existing software programs http://www.nce.gc.ca/en/success/9920/micronet3_e.h tm
I was wrong in my hasty calculations... 1 second != 1 minute...
So 140MB/minute = slower than 90MB/s. But the number of discs per day is correct.
-Adam
Do you really want to store ALL frames indefinatly?
I assume you are monitoring something (1000 cameras sounds like a whole city...) and you probably only want to keep the interresting stuff.
The way I read it, he won't know what's "interesting" until well after the events have been recorded. For example, assume that someone receives another anthrax letter, and assume that by then the post office is able to back-trace it to the original mailbox where it was deposited.
You'd pull the tapes of that location for the 24 hours between mail collections, and re-play them. Every person who dropped off an item would then be followed "back in time". Assuming a dense enough coverage of cameras (so that when they go out of frame on camera#320, they come into view on camera#319), you'd be able to follow everyone back to when they came out of their house that morning.
Then you just go to the court, get your wiretap orders and search warrants, talk to all their neighbors and offer cash rewards for incriminating information, etc. Soon someone will be rotting in jail (assuming there's room left after they've finished rounding up all the Linux programmers under the 2003 MPAA/RIAA Proud Bald Eagle Patriot Act). Voila, Freedom has been successfully defended once again!
Be seeing you...
If sending all data into 1 point is too hard.
Split it. perhaps 100 cams for 1 node not bad idea.
You can easly maintain and backup these nodes.
I believe this aproach can solve many problems.And cuts costs.
[My english is better than most other people's Turkish, so please point out mistakes politely. Thank you.]
I question why you would need to have recordings that are 20 years old that are as good as those in the last day so you could consider scaling and degrading the video quality with time.
Example: Record everything in color with high framerates and high resolution on hard disks. After the data is a week old (or whatever), compress it to B&W with half the frame rate and resolution. Repeat till you have a minimal desired quality then move it to long term storage (tape, etc).
Feel free to count me as a co-inventor- I have no application for this algorithm at this time.
-BxT
I am an Engineer for a company that does only storage, so I might be able to offer some suggestions. The best solution would probably be SamFS, which is a Hierarchical Storage Management product developed by LSC software, now part of Sun. SamFS runs only on Solaris Sparc, so that means a Sun box. Your reqs. would max out an E450, so you should look at a 4500 or 4800 at the minimum. For disk, avoid Sun T3's like the plague. They suck. For your needs, a Clariion FC4700 running RAID 3 is perfect. So perfect, that Sony just signed an OEM agreement to sell Clariions with their video editing solutions. For tape, I would suggest LTO drives in a StorageTek L700 library. SDLT is too new to be trusted. Also look at AIT-3 in SpectraLogic Gator 64000 libraries. If you have the cash, the ultimate tape solution would be STK T9940 or 9840B drives in a StorageTek 9310 powderhorn (as seen in the movie Eraser). Unfortunately, a powderhorn with no drives is about $200k, T9940 drives are $35k each, and 9840B drives are about $30k. Good luck.
first off, when i do find out were your trying to put this cameras in use, i will put on my $5 face mask and break as many of your cameras as i can.
o rage/DisplayPages/supplies.htm?DataPage=2200mx-juk ebox
secondly, cd burning silo's. im not familar with who makes them, but they do exist. your looking at 129 cd's a day so your looking at around $75 MAX to store all your data on cd-r.
this wasnt quite what i was thinking of but it might do the trick for you http://www.products.storage.hp.com/eprise/main/st
however the cost for storage is way up from using plane cd's
-- botsex is {grep;touch;strip;unzip;head;mount}
Sounds like you are going for one of the BIDS (https://www.bids.tswg.gov/) projects.
Based on other posts, I would guess your project involves face-rec in an airport. As others have noted, I think your requirements are way too high, even for the app you may have in mind.
Your res is OK, and is probably necessary - no need to change that - but I would drop the greyscale a bit, from 8 bit greyscale to 4-6 bit greyscale. If you went with 6 bit greyscale, you would end up with 230KB/frame - with 4 bit greyscale, it would be 153KB/frame.
Drop the frame rate to 15 frame/s, and you would end up with 3.5 MB/s per camera (6 bit) and 2.3 MB/s (4 bit). Do your vid compression and you would be looking at ~35 KB/s (35 MB/s) and ~23 KB/s (23 MB/s) respectively, or 2-3 TB/day - which is much more reasonable.
Even this can be improved - only record extreme movement - if you are doing face rec, you don't care if one pixel changes, or a few pixels change - even rapidly. But if a whole lot of pixels change, record that. In fact, you could even use a variable frame rate system to up the frame rate recorded, from say 0-1 FPS (little to no movement), up to 30 FPS (lots of movement, subject nearly fills camera FOV). Doing this will probably save a ton off of the bandwidth and storage requirements for the whole system - but this would be difficult to give numbers for without knowing the exact use of the system, camera placement, amount of traffic in camera path, etc. This would have to be known to decide whether to put such a system in place.
If making these changes to the requirements is not acceptable (I have seen a posting claiming you are a guy doing face-rec with an above 80% success rate, but that you need 8 bit greyscale, at 30 FPS) - you may be looking at a near impossible to implement system, without the ability to spend a lot of money for storage and processing needs. Something a government would be willing to do a few times for certain installations, but not something commercially viable, except for extreme economic situations (perhaps casinos and such).
Hope this helps...
Reason is the Path to God - Anon
Instead of trying to store 8TB a day of video, why not look at ways to reduce the video. If the video is for security reasons(that is my assumption) then why do you need 30fps? Why do you even need one frame every second?
:)
Another solution may be to take an initial image and then simply record the changes, similar to what CVS does. It's much more efficient to store just a few changes than an entire image*30 every second. This solution would probably require a lot more computing power, then it's easier to add computing power than infinate storage space.
MPEG-4 may do some of these things, if so, you already have a solution. If not, get cracking a "fun" algorthim
Sometimes I feel like a nut... Ok so it's most of the time
Of course. Frames that are 1/15 second apart are going to be roughly twice as different as frames that are 1/30 second apart. I think that's pretty obvious.
Look, if you can improve on a video compression scheme by doing something as simple as tossing every second frame, then it couldn't have been a very good compression scheme in the first place, could it?
Patrick Doyle
I mod down every jackass who puts his moderation policy in his sig. Oh, wait a sec....
Given Moore's Law, if you wait ten years, I will then lend you my palm top. It will probably be a bit overkill, but hey!
-
Faster throughput
- Less mechanical wear on the tape
- higher density on the tape
Dumping to disk first will also mean that you can use off the shelf backup software (like NetBackup, mentioned before). Given the kind of moneyo you're going to be spending, it's probably going to be worth it.If you can do it, alternate between dumping to one disk array and reading from another (better yet, go through three, so you have a bit of buffer) you can get an effective increase in the effeciency of drives if they're not seeking to write and read at the same time. Obviously there will be a real advantage to using RAID.
It would also be to your advantage to have multiple CPUs controlling the tape drives. Each one would have it's own small farm. You should be able to have multiple CPUs feeding the drives on one tape library.
Given that you're going to need tape monkeys feed the tape library, it may be worthwhile to not use a tape library, but I'd suggest that the drives be in at least small tape libraries. The reason for this is that a tape library can read the bar code on the back of each tape as it goes into the drive .. Otherwise you are almost sure to have errors logging which tape was where/when. With the volume of tapes you're going through it could be absolute hell trying to track down a mis-labeled tape. A small tape library would also allow you to keep drives in more constant motion.
The last thing is to make sure that you have more drives than you 'really' need. With many drives in constant use they will break down from time to time. Make sure that you keep that in mind when you design both your hardware and your software. The horrid thing is that, if they're reasonably well built, failure could be clustered (identical drives with similar usage).
For 90MB/sec Super DLT promises 10MB/sec which means you'll need at least 10 drives. I'd budget for 15 drives -- it gives you some reserve capacity, and allowance for things like non- streaming and occasional drive failure (in spite of their lofty promises) and people who want to read the tapes. It's usually easier to get the budget for the extra drives when you start the project, than after you run into the inevitable problems.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
TeraGlobal can do 100:1 wavelet-based compression in real time on any of the G4 macs. IIRC, they were actually able to compress one stream and decompress another simultaneously.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
First, a disclaimer...
I work for IBM SAN Support, so I am obviously most familiar with our products, which are indeed quite good, but there are several sources for most of this stuff (except the tape drives).
Here's what you are going to need...
1) A big honkin' network and fast video processing boxes (This is the most odious requirement, the rest is quite easy)... You are going to need some serious bandwidth to pipe this stuff. Unless your compression is onboard, Gigabit ethernet is your only option for multiple cameras. (You would be pushing the bandwidth for Fast Ethernet on an uncompressed stream) Even then, you aren't going to be able to pipe 1GB/s through the box, because you are probably not going to get an adapter to go at line speed. Also, unless video compression is a lot easier than I thought, you are going to need an damn fast machine to handle that many compression streams simultaneously.
There may be some professional boxes with an embedded MPEG encoder so you don't have to try and to that heavy lifting on a server. (I know the chips exist, as IBM sells them (they are used in the TiVo)) Where would you get such boxes? I don't know.
2) I am no sizing expert, but you are going to probably need, oh, about 3 impressive servers (read: plenty of 64-bit PCI slots, perferably on separate busses) to handle data "buffering" and tape backup chores. These servers would need Fibre Channel boards for the I/O, and of course your network adapters. Two of each.
3) Tape drives: Piece of cake. IBM happens to sell an LTO (Linear Tape Open) library, the 3584, that will do nicely. Fully configured, it has a half-petabyte capacity using 150GB carts. The max. theoretical spindle speed w/o compression (which would be useless here) is 15 MB/sec. My actual observed field speeds are around 10-12 MB/sec. (Backing up the uncompressible Win2k SP2.) Each drive frame can accommodate 12 drives, and you can bolt together up to six frames to form a library. (There can also be drive-less "expansion frames") The tapes aren't cheap ($150/each, IIRC) but still aren't bad. The LTO drives are currently the fastest and largest drives on the market, and I don't think anyone other LTO vendor sells a big honkin' library like the 3584. As far as your automation needs, the robotics can easily handle the puny number of daily changes you need, although you are going to be making a LOT of use of the 10-slot "I/O" door to insert and remove tapes.
4) It sounds like you need a highly avail. box for disk storage. A Symmetrix is a popular, but expensive, choice. IBM of course has the 2105 "Shark" box, which is just as good, and not nearly as pricey. I have heard Hitachi also makes a pretty decent storage machine. You don't need a very large one, because you only need to hold the data long enough to pipe it to tape, however, I'd suggest at least a day's worth. I think you will need more than one though, as 180 MB/sec of I/O is probably going to be rough on any box. (I really don't know what the I/O bandwidth of those boxes are, though. That's what sales drones are for.)
5) Infrastructure equipment. I reccommend the InRange FC-9000 fibre channel switch for your SAN. Non-blocking and quite reliable. I know nothing about network equipment.
5) Someone that knows real backup software cold... Veritas NetBackup or Tivoli Storage Manager (TSM Rocks!) is what you need, but it is not trivial to configure.
SirWired
Hey,
Ok, say you have 1000+ cameras emitting 30 frames/second worth of 640x480 grayscale video...and you have to store it indefinitely. What do you do?
You cut down on data. First of all, you don't want 30fps. PAL video is 25fps; You don't need that sort of quality. 15fps would be more than enough, probably even less.
Second, you don't record stuff that doesn't change. You need a codec that supports 'Map identical pixels to transparent'. That way, only changed data need be recorded, cutting down on file size substantially.
Thirdly, you delete all-identical frames. For all-still areas like stairwells, this will cut right down on files.
Fourth, use a good CODEC. With greyscale and not too much data to store, you can get VERY small files.
With all these measures, you won't need as much storage space as you estimate. It would be variable though; high-activity areas with all-day traffic would still produce a lot of data. I'll assume it'll output 30 kilobytes per second, being on the pessimistic side.
To store all the resultant data, I'd use a two-tier system of PC computers. I note you havn't mentioned how you plan to digitise all this data, but that's out of the scope of this post.
Let's connect 10 cameras to each PC, and we'll have 100 PCs. 30KB/s times ten cameras is 300 kilobytes every second. That would be *easy* with Fibre Channel (2 Gbits/sec!!). 300kbps for an hour would be 1055MB/hour, about a gigabyte every hour. If we can get an IBM 73 Gig drive, that'll give us just over 3 days of storage per computer.
I'd have the computer accepting the telemetry, compressing it and writing it to the drive. Every 24 hours, we'll open a new file. Each computer connects by 100Mbps ethernet to another computer. Every day, this will copy down the 24GB file from the last day, and record it to a DAT tape. Say we get 120 Gigabyte tapes, that's a tape every five days, times 100 computers, that's 20 tapes per day.
After this, decide on a set archive time. Keep the tapes for, say, 5 years. You'll need a good indexing system and a warehouse for tape storage (You can get automated, robotic systems), but if you have 1000 cameras, I expect you have a fair budget.
That's what I'd do, anyway.
Michael
"Goodness me, how unlike the FBI to abuse the trust of the American public." -- The Onion
Any video compression algorithms worth using for this kind of application do comparisons from one frame to the next, and only compress the differences (except for occasional reference frames.) Some of them do substantial motion compensation to model the differences, others don't. Many of them let you tweak the frequency of reference frames - is it every 10? Every 100? Do you need the ability to go backwards, or is smooth forward and clunky backwards good enough?
Very few locations actually generate much motion on a 24-hour basis, except for road traffic cameras, and I'd be extremely surprised to see an application need to store those on a long-term basis (as opposed to storing for a week or so in case there are traffic accidents - anything you need longer than that should probably be handled by license-plate recognizers.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
You see? You see? Your stupid minds! Stupid! Stupid!
*Obviously* you do the compression out near the cameras; anything else would be silly, with the possible exception of doing lightweight compression at the cameras and heavier-duty compression at a centralized server, if you're in a LAN / Metro area network where bandwidth is cheaper than distributing lots of hardware. But you still need to figure out what level of data transmission is realistic - "90KB/sec" is 720 kbps, which is half a T1 line. (As a telecom vendor, I'd be happy to sell you 1000 T1s or equivalent Internet or FrameRelay/ATM bandwidth :-), at least assuming you're in the US where I've got bandwidth to sell instead of subcontracting. But your problems are much different if you're trying to camerafy every FooBar Retail Store in North America vs. trying to camerafy every street corner in Toronto.) But do you really need this much? Most business video-conferencing is 384kbps or 128kbps,and even 64kbps can be surprisingly good, especially since you don't mind grayscale. This lets you use DSL or ISDN to connect your remote sites (or if your cameras are in groups, lets you put more cameras per T1 from each group going back to your central locations.) This also means that your central location (or locations, depending on your disaster recovery needs and your bandwidth-vs-operations needs) can get by with a much more realistic data connection - if there's any long-distance component to the communications, you'll be much happier buying an OC3 than an OC48 or GigE, though in a metro area, if there's dark fiber, it doesn't much matter what speed you run at.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Where I work, we have been having problems of people busting the windows and grabing whatever they can before the security people show up.
B/W cameras are very cheap (it cost more for the box and wire than the cameras) but recording the video is very expensive. Since I don't live in the land of low cost NTSC, I'm forced to go with PAL and the result is that a single VCR that can record for 24 hrs cost about as much as a well equiped PC. I can get a switch box for 4x the cost of the cameras that will switch between 4 cameras.
So whats the best option? How many vidoe capture cards can I put in one cheap PC? What compression can I use? I suspect if I use a multplexor then the compression will be very bad. How stable is video4linux with 4 cameras going at once?
This is an interesting problem I think because of the data longevity requirements.
w ar e/Storage/)
It will be extraordinary and expensive, but you can certainly get 100+ terrabyte solutions that are fully automated, both disk and tape. There is a small (almost entirely North American-based) industry and these things are used by a variety of people for a variety of uses - commercial, scientific, defense...
You can certainly drop cash over a fairly large range and to play with various speed/reliability/support tradeoffs; your budget will dictate how quickly you can get data back at a given age. One thing is certain: you're going to end up with some large, impressive looking pieces of equipment suitable for display in a harshly lit, well air conditioned space.
People have discussed brands and preferences - there is some competition in the marketplace. You will want to make a big project of shopping around. Have a good read...
(http://directory.google.com/Top/Computers/Hard
Call people up. They will certainly lavish attention on you if you give them the same spiel you gave us. Ask vendors who their competitors are - it's not taboo, and they will tell you, if they're confident their own solutions are better.
Moving right along. Depending on how important retrieval speed is at various ages of the data, you may opt for a very, very big hard drive array (think EMC). Regardless, I expect you'll ultimately be thinking tape output, and a very, very big tape library (or three, depending your data's popularity in relation to its age) (think StorageTek), and develop a regimen for emptying it and storing the tapes - probably in a very expensive safe place.
All this you can do. The relative price/performance to existing analog systems and "package deals" by existing security vendors (there's a google category for them, too; exercise for the reader) is up in the air, but you're not landing on the moon, and my guess is there are already some (<50?) systems like yours running as we speak.
BUT... and now it gets interesting: your system has one very unusual requirement. "Indefinitely" is a long time. And the system you chose to archive your data with has been designed for redundancy, not archival storage. Tapes tend to have a high rate of failure, and their rated lifetimes tend to be _short_ (generally 3-5 years, as it is, for instance, with DAT or AIT or DLT). If indefinitely is really "3-5 years" then fine, all good. However, if it's not...
Interestingly, tape capactities are losing the race against hard drive capacities quite spectacularly, so, if you just want to buy yourself a "little" more time (maybe 5-10 years instead of 3-5) you can (arguably) consider archiving the hard drives directly. This will cost more, of course, both in drives (my intuition says it will about double your ongoing media costs vs tape, unless you can find a way to use EIDE drives, in which case it might even be cheaper than tape - !) and in making sure your storage facility has a well-conditioned environment. But the fact remains, select your drive vendor well (and you will be able to develop an excellent relationship with a manufacturer, I'm sure), drives will last longer than tapes under the right conditions. Maybe twice as long. Maybe longer.
Of course, you will get HD failures too. And this is ultimately a troubling regimen because drives are complicated and have many moving parts - stiction, fragility, etc... Because of vagaries of the manufacturing process, some drives will last 50 years, and others 50 days. And anyway, maybe for you, indefinite is like unto the next generation...
In which case, you pretty much have to go optical. DVDs might prove very attractive. Given factors of size/weight/storage capacity even the commercial stuff available now doesn't look _so_ bad, and I would personally think about calling Sony (for instance) and chatting with them about it. There might be higher capacity formats you can look into (trading off vs. price). I expect you can find someone who makes a DVD-R robot that would fit the bill.
Your DVD-like media has exceptional durability and will last... well, a long time. I say this because I spent some time a while ago attempting to find a longevity rating for DVD/DVD-R/DVD-RAM media (when we were considering an archival project), calling all over hill and dale, and I finally got the answer that there really isn't a rating, because really nothing has failed yet due to "age," and based on the materials they don't have an "expectation of failre." Take that with a grain of salt - it's the manufacturer talking - but still... plastic lasting forever has some kind of upside at least.
Ideally in such a situation you want to be able to skip the tape step and go right from hard drives to optical. Your application's needs will dictate whether or not you can get away with that. Still, interesting to think about it. The time capsule people in Denmark were talking about using specially treated plastic paper to hit a 1,000 year lifetime, so it all comes full circle eventually...
We're on the road to Tycho.
but didn't tell the application.
;-) (don't get ruffled on that, I can't administer it alone either!:P)
... beyond ya
Sorry, but I've chatted with people about hospitals and gone thru the proofs on the load and distribution and backups (can you imagine what happens in an ER if the xray sent up for soft display goes down because some idiot cut the fibre?)
There's alot of room left to wiggle- you yourself have punched the numbers in and know what it takes. The first thing I thought of when I read the post is realtime face detection / catalog, database building, surveiliance for a small city / Olympic Games.
With that much video feed coming in you either have to have 1000 dedicated operators or some sort of computer assisted recognition. Or come to face the tape at a later time.
Very few cameras on the market will give you 30 fps non-interlaced. If you are willing to spend 5% of the budget of these 4K+ cameras... well, guess that gives an overall idea how much the budget is.
Getting ideas on Slash dot is a great step, but it is in no way the a substitute for a thorough analysis of your bandwidth, backup, routing, and storage requirements. Period. Don't think you can administer this alone
If you can have a decentralized storage, that helps. That also approximately doubles your cost as redundant arrays are needed for each location.
Anyway, get that analysis done- please! You may learn that the technology you have to use is priced
Put 4 realtime MPEG-2 encoders hooked up to cameras in a machine and call that a node.
You will need 250 of these machines to take care of your 1000 camera requirement.
Each node will require a RAID array capable of about 6MB/s sustained write.
If a typical MPEG-2 stream is about 900KB/s, then youre looking at around 3 gigabytes an hour per camera. That works out to about 2TB per week per node.
You are probably best just to archive the drives, and replace them weekly, as backing them up to tape will be too slow.
2TB per node x 250 means 500TB of disk drive to be archived and swapped into RAID enclosures per week.
My math may be off - i just threw this together, but this will be a total logistical nightmare, no matter which way you look at it.
Lets say you can use IDE disks - 100GB for US$300. Thats 20 x 250 disks per week, which is $US 1.5 million per week.
Over a year, youre looking at $US 78 million in drives alone, though thte real figure might even be sub $US50 million if you consider the rate at which drive prices plummet.
i would guess $US6000 per node including cameras, so you'd be looking at $US 1.5 million in setup hardware.
Space to archive the disks and staff to shuffle the drives is not included.
All in all, it would be expensive, but you could do this for about a century for less than the cost of a B2 bomber.
I gots ta ding a ding dang my dang a long ling long
. (The break-even is arounf 5-6k$. Below ~3TB, drives win and vice versa.)
My argument was just that the break even point is climbing, and it could be climbing really quickly. Maxtor is already ready to jump from 100GB IDE to 160GB IDE. I'll also avoid IDE-SCSI arguments, since this isn't really about that. The same roughly holds true of SCSI drives, just at 3X the price.
I've had enough abrasive sigs. Kittens are cute and fuzzy.