Slashdot Mirror
Large-Scale Video Archiving?
BondHeadGuy asks: "Ok, say you have 1000+ cameras emitting 30 frames/second worth of 640x480 grayscale video...and you have to store it indefinitely. What do you do? This is a real question, believe it or not. 30 frames/s * 300 KB/frame = 9 MB/s per camera. 100:1 video compression brings that down to ~90 KB/s. But 90 KB/s * 1000 cameras = 90 MB/s, or ~8 terabytes/day. Retrieval, though, can be essentially arbitrarily slow. Reliability should be good enough to not be annoying long term. Is there a solution that: has 8 TB/day storage capacity, can handle the 90 MB/s write speed, and lets you save some bucks on the (slow) read side?"
Gonna be expensive,
How long does the data need to be stored for? Tape is good if indefinate storeage is not a requirement. (Tape degrades fast.. but is reusable)
Terabyte tape libraries are fairly common. Check out any of the major datacenter manufacturers. Sun and HP both have a unit of about 7TB. But you're talking several 100k$ for a fully automated unit.
Cheapest route would be to go back to the dark ages. Buy a bunch of 100GB tape drives and lots of tape (70 tapes a day ain't bad). Hire a few minimun wage tape monkeys to change tapes on command. Setup a LED display or a big monitor for the computer to flash tape change commands on. (Old IBM trick)
Mark
The first solution I would look at is how they do this at the casinos in Vegas (or anywhere else). Everytime I see one of those 'On the Inside' shows about casinos on TLC/Discovery Channel, they are always boasting about their video camera capabilities, and their ability to archive everything that happens. Whatever solution you find there, while not always the most cost effective, has definately been tested in an environment on the scale of 100's, if not 1000's of cameras.
sig--we don't need no goddamn sig
Still, it's fun to read about
Are we presuming that there will be action most of the time in all 1000+ cameras? If not, then why store the images where there is no action? I'm doing a similar digital surveillance thing, albeit much smaller. (7 cameras, 1 fps, only at night [when there's not much action anyway]) My images all go to jpeg's and I wrote a little C program to throw away all the "similar" images. My algorithm is somewhat conveluded, but more-or-less only keeps the images if they differ by more than a certain amount. I'm sure video compression schemes like mpeg would pretty much take care of this if you're storing to video format. Storing to jpegs has the benefit of being able to easily pick out any time stamp, but I can't watch them like a video.
A simpler solution might be to put a motion detector on each camera and have them only record when there is motion. Using your 100:1 image compression, you estimate 8 terabytes/day. I would expect you could get (warning: pulled from thin air) 1/10th of that by ignoring anything that isn't moving.
But then you have a quandry: was there really no motion at camera #469 at 12:30am, or was something just broken?
Are you capturing in digital format? Are you sure that your systems are even capturing at 30fps?
It's unlikely that any digital conversion device(s) would be able to handle the input from 1000+ cameras and then be able to get that data to a central storage location through a local network.... the bandwidth needed for something like that would be incredible (90 MB/s ??), perhaps even requiring it's own seperate gigabit fibre network. Even in a high-end server with several devices connected to it you'd be lucky to capture 20fps.
But if you were able to get it done, storage options would probably consist of some type of RAID array (with a HUGE number of disks to be able to hold 8TB/day).
Storing that much data indefinately would require enormous rooms dedicated to storage devices, which may not be feasible. Storing data for a week or even a month would be a challenge in itself.
Having things in digital format is nice for indexing and fast retrieval, but in this case it may be too costly. Storing data on video tape may not be as fast or convenient, but it's much easier to store 2 twelve-hour tapes per day per camera than it is to set up and maintain 8 Terra-bytes of hard drive space per day.
If things aren't actually moving in most of the shots (ATM or warehouse surveillance cams, for example) then you'll be able to get far better than 100x video compression.
Also, how much a factor is comunication. 1000+ cameras ona LAN or WAN?
Any secondary logging going on here? Any metadata (ATM transactions, notes, etc.) that should be stored along with the media? Do you want to use this data for easier access? Is there any preprocessing (facial recognition)?
You mentioned recall could be arbitrarily slow, but if it's possible to speed it up with only small changes, is it worth it?
Feel free to ignore these questions. Largely I'm just curious about something you probably can't talk about, but then again as a systems engineer, I'd find it difficult to recommend a solution without knowing more factors that could impact on ways I can't think of until I know more factors...
Kevin Fox
Now you can make your own decision about helping him out (or not).
You are right on the money.
As odd as it may sound, good 'ol analog VCR's are the best and cheapest way to go for this sort of thing.
Casinos monitor every gaming table using 1/2 speed VHS tapes manned by operators swap tapes on a regular basis.
If there were an affordable (or even a not-so affordable) digital solution that provided what the original questioner wanted, the casinos would be using it.
Conformity is the jailer of freedom and enemy of growth. -JFK
not for nothing, but is this for either
1. a reality based web-tv show
2. some bizarre web porn thing
3. some actual legitamite venture
4. security issue ?
You need to ask this the week after the anti-terrorism bill made it through the Sentate?
You ARE being watched (but don't worry as long as you don't do anything wrong.)
Waltz, nymph, for quick jigs vex Bud.
Yes, or something close to it. See this Slashdot comment:
The guy who submitted this story "Bondheadguy" resolves to RobJMcCready@yahoo.com... A quick search on "Rob McCready" yields a University of Toronto grad student (or maybe former grad student now) who is developing hardware based face recognition equipment. Check out This link...
Now you can make your own decision about helping him out (or not).
Yes, casinos do have that much video; if you can get them to talk about it (after dealing with them for less than six months), my hat goes off to you. =)
Actually, the main problem with checking with the casinos is that they themselves are trying to move to digital. Untill recently, the technology simply was not available, so they were stuck using analog (and rack after rack of tapes).
You're right about the capture though, you need to either use cards that do the processing or move it to seperate machines (video encoder/decoders).
As for the server downtime, getting a reliable video server, setting up everything correctly, and going from there means very little (if any) downtime. This depends on how much money you are willing to spend; remember many tv stations broadcast from digital signals, and their downtime is minimal.
Prolly depends on the casino.
About 3 years ago I toured the security area of an indian casino, they had 2 vcrs for each recording node (some vcrs recorded 4 cameras)
the operator could cut over to the other vcr instantly to rewind the other tape to review something. people at tables frequently request that the camera be rolled back when a dealer or customer knocks over chips or does something clumsy and/or stupid.
i remember the security dude saying that they hold everything for 90 days, then erase everything except for cheating or other incidents that may require legal action.
Conformity is the jailer of freedom and enemy of growth. -JFK
You would still need to be doing real time compression, otherwise before you finished compressing the first days video (i.e. if it took more than 24 hours), you would get the second days video to compress.
Recently at a trade conference (Crestech) I saw a demo of a system some grad students had put together which paired a panoramic low-rez camera with a moveable hi-res camera. The panoramic would observe an area, and software would recognize areas of activity and focus the other camera on it. But this isn't really what the Ask Slashdot'er was looking for, obviously.
Freedom: "I won't!"
You forgot to add that the video tape itself will only last between 10 to 15 years before they start to deteriorate. At least thats what a TV engineer told me once. Something to do with the binder used to keep the metal particles on the tape breaking down and the magnetic particles flaking off.
If you want more info, I will grab the e-mail from him and post as a followup
WTF? Over?
Rob McCready, an electrical and computer engineering grad student at the University of Toronto has developed the first face-detection program that uses programmable hardware - which is much faster and more accurate at discerning faces than any existing software programs http://www.nce.gc.ca/en/success/9920/micronet3_e.h tm
If sending all data into 1 point is too hard.
Split it. perhaps 100 cams for 1 node not bad idea.
You can easly maintain and backup these nodes.
I believe this aproach can solve many problems.And cuts costs.
[My english is better than most other people's Turkish, so please point out mistakes politely. Thank you.]
Disclosure: I am VP/GM of Conservor, whose product & service offerings are discussed in this post.
At Conservor, we offer a new set of IP-based Storage Area Network services that offer the speed, capacity, and features of traditional Fibre Channel based SANs, but at an order of magnitude less cost. This approach uses Gigabit Ethernet to make disk and/or tape resources appear as locally attached devices at very nearly wire speed.
We have been told by our partners that our IP SAN experience is considerably broader and deeper than even the largest consultants such as EDS, Accenture, and IBM Global Services. We are also experts at storing and managing VERY large datasets - we work routinely with the oilfield service and exploration companies to do exactly this sort of thing with very large 3D seismic datasets. It's not clear from your post, but you may well need some type of content management system, as well, to ensure propoer indexing and speedy retrieval of such a volume of data, and we can do that, too.
Your write speed and capacity requiremeents, while larger than normal, are not a problem, and can be accommodated without having to resort to exotic
technologies. Of course, there aren't enough details to propose even a rough solution from what you've posted, but it sounds like a tape solution. Still, if your retention time is not too long, we might even be able to do it with disk (we can provide high-speed Fibre-Channel storage arrays as little as 2 cents/MB: MUCH LESS than the competition, and capable of RAID 3, which you may want to use if video streaming for playback is much of an issue - we're working with next-gen cable headend guys on this stuff, too.)
As for the tape components, I need much more info to be able to even speculate on your needs. Anything from mid-range high-performance LTO libraries to full-scale mainframe-type 3590 silos may be needed, depending on a number of variables.
All our solutions are also available as complete service offerings, preventing you from having to acquire, own, or maintain any hardware, software, or management staff. In addition, since our fee is entirely for the service, it is a tax-deductible operating expense, which most companies find quite attractive. (We can also make everything look like capital for those companies (Real Estate, etc.) that want to capitalize everything in an effort to boost EBITDA.)
Storage is changing - there's no reason to do things the old way anymore, when there are better and cheaper solutions at hand. Some of the big guys in storage are going to learn this the hard way...
"The future's good and the present is nothing to sneeze at." - Roblimo's last
I question why you would need to have recordings that are 20 years old that are as good as those in the last day so you could consider scaling and degrading the video quality with time.
Example: Record everything in color with high framerates and high resolution on hard disks. After the data is a week old (or whatever), compress it to B&W with half the frame rate and resolution. Repeat till you have a minimal desired quality then move it to long term storage (tape, etc).
Feel free to count me as a co-inventor- I have no application for this algorithm at this time.
-BxT
first off, when i do find out were your trying to put this cameras in use, i will put on my $5 face mask and break as many of your cameras as i can.
o rage/DisplayPages/supplies.htm?DataPage=2200mx-juk ebox
secondly, cd burning silo's. im not familar with who makes them, but they do exist. your looking at 129 cd's a day so your looking at around $75 MAX to store all your data on cd-r.
this wasnt quite what i was thinking of but it might do the trick for you http://www.products.storage.hp.com/eprise/main/st
however the cost for storage is way up from using plane cd's
-- botsex is {grep;touch;strip;unzip;head;mount}
I was asked to do an identical project several years ago for a major networking company. The numbers in terms of data have not really changed, but the costs have sure gone down to do it.
Here's what I recommend:
1. Reduce your FPS to, say, 10
2. Use a HFS model
3. Realize that what you are talking about requires an unbelieveable amount of time, money, and intelligent design. If your short on any, give up immediately.
Good luck.
-
Faster throughput
- Less mechanical wear on the tape
- higher density on the tape
Dumping to disk first will also mean that you can use off the shelf backup software (like NetBackup, mentioned before). Given the kind of moneyo you're going to be spending, it's probably going to be worth it.If you can do it, alternate between dumping to one disk array and reading from another (better yet, go through three, so you have a bit of buffer) you can get an effective increase in the effeciency of drives if they're not seeking to write and read at the same time. Obviously there will be a real advantage to using RAID.
It would also be to your advantage to have multiple CPUs controlling the tape drives. Each one would have it's own small farm. You should be able to have multiple CPUs feeding the drives on one tape library.
Given that you're going to need tape monkeys feed the tape library, it may be worthwhile to not use a tape library, but I'd suggest that the drives be in at least small tape libraries. The reason for this is that a tape library can read the bar code on the back of each tape as it goes into the drive .. Otherwise you are almost sure to have errors logging which tape was where/when. With the volume of tapes you're going through it could be absolute hell trying to track down a mis-labeled tape. A small tape library would also allow you to keep drives in more constant motion.
The last thing is to make sure that you have more drives than you 'really' need. With many drives in constant use they will break down from time to time. Make sure that you keep that in mind when you design both your hardware and your software. The horrid thing is that, if they're reasonably well built, failure could be clustered (identical drives with similar usage).
For 90MB/sec Super DLT promises 10MB/sec which means you'll need at least 10 drives. I'd budget for 15 drives -- it gives you some reserve capacity, and allowance for things like non- streaming and occasional drive failure (in spite of their lofty promises) and people who want to read the tapes. It's usually easier to get the budget for the extra drives when you start the project, than after you run into the inevitable problems.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.