Large-Scale Video Archiving?

BondHeadGuy asks: "Ok, say you have 1000+ cameras emitting 30 frames/second worth of 640x480 grayscale video...and you have to store it indefinitely. What do you do? This is a real question, believe it or not. 30 frames/s * 300 KB/frame = 9 MB/s per camera. 100:1 video compression brings that down to ~90 KB/s. But 90 KB/s * 1000 cameras = 90 MB/s, or ~8 terabytes/day. Retrieval, though, can be essentially arbitrarily slow. Reliability should be good enough to not be annoying long term. Is there a solution that: has 8 TB/day storage capacity, can handle the 90 MB/s write speed, and lets you save some bucks on the (slow) read side?"

Comments

Tape

[mfarver]

Gonna be expensive,

How long does the data need to be stored for? Tape is good if indefinate storeage is not a requirement. (Tape degrades fast.. but is reusable)

Terabyte tape libraries are fairly common. Check out any of the major datacenter manufacturers. Sun and HP both have a unit of about 7TB. But you're talking several 100k$ for a fully automated unit.

Cheapest route would be to go back to the dark ages. Buy a bunch of 100GB tape drives and lots of tape (70 tapes a day ain't bad). Hire a few minimun wage tape monkeys to change tapes on command. Setup a LED display or a big monitor for the computer to flash tape change commands on. (Old IBM trick)

Mark

What are they filming?

[KFury]

If things aren't actually moving in most of the shots (ATM or warehouse surveillance cams, for example) then you'll be able to get far better than 100x video compression.

Also, how much a factor is comunication. 1000+ cameras ona LAN or WAN?

Any secondary logging going on here? Any metadata (ATM transactions, notes, etc.) that should be stored along with the media? Do you want to use this data for easier access? Is there any preprocessing (facial recognition)?

You mentioned recall could be arbitrarily slow, but if it's possible to speed it up with only small changes, is it worth it?

Feel free to ignore these questions. Largely I'm just curious about something you probably can't talk about, but then again as a systems engineer, I'd find it difficult to recommend a solution without knowing more factors that could impact on ways I can't think of until I know more factors...

Face Recognition Application

[cwhittenburg]

The guy who submitted this story "Bondheadguy" resolves to RobJMcCready@yahoo.com... A quick search on "Rob McCready" yields a University of Toronto grad student (or maybe former grad student now) who is developing hardware based face recognition equipment. Check out This link...

Now you can make your own decision about helping him out (or not).

Re:Real-time 100:1 compression?

[dublin]

Disclosure: I am VP/GM of Conservor, whose product & service offerings are discussed in this post.

At Conservor, we offer a new set of IP-based Storage Area Network services that offer the speed, capacity, and features of traditional Fibre Channel based SANs, but at an order of magnitude less cost. This approach uses Gigabit Ethernet to make disk and/or tape resources appear as locally attached devices at very nearly wire speed.

We have been told by our partners that our IP SAN experience is considerably broader and deeper than even the largest consultants such as EDS, Accenture, and IBM Global Services. We are also experts at storing and managing VERY large datasets - we work routinely with the oilfield service and exploration companies to do exactly this sort of thing with very large 3D seismic datasets. It's not clear from your post, but you may well need some type of content management system, as well, to ensure propoer indexing and speedy retrieval of such a volume of data, and we can do that, too.

Your write speed and capacity requiremeents, while larger than normal, are not a problem, and can be accommodated without having to resort to exotic
technologies. Of course, there aren't enough details to propose even a rough solution from what you've posted, but it sounds like a tape solution. Still, if your retention time is not too long, we might even be able to do it with disk (we can provide high-speed Fibre-Channel storage arrays as little as 2 cents/MB: MUCH LESS than the competition, and capable of RAID 3, which you may want to use if video streaming for playback is much of an issue - we're working with next-gen cable headend guys on this stuff, too.)

As for the tape components, I need much more info to be able to even speculate on your needs. Anything from mid-range high-performance LTO libraries to full-scale mainframe-type 3590 silos may be needed, depending on a number of variables.

All our solutions are also available as complete service offerings, preventing you from having to acquire, own, or maintain any hardware, software, or management staff. In addition, since our fee is entirely for the service, it is a tax-deductible operating expense, which most companies find quite attractive. (We can also make everything look like capital for those companies (Real Estate, etc.) that want to capitalize everything in an effort to boost EBITDA.)

Storage is changing - there's no reason to do things the old way anymore, when there are better and cheaper solutions at hand. Some of the big guys in storage are going to learn this the hard way...