Slashdot Mirror

← Back to Stories (view on slashdot.org)

Large-Scale Video Archiving?

Posted by Cliff on from the now-that's-a-lotta-data dept.

BondHeadGuy asks: "Ok, say you have 1000+ cameras emitting 30 frames/second worth of 640x480 grayscale video...and you have to store it indefinitely. What do you do? This is a real question, believe it or not. 30 frames/s * 300 KB/frame = 9 MB/s per camera. 100:1 video compression brings that down to ~90 KB/s. But 90 KB/s * 1000 cameras = 90 MB/s, or ~8 terabytes/day. Retrieval, though, can be essentially arbitrarily slow. Reliability should be good enough to not be annoying long term. Is there a solution that: has 8 TB/day storage capacity, can handle the 90 MB/s write speed, and lets you save some bucks on the (slow) read side?"

3 of 494 comments (clear)

  1. Large Scale Storage by follower · · Score: 5, Informative

    What your really looking at is some kind of Heirarchical Storage Solution. What happens is that once you have predetermined how much data will be saved from the camera each night. You can get some kind of disk array to store it on. That disk array will also be attached to some kind of HSM solutions such as what is provided by StorageTek's SAMFs. That solution will automatically backup the data that is stored on your disk and remove it from your disk so new data can be stored on the costly disks. From now on your OS and applications think that the data is on disk but in reality its on tape. When the data is requested the software will automatically get it from tape and place it back on the disk. This can be rather costly however.

  2. Fibre Channel by Erk · · Score: 5, Informative

    Go with a FC solution - stay away from EMC, as they will try to sell you a massive Symmetrix for your needs. Sounds like you need a building block approach, one block a day. Doesn't need to be TOO fancy, eh?

    Here are some options for FC disk storage:

    - Sun T3
    - EMC Clariion
    - Compaq Storageworks
    - HP VA7400 -- my fav

    Just to warn you, you're looking at something on the order of 20k/day to operate this setup... now, I'm sure the price would go down QUITE a bit if you're purchasing 8-10TB a day, but even still, it's a huge cost.

    I looked at a 10TB solution from the above vendors, and the cheapest I got it was $0.0425/MB!

  3. Re:Face Recognition Application by BondHeadGuy · · Score: 5, Informative

    Actually, I developed (past tense) a hardware-based object detection system. Faces are interesting objects to try and learn to detect, because they have intresting and considerable variation. They also make for a fun demo and get more press. But I could have been detecting anything. Recognition, on the other hand, is different; it is the detection of individual faces rather than faces in general, and it would be nearly impossible to put entirely into a hardware system because of the large face database required.

    That system rocked. But it has nothing to do with this post.

    Rob