Large-Scale Video Archiving?

BondHeadGuy asks: "Ok, say you have 1000+ cameras emitting 30 frames/second worth of 640x480 grayscale video...and you have to store it indefinitely. What do you do? This is a real question, believe it or not. 30 frames/s * 300 KB/frame = 9 MB/s per camera. 100:1 video compression brings that down to ~90 KB/s. But 90 KB/s * 1000 cameras = 90 MB/s, or ~8 terabytes/day. Retrieval, though, can be essentially arbitrarily slow. Reliability should be good enough to not be annoying long term. Is there a solution that: has 8 TB/day storage capacity, can handle the 90 MB/s write speed, and lets you save some bucks on the (slow) read side?"

Comments

Analog video systems still work

[jgrider]

Although you are probably looking for a digital solution, don't overlook the solutions that already exist. Security camera VCR's (available at RadioShack et al.) can put 24 hours (or more) of video on a single VHS tape. Get a few VCR's (at $200 each), and a pallet of VHS tapes at Sam's club, and you could record all the video you want!

Are they looking at normally still settings?

[RobKow]

Like a security camera in a stairwell or something? In that case you can use motion detection to start/stop recording and save well over 100:1. The choice of video codec is going to be important if it's for security (so faces, etc. can be recognised), but if not, you can crank the compression ratio up quite high on most codecs, especially the video codecs that do frame-by-frame motion differencing (i.e. not MJPEG).

300k/sec???

[Foss]

300k/sec seems very excessive. You could try converting it all to mpeg4 with a DivX encoder (http://www.divx.com) and that should compress it right down. If you've got sound in there too, strip it out or at least convert to MP3.

You can do all this with a great program called Virtual Dub (http://www186.pair.com/vdub/)

Tape

[mfarver]

Gonna be expensive,

How long does the data need to be stored for? Tape is good if indefinate storeage is not a requirement. (Tape degrades fast.. but is reusable)

Terabyte tape libraries are fairly common. Check out any of the major datacenter manufacturers. Sun and HP both have a unit of about 7TB. But you're talking several 100k$ for a fully automated unit.

Cheapest route would be to go back to the dark ages. Buy a bunch of 100GB tape drives and lots of tape (70 tapes a day ain't bad). Hire a few minimun wage tape monkeys to change tapes on command. Setup a LED display or a big monitor for the computer to flash tape change commands on. (Old IBM trick)

Mark

Large Scale Storage

[follower]

What your really looking at is some kind of Heirarchical Storage Solution. What happens is that once you have predetermined how much data will be saved from the camera each night. You can get some kind of disk array to store it on. That disk array will also be attached to some kind of HSM solutions such as what is provided by StorageTek's SAMFs. That solution will automatically backup the data that is stored on your disk and remove it from your disk so new data can be stored on the costly disks. From now on your OS and applications think that the data is on disk but in reality its on tape. When the data is requested the software will automatically get it from tape and place it back on the disk. This can be rather costly however.

Store half as much

[Terry Cumming]

Be pragmatic and only archive 15fps. This cuts your archive media costs by ~50% no matter what solution you choose. 15fps should be adequate, although who knows your exact parameters.

Fibre Channel

[Erk]

Go with a FC solution - stay away from EMC, as they will try to sell you a massive Symmetrix for your needs. Sounds like you need a building block approach, one block a day. Doesn't need to be TOO fancy, eh?

Here are some options for FC disk storage:

- Sun T3
- EMC Clariion
- Compaq Storageworks
- HP VA7400 -- my fav

Just to warn you, you're looking at something on the order of 20k/day to operate this setup... now, I'm sure the price would go down QUITE a bit if you're purchasing 8-10TB a day, but even still, it's a huge cost.

I looked at a 10TB solution from the above vendors, and the cheapest I got it was $0.0425/MB!

Celestial Solution

[kc0dby]

and you have to store it indefinitely.

Retrieval, though, can be essentially arbitrarily slow

Oh, so your looking for a storage medium with infinite space but slow retrieval time?

Easy. Free-Space Medium.

Just use an extremely high gain antenna, a ton of power, and the space around us. Transmit the compressed data stream, aimed at a distant planetary body of your choosing. I would reccomend something in the 100 light year range or so. Now, when the waves hit the body and are reflected back to earth, you will have what is essentially a 100 light year long piece of storage.

And when the waves get back to earth, the technology for terrestrial storage will be extremely inexpensive, and the reception equipment will be too.

Real-time 100:1 compression?

[image]

100:1 video compression brings that down to ~90 KB/s.

Very interesting problem, with one more very interesting challenge that hasn't been raised yet:

Because the video is streaming in 24/7, you'd have to build a real-time compression system that could handle the 9MB/s and produce a 100:1 ratio. You could perhaps distribute that across multiple machines/CPUs, or build a custom parallel hardware setup to handle the encoding, but at this scale, the overhead of everything might prevent you from reaching the essential criteria of real-time.

Does anyone know what the hardware requirements are for real-time encoding one 640x480 stream? Now, multiply by 1000.

Re:Real-time 100:1 compression?

[dublin]

Disclosure: I am VP/GM of Conservor, whose product & service offerings are discussed in this post.

At Conservor, we offer a new set of IP-based Storage Area Network services that offer the speed, capacity, and features of traditional Fibre Channel based SANs, but at an order of magnitude less cost. This approach uses Gigabit Ethernet to make disk and/or tape resources appear as locally attached devices at very nearly wire speed.

We have been told by our partners that our IP SAN experience is considerably broader and deeper than even the largest consultants such as EDS, Accenture, and IBM Global Services. We are also experts at storing and managing VERY large datasets - we work routinely with the oilfield service and exploration companies to do exactly this sort of thing with very large 3D seismic datasets. It's not clear from your post, but you may well need some type of content management system, as well, to ensure propoer indexing and speedy retrieval of such a volume of data, and we can do that, too.

Your write speed and capacity requiremeents, while larger than normal, are not a problem, and can be accommodated without having to resort to exotic
technologies. Of course, there aren't enough details to propose even a rough solution from what you've posted, but it sounds like a tape solution. Still, if your retention time is not too long, we might even be able to do it with disk (we can provide high-speed Fibre-Channel storage arrays as little as 2 cents/MB: MUCH LESS than the competition, and capable of RAID 3, which you may want to use if video streaming for playback is much of an issue - we're working with next-gen cable headend guys on this stuff, too.)

As for the tape components, I need much more info to be able to even speculate on your needs. Anything from mid-range high-performance LTO libraries to full-scale mainframe-type 3590 silos may be needed, depending on a number of variables.

All our solutions are also available as complete service offerings, preventing you from having to acquire, own, or maintain any hardware, software, or management staff. In addition, since our fee is entirely for the service, it is a tax-deductible operating expense, which most companies find quite attractive. (We can also make everything look like capital for those companies (Real Estate, etc.) that want to capitalize everything in an effort to boost EBITDA.)

Storage is changing - there's no reason to do things the old way anymore, when there are better and cheaper solutions at hand. Some of the big guys in storage are going to learn this the hard way...

What are they filming?

[KFury]

If things aren't actually moving in most of the shots (ATM or warehouse surveillance cams, for example) then you'll be able to get far better than 100x video compression.

Also, how much a factor is comunication. 1000+ cameras ona LAN or WAN?

Any secondary logging going on here? Any metadata (ATM transactions, notes, etc.) that should be stored along with the media? Do you want to use this data for easier access? Is there any preprocessing (facial recognition)?

You mentioned recall could be arbitrarily slow, but if it's possible to speed it up with only small changes, is it worth it?

Feel free to ignore these questions. Largely I'm just curious about something you probably can't talk about, but then again as a systems engineer, I'd find it difficult to recommend a solution without knowing more factors that could impact on ways I can't think of until I know more factors...

Face Recognition Application

[cwhittenburg]

The guy who submitted this story "Bondheadguy" resolves to RobJMcCready@yahoo.com... A quick search on "Rob McCready" yields a University of Toronto grad student (or maybe former grad student now) who is developing hardware based face recognition equipment. Check out This link...

Now you can make your own decision about helping him out (or not).

Re:Face Recognition Application

[BondHeadGuy]

Actually, I developed (past tense) a hardware-based object detection system. Faces are interesting objects to try and learn to detect, because they have intresting and considerable variation. They also make for a fun demo and get more press. But I could have been detecting anything. Recognition, on the other hand, is different; it is the detection of individual faces rather than faces in general, and it would be nearly impossible to put entirely into a hardware system because of the large face database required.

That system rocked. But it has nothing to do with this post.

Rob

Storage silos...

[supabeast!]

Since you did not state a retrieval time or storage/retention needs, I am going to offer to scenarios; one for long term, fast access storage, one for short term and/or slow access storage.

Storing 8TB/day for a long time with quick access would probably require a tape silo, which is essentially a tape library the size of a small house. StorageTek is one of the leaders in silos (And might be the only vendor making them these days.), and they make some pretty nice stuff. Their PowderHorn 9310 is a nice model for bulk storage and quick recovery. A downside to the silos is that they do not often handle DLT tapes, which can make it hard to use tapes outside of the library.

If you do not need fast access to the data, and have time to root through tapes for restores, just get a smaller tape library (Anything in the 50-100 tape range from ATL/Quantum Adic or Qualtstar running SuperDLT drives controlled by Veritas Netbackup would give you an easy way to handle all the data. NetBackup has excellent archiving capabilites (IE record data, wipe data from disk.), works on just about any platform out there, scales well, and keeps files in GNUTar format for easy access. As for storing the tapes themselves, if you have a small retention time just keep around a few hundred tapes to cycle through. If you need to store the data for a long time, get a few thousand tapes and a set of nice shelves to keep them on. If you do not have somewhere to store them, Iron Mountain does a great job storing data, I have worked with them before and toured one of their facilities, and I can vouch that they do a great job storing data.

Easy

[NMerriam]

This is actually a pretty easy question to answer:

Don't Do It.

This is someone either playing a theoretical game (in which case, the answer is "outsource it") or its someone who has no idea what they really want. You have, ultimately, many conflicting specs here.

You may as well ask for a space shuttle that can fly to pluto in two minutes with no fuel.

Any system that is recording a thousand video inputs is unlikely to need 30 fps for 24/7 (I can't think of anything short of national security installations that would even desire to record 30 fps 24/7, and you'd still have trouble justifying 1000 cameras to cover every building in Washington, DC). Not to mention the logistical implications of DELIVERING 1000 full-frame video feeds to a central location -- you could saturate the entire radio spectrum for the eastern seaboard or have to build the largest gigabit LAN ever deployed.

If you have a real question, please ask it, but this is as bad as a pointy-headed boss spouting off insane specs as the "requirements" for a project because he wants to be on the cutting edge.

And BTW, you won't need 300k per frame for a grayscale 640x480 video image (except that you desire insane specs, which point we've already covered). A fine quality image could be stored in 25-50k, even less depending on the real needs (of which this project seems to lack).

Re:Easy

[NMerriam]

30 fps for 24/7 is what our customer wants. End of discussion.

Hey, I want a lot of things I can't have, either.

Part of your job is to make sure the customer isn't making life harder for himself than it needs to be (at least if you're a good consultant/engineer and not just trying to get the bucks regardless of outcome).

I suspect the end of this particular discussion is going to end up with:

- the customer not getting what they want
or
- the customer spending a hundred million dollars to custom-build a system that in three years will cost ten grand, be available by mail-order, and fit under your desk.

SO, the real answer to your question always has been and still remains to call up someone who has a clue, not slashdot -- we can't spec out custom hardware installations for you. This is not a software problem.

If this really is necessary, then call up a video company and get a VAR in your office to figure out how to build a thousand+ cameras with 100:1 hardware codecs that can transmit video over whatever arbitrary distances to whatever arbitrary equipment you have.

Then get THAT VAR and a storage VAR together and figure out how the hell you can store terabytes a day -- they'll build a nice online/offline disk and tape mixture that will cost enough to fund a third-world country, but it'll work, but you'll probably be able to buy the entire storage company for the budget you'll be spending, so look into building your own company or buying one out in order to save some money.

Then call the contractors to install all the cameras and network links and build a control room with monitoring equipment.

THEN -- figure out how to back up all this data, since its clearly very valuable stuff that you've spent a couple million on already, you don't want to lose it. Have that storage VAR get a system that'll automatically dupe all the tapes before storage for redundant storage.

Who would need such a system?

[TheSHAD0W]

Do you work for that new "Homeland Security" agency??

How About....

[John_Booty]

/dev/null ???

Pros: Extremely high write speed
Cons: Hard to get data back out, but since "Retrieval... can be essentially arbitrarily slow" you've can just re-film whatever it was that you missed. With the money you save on the video gear, you should have a nice little production budget, too...

Consult with a Casino if you can...

[MadCow42]

The Casino industry is probably the most advanced in the business of surveilence... the average Vegas casino probably approaches the scale you're talking about already, however they probably don't archive indefinately.

However, any information I've seen shows them to still be mostly analog capture for any storage, or at least digital-to-analog conversion for storage.

Although they probably won't talk about their security systems, they'd be a great resource.

MadCow.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Analog is the way to go

[CKW]

That'd be a storage nightmare.

I don't think so.

Let's assume one camera per VCR, full 30 fps. That's 3 8-hour tapes per day per camera, 3000 tapes a day from 1000 VCRs. 1000 VCR's should cost you $100,000 and take up one
medium sized room (power and AC will need to be enhanced). 3000 tapes per day shouldn't cost more than $3000, or $1 million per year.

You'll only need a few tape monkeys at any given instant, because they'll be around one tape needing changing every 28 seconds. A days's worth of VCR tapes (assuming we pack them in boxes with NO room to spare and stack the boxes in blocks) will take up about 1.5 cubic meters or 50 cubic feet (based on 1x4x8 inches per tape, my rough estimate). That means for a year's worth of tape you need 550 cubic meters or 20,000 cubic feet, which is 3300 square feet if piled six feet high. 3300 square feet is about the floor-size of one big house.

Question to original: Are you still sure you want to do this? If so you might be best off "spreadking the load around". IE: Don't do it all in one place. There are a million convenience store camera's and vcr systems in the world, but they're not all in one place.

Off-hand I can only think of one thing that would handle 3,000 terrabytes per year, and that's if the half million people using Morpheus donated 6 Gigabytes of space each year to your cause.

Re:For a specialized solution

[sql*kitten]

NASA also thought about this, all the way up to Petabytes.