Large-Scale Video Archiving?

BondHeadGuy asks: "Ok, say you have 1000+ cameras emitting 30 frames/second worth of 640x480 grayscale video...and you have to store it indefinitely. What do you do? This is a real question, believe it or not. 30 frames/s * 300 KB/frame = 9 MB/s per camera. 100:1 video compression brings that down to ~90 KB/s. But 90 KB/s * 1000 cameras = 90 MB/s, or ~8 terabytes/day. Retrieval, though, can be essentially arbitrarily slow. Reliability should be good enough to not be annoying long term. Is there a solution that: has 8 TB/day storage capacity, can handle the 90 MB/s write speed, and lets you save some bucks on the (slow) read side?"

Comments

Analog video systems still work

[jgrider]

Although you are probably looking for a digital solution, don't overlook the solutions that already exist. Security camera VCR's (available at RadioShack et al.) can put 24 hours (or more) of video on a single VHS tape. Get a few VCR's (at $200 each), and a pallet of VHS tapes at Sam's club, and you could record all the video you want!

Re:Analog video systems still work

[rho]

There are specialty security camera/VCR combinations that are built to handle this kind of storage.

If you are willing to sacrifice full 30fps, you can use these. The way VHS is laid down on the tape is the data is written diagonally across the width of the tape. Look at your VCR heads -- see how they're cockeyed? that's intentional.

These specialty camera/VCR combos lay down the data from each camera on successive stripes, and can display 9 different cameras at once on a regular TV screen. You can choose a single camera to fill the screen, but at a reduced framerate.

This will reduce your storage needs by a factor of 9, but at the cost of framerate. This wil not be able to be digitized, unless there are specialty capture cards that can handle this.

From your description, it sounds like you're setting this solution up for a casino, or something very similar. If that's the case, you NEED the full framerate.

You are at the fork in the road -- fast, cheap, reliable: pick any two.

Are they looking at normally still settings?

[RobKow]

Like a security camera in a stairwell or something? In that case you can use motion detection to start/stop recording and save well over 100:1. The choice of video codec is going to be important if it's for security (so faces, etc. can be recognised), but if not, you can crank the compression ratio up quite high on most codecs, especially the video codecs that do frame-by-frame motion differencing (i.e. not MJPEG).

Store half as much

[Terry Cumming]

Be pragmatic and only archive 15fps. This cuts your archive media costs by ~50% no matter what solution you choose. 15fps should be adequate, although who knows your exact parameters.

A (very) large DLT array.

[j.e.hahn]

A suitably large DLT library with a fairly large number of drives would probably do this. Couple it with some HSM (Hierarchical Storage Management software) and you're probably all set.

In terms of sizing, assuming you get 6MB/s per DLT drive, you'll need at least 15 drives. Go for 20. This gives you room to do cutovers, and the like. I'd recommend fronting this with a LARGE disk for scratch space (preferably solid state, but if that's not in the budget, a big old SCSI disk'll do.) You'll need a pretty hefty server to handle all this (at least a pair of Sun E450s for redundancy). You'll also chew through at least 200 tapes a day at a native capacity of 40G/tape.

HOWEVER, this is by no means cheap. The virtue of the fact that you're talking about 8 terabytes a day should be a clue to that. The sort of tape archive, tape supply, and tape library you'll need is... vast. You're talking very high-end hardware here. You'll need a good cataloging system, and some serious software to maintain all this. You'll need to keep about 75% of your drives streaming all day every day. Tape costs alone will run to about 10k/day, let alone electricity, storage, maintenance and initial outlay. I'd venture a project like this is probably a $15 million dollar outlay to do it right, with at least 2 full time support staff and budget on the order of $40k/day . But if you've got the money, go for it.

Real-time 100:1 compression?

[image]

100:1 video compression brings that down to ~90 KB/s.

Very interesting problem, with one more very interesting challenge that hasn't been raised yet:

Because the video is streaming in 24/7, you'd have to build a real-time compression system that could handle the 9MB/s and produce a 100:1 ratio. You could perhaps distribute that across multiple machines/CPUs, or build a custom parallel hardware setup to handle the encoding, but at this scale, the overhead of everything might prevent you from reaching the essential criteria of real-time.

Does anyone know what the hardware requirements are for real-time encoding one 640x480 stream? Now, multiply by 1000.

Storage silos...

[supabeast!]

Since you did not state a retrieval time or storage/retention needs, I am going to offer to scenarios; one for long term, fast access storage, one for short term and/or slow access storage.

Storing 8TB/day for a long time with quick access would probably require a tape silo, which is essentially a tape library the size of a small house. StorageTek is one of the leaders in silos (And might be the only vendor making them these days.), and they make some pretty nice stuff. Their PowderHorn 9310 is a nice model for bulk storage and quick recovery. A downside to the silos is that they do not often handle DLT tapes, which can make it hard to use tapes outside of the library.

If you do not need fast access to the data, and have time to root through tapes for restores, just get a smaller tape library (Anything in the 50-100 tape range from ATL/Quantum Adic or Qualtstar running SuperDLT drives controlled by Veritas Netbackup would give you an easy way to handle all the data. NetBackup has excellent archiving capabilites (IE record data, wipe data from disk.), works on just about any platform out there, scales well, and keeps files in GNUTar format for easy access. As for storing the tapes themselves, if you have a small retention time just keep around a few hundred tapes to cycle through. If you need to store the data for a long time, get a few thousand tapes and a set of nice shelves to keep them on. If you do not have somewhere to store them, Iron Mountain does a great job storing data, I have worked with them before and toured one of their facilities, and I can vouch that they do a great job storing data.

Seriously non trivial

[CharlieG]

I've looked into almost this exact problem (we had about 100 hours of full color video/day - broadcast quality)

Your going to have to get VERY friendly with your local "Storage Area Network" vendors. What we came up with as a best SHORT term solution was this - Store the video on Video tape or DVD (depending on quality requirements - DVD is NOT broadcast quality), and then use multiple players - things like DVD jukeboxes/tape changers. They can either be manually loaded, or a robot. You then use a cache to store the vidio on a last in/last out basis if you need fast playback (assumption here - the most recently used tapes are most likely to be used again)

Encoding isn't that bad a problem - you just use multiple encoding stations - You say you have 1000 cameras - you're probably going to need better than 1000 encoding stations (don't forget spares) - you batch up 1/2 hour (for example) files and write those out to the SAN when your done - while one station is encoding, the next is recording, and you batch the encoded file up into Near line storage, so you don't NEED real time

Storage is going to take space/money BIG MONEY - your talking about 30 DVDs worth of data/day depending on your robots. Figure 1000s/day

Charlie

Casino

[Sixty4Bit]

Obviously this is some sort of security system that watches a large amount of space. So we are talking either a Casino or park of some kind. If not, then these are the people to ask.

Also, is keeping all of the footage forever a requirement? Or just some of the footage? I would think you may want to keep the footage for a couple of days or weeks at most. If something requires footage to be kept longterm then you would move that from the harddrive to cdrom or dvd.

This is a job for a cluster of iMac's if I ever saw one :)

Easy

[NMerriam]

This is actually a pretty easy question to answer:

Don't Do It.

This is someone either playing a theoretical game (in which case, the answer is "outsource it") or its someone who has no idea what they really want. You have, ultimately, many conflicting specs here.

You may as well ask for a space shuttle that can fly to pluto in two minutes with no fuel.

Any system that is recording a thousand video inputs is unlikely to need 30 fps for 24/7 (I can't think of anything short of national security installations that would even desire to record 30 fps 24/7, and you'd still have trouble justifying 1000 cameras to cover every building in Washington, DC). Not to mention the logistical implications of DELIVERING 1000 full-frame video feeds to a central location -- you could saturate the entire radio spectrum for the eastern seaboard or have to build the largest gigabit LAN ever deployed.

If you have a real question, please ask it, but this is as bad as a pointy-headed boss spouting off insane specs as the "requirements" for a project because he wants to be on the cutting edge.

And BTW, you won't need 300k per frame for a grayscale 640x480 video image (except that you desire insane specs, which point we've already covered). A fine quality image could be stored in 25-50k, even less depending on the real needs (of which this project seems to lack).

Re:Easy

[NMerriam]

30 fps for 24/7 is what our customer wants. End of discussion.

Hey, I want a lot of things I can't have, either.

Part of your job is to make sure the customer isn't making life harder for himself than it needs to be (at least if you're a good consultant/engineer and not just trying to get the bucks regardless of outcome).

I suspect the end of this particular discussion is going to end up with:

- the customer not getting what they want
or
- the customer spending a hundred million dollars to custom-build a system that in three years will cost ten grand, be available by mail-order, and fit under your desk.

SO, the real answer to your question always has been and still remains to call up someone who has a clue, not slashdot -- we can't spec out custom hardware installations for you. This is not a software problem.

If this really is necessary, then call up a video company and get a VAR in your office to figure out how to build a thousand+ cameras with 100:1 hardware codecs that can transmit video over whatever arbitrary distances to whatever arbitrary equipment you have.

Then get THAT VAR and a storage VAR together and figure out how the hell you can store terabytes a day -- they'll build a nice online/offline disk and tape mixture that will cost enough to fund a third-world country, but it'll work, but you'll probably be able to buy the entire storage company for the budget you'll be spending, so look into building your own company or buying one out in order to save some money.

Then call the contractors to install all the cameras and network links and build a control room with monitoring equipment.

THEN -- figure out how to back up all this data, since its clearly very valuable stuff that you've spent a couple million on already, you don't want to lose it. Have that storage VAR get a system that'll automatically dupe all the tapes before storage for redundant storage.

Re:Easy

[monkeydo]

30 fps for 24/7 is what our customer wants. End of discussion.

Your customer could have saved themselves your fee by posting this to Slashdot themselves.

Two questions:
1. How did you get this customer if you have no idea how to meet their needs, and
2. What kind of consultant doesn't work with the customer to refine project goals and requirements?

Your customer has a need, 30fps is not a need, and they belive the only way to satisfy it with full framerate video 24x7. If they are wrong it is your job to show them why and how you can satisfy the need without 30fps.

Gigabit ethernet is becoming common in network cores. Unfortunately GigE is still very expensive, and to go a reasonable distance you need costly single mode fiber. Even 100MB goes much farther over fiber than copper, and I can't imagine your 1000 cameras will be very close together. Just running the fiber to 1000 cameras could easily cost you several million.

If this projecet is going to happen it will probably cost more than 10 million so your customer is wither a government agency or someone with really deep pockets. Either way if you want to get good information here you will need to give more information. If you are worried about disclosing too much you should go talk to some people at vendors like IBM and EMC. They do this kind of thing everyday. They can handle the data storage part and you evidently can handle the video part. Talk to the folks at Cisco about putting it all together. Of course when you go to these vendors they will need more information as well. "That's what the customre wants," doesn't cut the mustard when developing specs for cutting edge projects. Sorry.

Re:Easy

[onyxruby]

Good post, interesting dilemma, I would ignore those morons who complain that this is a hardware issue and not a software issue.

Now, I have some idea of what your talking about. Before choosing my current job I was interviewing with Dictaphone as a support tech for their systems. I have also worked with a lot of techs from Vegas (and spent a bit of time working out there). I have also worked in security, including a nice high profile skyscraper that housed a very large banks headquarters. Their system comprised over 600 cameras, with 95% of those pinhole. So I might have some idea of what I'm talking about.

I think several assumptions can be made from your other posts. First, you are going to run this over a Gigabit backplane, probably fibre. Second, the cameras are sending their signal back to a central (or one of several sub-central locations) location. From here you want to archive the data. I am assuming that the problem of getting the digital camera feed from the camera to where the data is processed is already solved. Your probably wanting to convert an existing tape system over to a digital format.

Since this isn't a gov job there are only a few customers it could be. Most skyscrapers don't require this kind of coverage, with a very few notable exceptions. Even so, they typically only archive data to 90 days, 1 year or three years depending on the location being filmed.

It can also be assumed that since you want to record everything, that you don't want motion activated systems and with your frame rate of 30 FPS you are looking for quick subtle movements that would be easily missed with a standard security camera system (filming once every so many seconds). This requirement rules out something like an airport, as they would not need that speed. This leads me to believe you are working for a large casino since they like to keep their data a nice long time (lawsuits etc), and they would need a frame rate of 30 fps for sleight of hand issues and the higher resolution that you listed. Regardless, even if that is the case, there is nothing wrong with submitting an ask slashdot article about your needs.

As for solutions,

Re:Easy

[onyxruby]

Hit submit too soon.

As for solutions, you only have a few practical solutions. Certain high end tape jukebox silos (StorageTek) etc can do this. You could also record to DVD with similiar jukebox silos from Dictaphone. I am guessing that there is going to a centralized processing and collection point.

From there you are probably looking for long term storage at an off site location that is accessable from the central location. This is where you can enter the real problem. Jukebox silos are large, and take up a lot of prime (expensive) real estate that you probably don't want occupied for long term storage. This means you need a way to archive the data but still search it from a centralized location (what happened at door #117 on Sept 23rd 2001). This means that you have to have fibre to an offsite location where additional jukeboxes are stored.

Since the speed that you would need makes telco solutions ungodly expensive you make need to lay your own fibre underground to an offsite storage center. You couldn't get the speed you need through solutions like microwave. This also rules out solutions like iron mountain. Make sure your data backbone for this system is completely seperated from normal building IT traffic. Best yet, put in an air gap for this system from the normal IT system, good for security too.

Assuming you lay your own fibre (or already have it as many large financial institutions do), than you would simply add in additional jukeboxes at the other end of the fibre. I also have to assume that any such fibre would have a redundant connection no less than six feet parallel from the primarly line (Norwest Airlines learned that lesson the hard way when their Sonet ring lost an argument to a backhoe - cutting the redundant fibre line right along with the primary).

Storage media that could suit your needs would include tape and dvd solutions. CDr's would be too impractical with the standards you dictated that must be met. Fibre hard drive solutions are a possibility but would be very expensive now, getting cheaper as hard drive capacities grew (assuming the number of cameras and data collection rates stay the same), so might be more favorable from a long term cost analysis than a short term one. Since you want to keep data "forever" this would weigh in more than normal.

Hope this helps

Consult with a Casino if you can...

[MadCow42]

The Casino industry is probably the most advanced in the business of surveilence... the average Vegas casino probably approaches the scale you're talking about already, however they probably don't archive indefinately.

However, any information I've seen shows them to still be mostly analog capture for any storage, or at least digital-to-analog conversion for storage.

Although they probably won't talk about their security systems, they'd be a great resource.

MadCow.

640x480?

[hatless]

Why 640x480? That's higher resolution than broadcast TV. Do you need that? Broadcast TV is 460x360. Capturing at that resolution will lose you detail, of course, but if it's detail you can lose, your storage requirement just dropped by 40%.

And since you said retrieval can be "arbitrarily" slow, I'd look into using VHS videotapes--even if you store compressed digital on them--as a storage medium. They're slow as hell for rerieval, but the media might be cheap, especially compared to the likes of AIT and such.

The first question is, WHY are you doing this?

[sphealey]

That's an awful lot of data. Why exactly are you doing this? What is the application? Who are you working for? If you are working for/in the United States, does this application meet the requirements of the 4th and 1st Amendments to the Constitution?

Just a few minor questions.

sPh

Doesn't scale up very well

[DickBreath]

Only 1000 cameras?

I mean 1000 cameras is only enough to put one camera into each home in a fairly small community. Most of the solutions I'm seeing posted so far don't scale up very well. What if you need multiple cameras per home? And what to do about large cities? Maybe this should be a seperate Ask Slashdot question?

Analog is the way to go

[CKW]

That'd be a storage nightmare.

I don't think so.

Let's assume one camera per VCR, full 30 fps. That's 3 8-hour tapes per day per camera, 3000 tapes a day from 1000 VCRs. 1000 VCR's should cost you $100,000 and take up one
medium sized room (power and AC will need to be enhanced). 3000 tapes per day shouldn't cost more than $3000, or $1 million per year.

You'll only need a few tape monkeys at any given instant, because they'll be around one tape needing changing every 28 seconds. A days's worth of VCR tapes (assuming we pack them in boxes with NO room to spare and stack the boxes in blocks) will take up about 1.5 cubic meters or 50 cubic feet (based on 1x4x8 inches per tape, my rough estimate). That means for a year's worth of tape you need 550 cubic meters or 20,000 cubic feet, which is 3300 square feet if piled six feet high. 3300 square feet is about the floor-size of one big house.

Question to original: Are you still sure you want to do this? If so you might be best off "spreadking the load around". IE: Don't do it all in one place. There are a million convenience store camera's and vcr systems in the world, but they're not all in one place.

Off-hand I can only think of one thing that would handle 3,000 terrabytes per year, and that's if the half million people using Morpheus donated 6 Gigabytes of space each year to your cause.

Re:Face Recognition Application

[darkonc]

That system rocked. But it has nothing to do with this post.

So then, what IS the system being used for?