Slashdot Mirror
Large-Scale Video Archiving?
BondHeadGuy asks: "Ok, say you have 1000+ cameras emitting 30 frames/second worth of 640x480 grayscale video...and you have to store it indefinitely. What do you do? This is a real question, believe it or not. 30 frames/s * 300 KB/frame = 9 MB/s per camera. 100:1 video compression brings that down to ~90 KB/s. But 90 KB/s * 1000 cameras = 90 MB/s, or ~8 terabytes/day. Retrieval, though, can be essentially arbitrarily slow. Reliability should be good enough to not be annoying long term. Is there a solution that: has 8 TB/day storage capacity, can handle the 90 MB/s write speed, and lets you save some bucks on the (slow) read side?"
Although you are probably looking for a digital solution, don't overlook the solutions that already exist. Security camera VCR's (available at RadioShack et al.) can put 24 hours (or more) of video on a single VHS tape. Get a few VCR's (at $200 each), and a pallet of VHS tapes at Sam's club, and you could record all the video you want!
Well, you could just use time-delay video tape. That's always worked for me. :-)
Perhaps a recordable DVD of some sort, or lots and lots of tapes.
"A coward dies a thousand deaths, the brave but one."
Depends. If you can have a cd or dvd writer for every camera, then what's the problem - just run 'em in parallel, and have lots of filing cabinets full of CDs...
It's called tape. I hear it's been used for years to store video in analog format. It works even better for digital data.
simply get 1000+ computers with as many large SCSI hard drives as possible.
:) ]
[1000+ can easily be 10000+ or 100000000+, but lets be realistic
not for nothing, but is this for either
1. a reality based web-tv show
2. some bizarre web porn thing
3. some actual legitamite venture
4. security issue ?
hope you pull it off.
--donabal
Safety First Day?
Check out www.emc.com , if you need MASS storage, they're who you need to talk to.
Yep, I never spell check.
More incorrect spellings can be found he
Read his question... He assumes 100:1 compression for his totals.
Like a security camera in a stairwell or something? In that case you can use motion detection to start/stop recording and save well over 100:1. The choice of video codec is going to be important if it's for security (so faces, etc. can be recognised), but if not, you can crank the compression ratio up quite high on most codecs, especially the video codecs that do frame-by-frame motion differencing (i.e. not MJPEG).
. 30 frames/s * 300 KB/frame = 9 MB/s per camera. 100:1 video compression brings that down to ~90 KB/s. But 90 KB/s * 1000 cameras = 90 MB/s, or ~8 terabytes/day. Read the post, this was already addressed. This seems so obvious to me, why did _YOU_ even make it a question!?!
300k/sec seems very excessive. You could try converting it all to mpeg4 with a DivX encoder (http://www.divx.com) and that should compress it right down. If you've got sound in there too, strip it out or at least convert to MP3.
You can do all this with a great program called Virtual Dub (http://www186.pair.com/vdub/)
You've got mail. Pattern baldness. - Crow
Read his post. He assumes 100:1 compression for the final result. Even if you improve that an order of magnitude, the storage requirements will still pose a problem.
Gonna be expensive,
How long does the data need to be stored for? Tape is good if indefinate storeage is not a requirement. (Tape degrades fast.. but is reusable)
Terabyte tape libraries are fairly common. Check out any of the major datacenter manufacturers. Sun and HP both have a unit of about 7TB. But you're talking several 100k$ for a fully automated unit.
Cheapest route would be to go back to the dark ages. Buy a bunch of 100GB tape drives and lots of tape (70 tapes a day ain't bad). Hire a few minimun wage tape monkeys to change tapes on command. Setup a LED display or a big monitor for the computer to flash tape change commands on. (Old IBM trick)
Mark
If you are planning to steal all the television in the world, remember you still have to have time to watch it, and for each day you need to dedicate 3 years, and even here on slashdot there can't be that many people with so little social life.
But I fear that the real reason for doing this is to record all the citizens of a city and keep it and all your personal habits indefinitely, and so I suggest that nobody here answers the question - you may find it bites you later.
None are more hopelessly enslaved than those who falsely believe they are free. Johann Wolfgang von Goethe.
For really big clips, you'll need really big HDDs.
One of my friends who visited Intel Research saw this really cool storage solution. It basically had a huge current accessable terabyte storage system. He told me it wasn't a magnetic based storage. At a certain low point in the day, the storage would be archived to a magnetic form. It was used for storage of movies actually. He mentioned that it was optical based. I'm not sure about that though.
Also, NASA and some hospitals achieve the level of image storage that you're looking for. And hospitals need quick retrival. A distributed solution might also do it. Write to harddrive and then backup to tape?
What your really looking at is some kind of Heirarchical Storage Solution. What happens is that once you have predetermined how much data will be saved from the camera each night. You can get some kind of disk array to store it on. That disk array will also be attached to some kind of HSM solutions such as what is provided by StorageTek's SAMFs. That solution will automatically backup the data that is stored on your disk and remove it from your disk so new data can be stored on the costly disks. From now on your OS and applications think that the data is on disk but in reality its on tape. When the data is requested the software will automatically get it from tape and place it back on the disk. This can be rather costly however.
Be pragmatic and only archive 15fps. This cuts your archive media costs by ~50% no matter what solution you choose. 15fps should be adequate, although who knows your exact parameters.
is he trying to create a "pr0n stream archive of the internet"?
The advantage to digital storage is mostly quick retrieval. Since you don't care about that, you can go the older, more cost-effective route:
Videotapes can hold up to a day's worth of video. Since you have 1000 cameras, you can have a split-screen four-camera recording on a single tape. Security companies sell large-scale recording systems like this, and it would cost much less than buying 250 VCRs. Your daily set of tapes would fill a large box that shouldn't be too hard to store.
Ceci n'est pas une sig
only God my friend, and even then, only on unix.
"i was saying gnu-rd"
cp -a *.jpeg
Do you really want to store ALL frames indefinatly?
I assume you are monitoring something (1000 cameras sounds like a whole city...) and you probably only want to keep the interresting stuff.
Take a look at a program like motion, you will still need a lot of pcs to be able to monitor 1000 cameras, but you will be able to drasticly reduce the storage size needed.
Disclaimer: I wrote motion.
Jeroen
Secure messaging: http://quickmsg.vreeken.net/
I saw something on TV the other night about this new securitycam system.
It combined the input of multiple camera's into one image. It was a little bit like those 360 pictures but even bigger. So you had a real-time virtual world on the computer build up by security cams. It was pretty nifty. You could walk through it and all.
The interessting part fot you is that out of several camera-source's, one "file" was created. So you would only have to backup that one file.
Perhaps the best solution may require some creative programing in addition to hardware. If cost is no object you should be able to afford a couple of terabytes of disk storage. Divide this into two areas - one which stores compressed (tar.gz, or bz2) video and one which stores uncompressed video. Also you have want a large ram disk of about 10 GB or so (up to you). I assume you will be looking at recent footage most often. Store this in the ram disk. After it becomes a few hours old, move it (uncompressed) to the first disk area. After it becomes a few days old, compress it and move it the the second disk area.(a heavy duty perl script could take care of this) This solution should take care of poor read times for often used data.
I imagine those camera's record something, but not all the time (typically for security camera's).
So reundant images could be skipped.
Or perhaps you should run a good video compression codec over it, say mpeg.
Should be able to strip down the datapool down to at least a tenth...
Go with a FC solution - stay away from EMC, as they will try to sell you a massive Symmetrix for your needs. Sounds like you need a building block approach, one block a day. Doesn't need to be TOO fancy, eh?
Here are some options for FC disk storage:
- Sun T3
- EMC Clariion
- Compaq Storageworks
- HP VA7400 -- my fav
Just to warn you, you're looking at something on the order of 20k/day to operate this setup... now, I'm sure the price would go down QUITE a bit if you're purchasing 8-10TB a day, but even still, it's a huge cost.
I looked at a 10TB solution from the above vendors, and the cheapest I got it was $0.0425/MB!
The best solution is prolly a tape drive. Something like www.exabyte.com, they list a auto-loader that stores 12TB in the tower (X200), with 324MB/hr transfer rates. That'd leave a lot of tape reloading every day, though. I'm not too sure about other providers in the area.
That's alot of simultaneous pr0n recording...
The Teradata Database is an expensive but workable solution. This is a database warehouse engine tuned for large scale data storage. It supports online retrieval in an OLTP fashion so the retrieve does not have to be a typical warehouse style query. This is a solution from NCR that currently hosts databases in excessive of 100tb.
No I don't work for NCR but I have worked with their database.
The first solution I would look at is how they do this at the casinos in Vegas (or anywhere else). Everytime I see one of those 'On the Inside' shows about casinos on TLC/Discovery Channel, they are always boasting about their video camera capabilities, and their ability to archive everything that happens. Whatever solution you find there, while not always the most cost effective, has definately been tested in an environment on the scale of 100's, if not 1000's of cameras.
sig--we don't need no goddamn sig
Analog: cheap, standardized video standard, although at the volume you have it will be tough to implement and maintain. While I worked in broadcast there was a tape system using a standard Sony VPR video tape recorder that was normally analog. It would hold a couple hours of broadcast quality video on a reel of 2inch tape. Rumor had it that banks were modifying off the shelf VTRs to do digital archival to the tune of about 300TB per tape. In terms of seek time, the tapes shuttle at 60 MILES per hour so seek time to any part of the tape is no more than 2 minutes. Its an elegant, although slightly expensive solution, but if you are already invested in 1000 cameras, then it may be a viable option. Single unit pulling in about 8TB per day could back up a little over 1 month per tape. Just my $.02.
"Quando Omni Flunkus Moritati" -- Red Green
Multiple SDLT or Ultrium tape decks can handle the throughput, but with a couple of dozens of tapes a day the costs are gonna be rather high (still ... with 8 TB a day I guess you expected as much).
As for the original question: (digital) tape libraries are the way to go, I'd say. Maximum write speed depends on the number of drives you have, and if you rarely need read access, you can just buy as many drives as you need to have the necessary write speed. The expensive part is the access robot, and you can probably save there big time because you don't need random access to old tape - a small tape library is sufficient, just take out the old tapes and store them in cabinets.
The illegal we do immediately. The unconstitutional takes a little longer.
--Henry Kissinger
A suitably large DLT library with a fairly large number of drives would probably do this. Couple it with some HSM (Hierarchical Storage Management software) and you're probably all set.
In terms of sizing, assuming you get 6MB/s per DLT drive, you'll need at least 15 drives. Go for 20. This gives you room to do cutovers, and the like. I'd recommend fronting this with a LARGE disk for scratch space (preferably solid state, but if that's not in the budget, a big old SCSI disk'll do.) You'll need a pretty hefty server to handle all this (at least a pair of Sun E450s for redundancy). You'll also chew through at least 200 tapes a day at a native capacity of 40G/tape.
HOWEVER, this is by no means cheap. The virtue of the fact that you're talking about 8 terabytes a day should be a clue to that. The sort of tape archive, tape supply, and tape library you'll need is... vast. You're talking very high-end hardware here. You'll need a good cataloging system, and some serious software to maintain all this. You'll need to keep about 75% of your drives streaming all day every day. Tape costs alone will run to about 10k/day, let alone electricity, storage, maintenance and initial outlay. I'd venture a project like this is probably a $15 million dollar outlay to do it right, with at least 2 full time support staff and budget on the order of $40k/day . But if you've got the money, go for it.
Not just a prick, but stupid, too (the usual case, actually). The original question posits 100:1 compression to 3kB per frame.
Brush up on those reading skills here.
This is either for the Truman Show or a bad case of big brother.
and you have to store it indefinitely.
Retrieval, though, can be essentially arbitrarily slow
Oh, so your looking for a storage medium with infinite space but slow retrieval time?
Easy. Free-Space Medium.
Just use an extremely high gain antenna, a ton of power, and the space around us. Transmit the compressed data stream, aimed at a distant planetary body of your choosing. I would reccomend something in the 100 light year range or so. Now, when the waves hit the body and are reflected back to earth, you will have what is essentially a 100 light year long piece of storage.
And when the waves get back to earth, the technology for terrestrial storage will be extremely inexpensive, and the reception equipment will be too.
I apparently forgot that sig != uptime...
What about Digital VHS? Works like normal VHS, but stores everything digitally. IIRC you can put up to 8 times as much data on a VHS tape as compared to the analog alternative (preserving the normal frame rate).
1) Watch WTC terrorism events and repercussions on TV
2) Realise there'll be big money in the video surveillance market
3) Create video surveillance company, but don't have any products
4) Post an Ask Slashdot for free technical advice
5) Create surveillance product
6) Make serious bank!
shut up man
Still, it's fun to read about
Are we presuming that there will be action most of the time in all 1000+ cameras? If not, then why store the images where there is no action? I'm doing a similar digital surveillance thing, albeit much smaller. (7 cameras, 1 fps, only at night [when there's not much action anyway]) My images all go to jpeg's and I wrote a little C program to throw away all the "similar" images. My algorithm is somewhat conveluded, but more-or-less only keeps the images if they differ by more than a certain amount. I'm sure video compression schemes like mpeg would pretty much take care of this if you're storing to video format. Storing to jpegs has the benefit of being able to easily pick out any time stamp, but I can't watch them like a video.
A simpler solution might be to put a motion detector on each camera and have them only record when there is motion. Using your 100:1 image compression, you estimate 8 terabytes/day. I would expect you could get (warning: pulled from thin air) 1/10th of that by ignoring anything that isn't moving.
But then you have a quandry: was there really no motion at camera #469 at 12:30am, or was something just broken?
Your talking about expanding the bounderys of storage technology. Do you really want to use bleeding edge technology?
I would sugest, investing more cash in a lower bandwith recording message, instead of keeping up with 8 tb a day. That is a more eligant, and more reliable solution.
Are you capturing in digital format? Are you sure that your systems are even capturing at 30fps?
It's unlikely that any digital conversion device(s) would be able to handle the input from 1000+ cameras and then be able to get that data to a central storage location through a local network.... the bandwidth needed for something like that would be incredible (90 MB/s ??), perhaps even requiring it's own seperate gigabit fibre network. Even in a high-end server with several devices connected to it you'd be lucky to capture 20fps.
But if you were able to get it done, storage options would probably consist of some type of RAID array (with a HUGE number of disks to be able to hold 8TB/day).
Storing that much data indefinately would require enormous rooms dedicated to storage devices, which may not be feasible. Storing data for a week or even a month would be a challenge in itself.
Having things in digital format is nice for indexing and fast retrieval, but in this case it may be too costly. Storing data on video tape may not be as fast or convenient, but it's much easier to store 2 twelve-hour tapes per day per camera than it is to set up and maintain 8 Terra-bytes of hard drive space per day.
Read the story, genius, he already thought of that.
Why not do what TV stations and big production companies do to store large quantities of footage: make a big library of digital recordings. I know the restrictions are 640x480x16, but why not go with 720x480x32 instead? Because its digital, you can get lossless recall through firewire in any halfway decent video camera, and backups to hard drives for compression, editing and analysis is pretty easy.
Why set up a high-tech solution for a low-tech problem?
"Look at me, I invented the stove!" -- Ben Franklin
100:1 video compression brings that down to ~90 KB/s.
Very interesting problem, with one more very interesting challenge that hasn't been raised yet:
Because the video is streaming in 24/7, you'd have to build a real-time compression system that could handle the 9MB/s and produce a 100:1 ratio. You could perhaps distribute that across multiple machines/CPUs, or build a custom parallel hardware setup to handle the encoding, but at this scale, the overhead of everything might prevent you from reaching the essential criteria of real-time.
Does anyone know what the hardware requirements are for real-time encoding one 640x480 stream? Now, multiply by 1000.
Now, for data recovery as you said it may be as slow as it gets, something
Throw your stuff to /dev/null and it will be very happy to watch your videos.
You could archive it to CD or DVD with a jukebox. that would probably solve your access time problem, but you'd need alot of them....
FBI/DOD/CIA/SS contract but has yet to implement his promises...
I'm not sure of the brand name, but Drury college in Springfield, MO is utilizing the technology. The software runs on Windows NT and is connected to mass storage devices. Any computer on the network with privaleges can watch or review footage from any time frame, instantly.
The benefit to digital storage is that, in the case of a break in, instead of spending 24 to 36 hours going through the tapes to find the incarcerating frames, the computer can instantly jump to any indexed point, or even better, ONLY record if there's a difference between the frame before it and the one it's recording. So, instead of recording TONS of useless video (especially at night) the cammeras only record motion. So, 8 TB becomes 1 GB. MPEG4 is used.
D&J Automatic Gates in Springfield, MO has installed the technology. Anyone looking to obtain it might call them: 417-725-5215
You might want to check out http://www.yottayotta.com if you need a high capacity solution for archival of 8TB per day.
An EMC solution would be filled up within a short period of time.
Now, last time I went to a bank (the behind the seances(sp) they had was was basicly a hugh 9" CD changer. This backed up all the accounts the bank had, it could hand the data & speed you require, then just have a spreate machine to read 'em off as and when needed.
t ml
If the data you need is not pernament, then get the big feck of tape based systems.
I think (I'll check now) that the system was Sun based (http://www.sun.com/storage/)
Just taking a quick look at there site:
http://www.sun.com/storage/highend/9960/index.h
6.4GB/sec
Total raw capacity per subsystem 37.3 TB
Your'd need a couple of 'em, but the amount of data you want, your going to have to have some big feak off system.
I'm sure if you sent an email to Sun asking what they'd recomend, they would have a better answer than most of the people on this forum.
CS!
Insightful but Overrated Troll
If things aren't actually moving in most of the shots (ATM or warehouse surveillance cams, for example) then you'll be able to get far better than 100x video compression.
Also, how much a factor is comunication. 1000+ cameras ona LAN or WAN?
Any secondary logging going on here? Any metadata (ATM transactions, notes, etc.) that should be stored along with the media? Do you want to use this data for easier access? Is there any preprocessing (facial recognition)?
You mentioned recall could be arbitrarily slow, but if it's possible to speed it up with only small changes, is it worth it?
Feel free to ignore these questions. Largely I'm just curious about something you probably can't talk about, but then again as a systems engineer, I'd find it difficult to recommend a solution without knowing more factors that could impact on ways I can't think of until I know more factors...
Kevin Fox
Cutting your fps will markedly increase the differences between frames, which will hurt your compression, which will mean that you won't get your %50 reduction.
Slashdot 's editors are dickheads
Whatever your solution is, I think you'll also need quite a beowolf cluster of pc's to tell you exactly what you've captured on camera.
The figure you gave is for 8bit greyscale. Wow, 256 different shades of you know what 30 times per second times 1000+ camera... Holy Geez, that's big time. Which means, you're gonna need a bigtime solution. No, managing 1000 computers will not do. Think about it, it'd take 1 dude (or temp? nah gonna happen) per 5 computers. 200 dudes just do shove removable media in each day. Sheesh. 200 * 50k/year salary = $10Million just to pay to poor saps humping to each computer. You need a large scale EMC solution, you'll probably pay at least $50M for starters, at least you won't be hiring 200 temps.
EMC will sell you a system of 1000 100Gig each with 100Gig drives. So you get some better compression, and you buy about 3,650 of these per year.... Hmmm, maybe the $10M payroll doesn't sound so bad.
Judging by your comment that the cameras are black and white, I'm going to take a random guess and assume they are security cameras.
:). The system will even alert you if there is usage at night, or other prespecified hours. The upshot is you can get about 30 days worth of video, replayable on demand, through a system like this.
I unfortunately do not work in Physical Security, but I have managed to visit my buildings' security room. We use a system that stores video from 16 high res cameras to what is essentially a PC with a 30GB hard drive.
When I heard this I figured that if Tivo could store 30 hours from one video source, this would be two hours of video, and quite useless. Guess again. The software uses a pretty smart motion detection algorithm that reduces the cameras to 1 or 2 frames per second when there is no on-screen movement. The sensitivity is arbitrary - watch out for cameras focusing on monitors, the scan line refresh rate will trigger the movement sensor
I'm sorry I do not remember the name of the vendor, but with your kind of system and a thousand cameras I'm sure you can do some of the research yourself. Good luck,
JackAsh
Get a thousand TiVO's. Why settle for AVI quality when you can see your terrorists and burglars in stunning MPEG-2?
Now you can make your own decision about helping him out (or not).
I have a few gig spare at the mo and im connected to the net at least 2-3 hours a day, im sure you could write a Napster file sharing clone to hold the data everywhere.
ERR 411[Max number of witty sigs reached]
I once discussed with some colleagues about the possibility to store data indefinitely by sending them around on the Internet...
:
Freenet would be a good idea : store it "everywhere" but "nowhere"...
Or you may put crypt/uuencode it and have Google cache it for you...
Or post bits in (lots of) public fora...
Or all at once:
because your 8TB data per days just seem to be a lot it is obvious you'd even need 10,000 square inch of these to store one year of your data...
But now, tell us, what is it for ?
Is it to monitor some people ?
Is it legal ?
(Actually do I mind ?)
OK, let's take your problem the opposite way
How much storage can you afford ?
Just decide which of the nb of cameras, pictures resolution, fps, etc. you'd sacrify to fit in what you'll actually get...
Trolling using another account since 2005.
Great job, good pay for the time and on the weekends and night shifts you didnt have to do jack (I had no life anyway so the hours didnt bother me).
... tape monkey's are history.
Even with US minimum wage you are going to quickly get the money back from a couple of autoloaders though (a couple of them in parallel should do fine, dont need the big tape bots)
How smart are the cameras? If they could compress and process the image before it reaches the 'wire', then you'll also dramatically cut down on the network requirements, and the Processor requirements to compress the info at the library.
So, using 15fps instead of 30, and using motion to activate recording, and compressing at the source, you've DRAMATICALLY reduced the storage and traffic requirements for cameras that are looking at normally empty hallways.
It might give you the budget to get higher resolution images at places of interest...
"Draco dormiens nunquam titillandus."
300 KB/frame for 640x480 sounds a bit outrageous to me. Better compression ratios can be achieved relatively easy. Try a custom solution using fractal compresion, processing will be slow on a single processor, but ratios are really good. If you want something with less compression ratio, but easier on the processor, try wavelet-based algorithms. I can store a 480x480 8bpp grayscale image at 5Kb, and the faces depicted in the image are still recogniseable. It takes half a second to process, and it could be improved.
This is being done all the time, though usually not with so many cameras. The way this works in casinos, and other facilities in that you multiplex the video digitally using a mux. You dont get every frame. If an "event" occurs, you switch that camera off to a real-time VCR. The all digital stuff is done primarally by a company out of Durango, CO, (their name escapes me at the moment) and they stream everything real-time to hard drive arrays and back up to tape off-line as I recall. I used to do a lot of Vicon cameras, but there are many, many manufacturers to choose from.
When the going gets weird, the weird turn pro- Dr. Hunter S. Thompson
10 machines with SANArray FFx-2 from Mylex - that's up to 9TB per machine... It writes up to 270MB/s...
;)
Set up a Round Robin DNS system - and write a small application to run on the servers that makes the DNS point to the next machine when it's full...
Put a DVDWriter on each machine and viola
Any technology distinguishable from magic, is insufficiently advanced.
Since you did not state a retrieval time or storage/retention needs, I am going to offer to scenarios; one for long term, fast access storage, one for short term and/or slow access storage.
Storing 8TB/day for a long time with quick access would probably require a tape silo, which is essentially a tape library the size of a small house. StorageTek is one of the leaders in silos (And might be the only vendor making them these days.), and they make some pretty nice stuff. Their PowderHorn 9310 is a nice model for bulk storage and quick recovery. A downside to the silos is that they do not often handle DLT tapes, which can make it hard to use tapes outside of the library.
If you do not need fast access to the data, and have time to root through tapes for restores, just get a smaller tape library (Anything in the 50-100 tape range from ATL/Quantum Adic or Qualtstar running SuperDLT drives controlled by Veritas Netbackup would give you an easy way to handle all the data. NetBackup has excellent archiving capabilites (IE record data, wipe data from disk.), works on just about any platform out there, scales well, and keeps files in GNUTar format for easy access. As for storing the tapes themselves, if you have a small retention time just keep around a few hundred tapes to cycle through. If you need to store the data for a long time, get a few thousand tapes and a set of nice shelves to keep them on. If you do not have somewhere to store them, Iron Mountain does a great job storing data, I have worked with them before and toured one of their facilities, and I can vouch that they do a great job storing data.
hitachi has several very large storage arrays that are very competitive with EMC last i checked. again, that is if you need it to be in digial format and need it to be online.
alex
If retrieval can take arbirarily long you can just not store anything and make retrieval time aproach infinite time. That would to the trick.
I've looked into almost this exact problem (we had about 100 hours of full color video/day - broadcast quality)
Your going to have to get VERY friendly with your local "Storage Area Network" vendors. What we came up with as a best SHORT term solution was this - Store the video on Video tape or DVD (depending on quality requirements - DVD is NOT broadcast quality), and then use multiple players - things like DVD jukeboxes/tape changers. They can either be manually loaded, or a robot. You then use a cache to store the vidio on a last in/last out basis if you need fast playback (assumption here - the most recently used tapes are most likely to be used again)
Encoding isn't that bad a problem - you just use multiple encoding stations - You say you have 1000 cameras - you're probably going to need better than 1000 encoding stations (don't forget spares) - you batch up 1/2 hour (for example) files and write those out to the SAN when your done - while one station is encoding, the next is recording, and you batch the encoded file up into Near line storage, so you don't NEED real time
Storage is going to take space/money BIG MONEY - your talking about 30 DVDs worth of data/day depending on your robots. Figure 1000s/day
Charlie
-- 73 de KG2V For the Children - RKBA! "You are what you do when it counts" - the Masso
Obviously this is some sort of security system that watches a large amount of space. So we are talking either a Casino or park of some kind. If not, then these are the people to ask.
:)
Also, is keeping all of the footage forever a requirement? Or just some of the footage? I would think you may want to keep the footage for a couple of days or weeks at most. If something requires footage to be kept longterm then you would move that from the harddrive to cdrom or dvd.
This is a job for a cluster of iMac's if I ever saw one
This is not the sig you are looking for...
I think by lowering the framerate a tad could save some resources, along with using the motion-sensing technique mentioned...
With 1000 cameras being archived to hard drives, you are going to burn space fast. There is no way that doing this on a single system or even a handful of systems is going to work... I dont know alot of detail, but the computing center I work in now has nothing but digital security cameras, which are being saved onto hard drives, and we dump each weeks data to tape for archiving. Software and hardware from Axis... not sure of the address, but they also make the nifty webcam server that I manage as part of my webmaster job...
You can cut the data rate by using motion detection and only recording when there is something interesting going on. This would make a HUGE difference in most security applications.
"Oh no, not again"
OK, this scenario sounds like a casino.
Don't they still use VHS to record the eye in the sky, and doesn't videotape fit the requirements?
-David.
If the retrieval can be arbitrarily slow, then just transmit all the data out into space. You can view the data as a bitstream extending out from your transmitter. The downside is that you have to travel faster than the speed of light in order to catch up with the first bit you transmitted. But as they say, LOLI (lightspeed out, lightspeed in).
:)
Your needs sounds a lot like those of a Casino. Like someone suggested, check out EMC, this is how they really make their living.
BTW, EMC has a LOT of experience in this field, providing some gobernment agencies with massive storage (some satelites send to Earth several TeraBytes a day of information which needs to be stored for several months). I can assure you that if you're serious about what you want to do that they'll send some smart guy to your location for an in-depth interview with you and to come back later with a specific solution to your needs and a price quote.
This is actually a pretty easy question to answer:
Don't Do It.
This is someone either playing a theoretical game (in which case, the answer is "outsource it") or its someone who has no idea what they really want. You have, ultimately, many conflicting specs here.
You may as well ask for a space shuttle that can fly to pluto in two minutes with no fuel.
Any system that is recording a thousand video inputs is unlikely to need 30 fps for 24/7 (I can't think of anything short of national security installations that would even desire to record 30 fps 24/7, and you'd still have trouble justifying 1000 cameras to cover every building in Washington, DC). Not to mention the logistical implications of DELIVERING 1000 full-frame video feeds to a central location -- you could saturate the entire radio spectrum for the eastern seaboard or have to build the largest gigabit LAN ever deployed.
If you have a real question, please ask it, but this is as bad as a pointy-headed boss spouting off insane specs as the "requirements" for a project because he wants to be on the cutting edge.
And BTW, you won't need 300k per frame for a grayscale 640x480 video image (except that you desire insane specs, which point we've already covered). A fine quality image could be stored in 25-50k, even less depending on the real needs (of which this project seems to lack).
Recursive: Adj. See Recursive.
No problem. Just broadcast the video streams into outer space on 1,000 different channels of TV. When you want to fetch that video, just go the appropriate number of light years away from earth and start watching.
-russ
Don't piss off The Angry Economist
his account or IP has been temporarily disabled. This means that this IP or user account has been moderated
down too much in the last 72 hours. If you think this is unfair, you should contact pater@slashdot.org. If you are
being a troll, now is the time for you to either grow up, or change your IP.
DIAPER TROLL
Do you work for that new "Homeland Security" agency??
So what HSM means is Hiarchial Storage Management. Basically, when file hits a threshold of time, space or whatever, it will take that file and put it to tape. Then, it will replace the original file with a stub of a file that says 'when this file is needed, it's located here!'
Now, for tape storage, I highly recommend going with LTO as a tape format. You might consider doing SCSI LTO tape drives with a Crossroads 4450 connected to Broccade switches to make a SAN as well. By putting it on a SAN, you'll have the ability to spread around your clusters that you'd be putting in. LTO can spool data at about 10-20 MB/sec. Hence, if you get an STK or IBM storage library with LTO, you can fit around 20 tapes in there, and do 200 MB/sec. Plus, LTO has variable speed when writing to it, so it's better than DLT in that regard. Not to mention LTO's 100 Gig native capacity and a better compression ratio than DLT. (2.0 vs 2.2) Then, it's just a matter of cycling tapes through. If you're honestly talking that high amount of data to keep INDEFINATLY, then you might want to look at STK's Powererhorns, which hold around 2000 tapes. Plus, you can always add another wall of Tapes if you're not getting the throughput you're expecting. Or you could look at some of the larger scale robots out there, but they don't support LTO tape format yet.
By doing the EMC SAN solution to an STK powderhorn, you're looking at an enterprise level solution that will support you for years to come. Course, this comes from someone who's a vendor-neutral consultant with experience with similar technology, so your YMMV. `8r)
Let us know how it goes!
Gonzo Granzeau
"Nothing the god of biomechanics wouldn't let you into heaven for.." -Roy Batty
If you do a quick google check for DVD jukebox you should find a whole raft of solutions for multi-terabyte backup solutions. Your probably looking in the $20000 range for 8TB and that's just for the jukebox(es). The media price would be on top of that. All in all though, I think a jukebox is the way for you to go... backing up that amount of data to tape would take absolutely forever and a hard disk solution is rather space consumptive.
Save the whales... Collect the whole set.
/dev/null ???
Pros: Extremely high write speed
Cons: Hard to get data back out, but since "Retrieval... can be essentially arbitrarily slow" you've can just re-film whatever it was that you missed. With the money you save on the video gear, you should have a nice little production budget, too...
OtakuBooty.com: Smart, funny, sexy nerds.
The Casino industry is probably the most advanced in the business of surveilence... the average Vegas casino probably approaches the scale you're talking about already, however they probably don't archive indefinately.
However, any information I've seen shows them to still be mostly analog capture for any storage, or at least digital-to-analog conversion for storage.
Although they probably won't talk about their security systems, they'd be a great resource.
MadCow.
I used to have a sig, but I set it free and it never came back.
Then let the world archive your data...
With multiplexing your video, you can store up to 32 images on one screen and due to multiplexing, you can get each screen at full size.
check http://www.cctvsupplier.com for more info.
Here is a direct link to the color multiplexers
http://www.cctvsupplier.com/colmul.html
and here is the B&W multiplexers
http://www.cctvsupplier.com/bwmultiplexers.html
Pelco has the higher end product and is very effective as it is used in Casinos to give high quality recordings.
you can check them out at
http://www.pelco.com
I have seen some IP oriented versions, however, their quality is not the greatest.
Here is a link to an IP Based Multiplexer with a real live demo.
http://www.atvideo.com/vipmux_demo_page.htm
*Headline News* censorship shuts down the Internet! More at 6PM!
The guy who wrote the post, is working on face-recognition technology. (thanks for researching him, cwhittenburg)
This exactly the type of thing that could be done, but shouldn't. Stay away from this, we can only hope that any competent people will stay away from this and they will never get it to work.
Sleep with one eye open.......
What you need are some scribes from the Roman Times. Just leave a few of them around town,
and have them write down everything that happens.
costs only sheckels a day.
Why 640x480? That's higher resolution than broadcast TV. Do you need that? Broadcast TV is 460x360. Capturing at that resolution will lose you detail, of course, but if it's detail you can lose, your storage requirement just dropped by 40%.
And since you said retrieval can be "arbitrarily" slow, I'd look into using VHS videotapes--even if you store compressed digital on them--as a storage medium. They're slow as hell for rerieval, but the media might be cheap, especially compared to the likes of AIT and such.
Spruce Technologies, the makers of the also-ran Spruce DVD maestro platform, that's attempting to compete, (unsuccessfully) with Sonic, and Daiken (which is now owned by Sonic, make a solution for just such occasions. Included with their Maestro flagship product, but also available sperately is a straight DVD encoding platform that takes any ntsc compatible signal and encodes it by a bit-rate that you set according to how much time you need for each disc. A windows box with the encoder and then a Pioneer A-03 burner would come to about 3-4 grand. Then, the discs can be found in bulk for as little as $7 a disc. Might be worth a shot. You can use a video distro amp to hook them up to a the primary viewign source and the Recorder at the same time, or a switcher to hook all cameras up to the recorder.
Of course Apple just bought Spruce so say goodbye to windows firendliness shortly.
http://www.sprucetechnologies.com/
Andonyx www.andonyx.com
They didn't have one system that could do it, but they did have huge combinations of those systems which could probably do what you are talking about. Do all 1000 cameras need to go to the same piece of hardware? Look at switches. Most companies used a few smaller ones instead of one huge switch. Of course, it was a while back that I did this research so if anything has come out recently, I don't know about it. Hope that helps
I don't have time to make a sig
That's an awful lot of data. Why exactly are you doing this? What is the application? Who are you working for? If you are working for/in the United States, does this application meet the requirements of the 4th and 1st Amendments to the Constitution?
Just a few minor questions.
sPh
Still compression might be the way to go, except not compression of the single frames, but some way to only store each x'th frame, and between such a store, and the next xth, only store differences, and again compress the entire stream (stored frames +differences).
The efficiency of the differences storing should be improved by a preprocessing (to compression) step to try to reduce small variations in color,
iow to make two surfaces match
in color in the digitized picture, if they are equal in color
If one, while designing the difference-detection
algorithm, is able to differ between background and foreground, one could try to further increase
compression, while maintaining quality by using lossy compression for the
background only, while keeping the foreground (e.g. faces, important with security cams) sharp.
Since some security camera's send home nearly static, or a set of static images recurring after
a certain time (moving cams), this should increase compression.
I can't imagine that something like this is not already available, e.g. as a sideproduct of all
the research that went into the DVD/DivX/MPEG
standards.
We have about 16TB in the Data Center here and we have a combination of eq to take care of the data - Optical, Disk and Tape.
If I was charged with this project - I would look into The OTG DiskExtender software. The software is a bit involved - but once its working its great. It basically creates a cache out of hard drives and writes it off to tape or optical for archival/storage purposes whenever you tell it to. It can also make copies of the platters once they are full and the copies can be removed for storage and backup.
I would look into EMC - or Hitachi (just as good and not as costly as the EMC gear) - or MTI (runs good - not as fast or as much $ as the other 2) for disk space and get a storage box that will handle at least 2 times the amount of storage that you will need in a day - then buy either several Qualstar 412600 (http://www.qualstar.com/146073.htm) with AIT3 drives or several big Plasmon Magento optical boxes like the G638 (http://www.plasmon.com/mo/g5.html) and use those for archive and retrival (if it's been purged from the magnetic) for the system. We use a lot of optical because of the stability and life of the disks and use the Qualstar stuff to to mostly backups with - because they are ROCK solid tape libraries. One more thing - we use the HP opticals also, and they are very nice, but the biggest they have is the 2200mx and is only 2.2TB and they are a bit more money than the Plasmon boxes.
You could go with tape as the backend - but it will be pretty slow - Optical is not that much more $ but works very well. BUT you will need to have enought magnetic to front all of this to make it usable.
We have custom built a viewer for all this - and you might have to also?? But this might be a bit off - we deal with millions of "images" here - not video, so if I am misleading with any of this - sorry. But one thing is true - don't get sucked into a one vendor solution - not going to happen with something like this....
It would be pretty neat to set this up and get it running - want some help? :) Oh and this is going to be BIG BUCKS - no matter what you do.
Duke :)
(please note - I dont work for any of these guys - I just "live" with them a lot
FreeBSD: Nothing runs like a daemon with a pitch fork.
Cut the framerate: If it's just for security purposes, you can safely cut the framerate to (less than)10 FPS and be fine. Remember, US cartoon animation runs at 24fps and average anime runs at 12-15fps. As long as someone can't get in and back out of the camera range between the frames, it's doing it's job. Some security cameras run at 2-3fps and make a "Slide Show", and it works pretty well.
motion sensors: only turn the cameras on when you have to.
DVT: digital video tape. I dunno if this is still around, but back when DAT was attempting to make a go of it, some high-end users (like TV studios) used a version of DAT for video. It was a three-inch-wide tape that looked like a CD (all silver/rainbow hues) and had >4X the storage per inch of VHS. Like I said - dunno if these things are still around, or how much they'll chew on your wallet.
Human review: Actually LOOK at the data before it's archived, and see if you have to archive it. (depends on the application) If you're only archiving "incidents" and not required to archive the "lack of incidents" then this works WONDERS on storage needs.
Most of the other stuff I can think of entail some pretty heavy software design..
such as:
Variable compression: compress alot when nothing's happening - a little when there's action.
Variable Framerate: same thing, just with the framerate. (just in case the motion sensors miss a ninja inching his way REAAAAL slow across the floor or something)
While the ultimate point may be to store this data indefinitely off-line or near line, the immediate question I've got is - How is this being recorded to begin with? Analog cams? Proprietary devices? Telepathy?
Important question to identify, is how much LOCAL storage or buffer space you've got available prior to getting all this info shoved out to the storage farm.
Not knowing the storage requirements, I'd envision something like: Individual day's captures broken into blocks [hour, 4 hour, 12 hour . . something smaller than the entire day] & stored locally on R0/1 set. Data moved live through the day to disk farms after 24 hours for short-term near-line storage . . . R5 sets via FCAL. Data moved to long-term off-line storage [DLT or similar magnetic, or CDROM/DVD-ROM or similar optical storage] . . . again, FCAL SAN attached libraries.
There's only a finite number of vendors that I've seen commercially do this in the type of scale that you're talking about . . . Veritas comes to the top of the list.
I don't think that the magic will be with building a single huge disk farm for this stuff . . . several smaller farms, geographically separated based on where this data is coming from. The tricky part will be managing the data & movement from device to device.
Other issue that you may run into, is the long-term storage of data . . . DLT has a shelf-life of 10 years from what I remember. This may or may not fit your need . . . and may force you to go Optical for long-term storage [which, may not be a problem realizing that you only need write-once ability . . anyway].
As far as reducing the overall amount of data . . . why 30 FPS? Human eye/brain only realistically sees 24 FPS, which doesn't necessarily mean much if you've got security to deal with . . but helps with picking an arbitrary number for capture rates. Oh yeah . . . and it's an instant 20% reduction of data . . from a total of 9TB/Day to 7TB/Day.
BTW . . . Got diesel?
They have put together some nice storage systems for broadcast video houses. Here are some specs on their high end DST 914 tape auto loader.
You are still going to need something to multiplex and compress all those streams, but it sounds like you may have some other leads on that.
Like we're going to tell you Mr. NSA/CIA/FBI/Big Brother. :)
Need Free Juniper/NetScreen Support? JuniperForum
I beleive the answer would lie in the field of quantum computing. Instead of having to worry about storing all the data in our universe, you can store the data in thousands of other universes, and put the bandwidth constraints on those universes, not just our own. This is probably a few hundred years off, but it may provide the solution.
One thing that most posts are overlooking is that 8tb per day isn't the real hurdle -- you can get storage media with that capacity. Handling a 90Mb/s river of data is the hard part. One field where data streams at this sort of rate is in particle and nuclear physics experiments. ... I've been there and done that, got the PhD ... silos are good. Depending on data rates, part of the staging process is to throw out much of the "background" as practical/safe. In summary: this is whopping pile of data, compress, compress, compress and buy good tape silos.
(See www.cern.ch, www.fnal.gov www.jlab.org, www.halld.org as a few examples.) The solution most people settle on is to buffer everything on disk and stage to tape silos -- big tape silos -- you don't want to be screwing around chaging tapes by hand, using consumer-level tape drives or managing an army of 7-cart DLT stackers
Try this
You could use a tape-raid solution. :o)
/.Mattsson - My native language is not English, so please don't whine over linguistic errors. (That's lame anyway...)
Only 1000 cameras?
I mean 1000 cameras is only enough to put one camera into each home in a fairly small community. Most of the solutions I'm seeing posted so far don't scale up very well. What if you need multiple cameras per home? And what to do about large cities? Maybe this should be a seperate Ask Slashdot question?
I'll see your senator, and I'll raise you two judges.
This sounds like a goverment contract so we'll assume you have infinite resources :-) (If not, just ask congress for more money :-P)
BTW, I'm a Systems Engineer, but I'm not completely sure about how your actually capuring the data. But we'll ignore that question, we'll just focus on the archiving part.
Well as for the 9MB/s, gigE can handle that over copper with no problem.
As for the storage, you can go a couple of routes. I personally dislike EMC, I like MTI and XioTech, cheaper and just as fast, plus expandable. Plus MTI's got solid state disk called the V-Cache (I think) that's got crazy bandwidth so you could stream the raw data into that thing, then have a couple of fast boxes connected to the v-cache that could compress it into some sane format like MPEG. I like DIVX:-) but its not exactly a standard that you know will be around in 10 years, MPEG is sure to be around. (even though DIVX is a bit better.) After you've compressed it down, you could them move it over to convential storage FC disks. Then you could have a couple nice big fibre attached tape or MO libraries to archive the stuff. I'd say go with Super DLT or HP's new crazy tape solution. Now also, we haven't approached the idea of organization. Just because you can now reliablly record 1000 cameras worth of junk, doesn't mean that you'll be able to find what the hell your looking for. You'll need some sorta large scale database that contains info such as date/camera/tape. So that when someone comes along and says "I need to see video for the east hallway for the night of June 22" you just have to do a quick search and it'll come back with "Tape 32" it would also be nice if they could then pop in a piece of removeable solid state storage and it owuld get the tape data and just put it on the device for them.
As for what that would all cost, I can only guess it would be utterly insane.
(((640 *480) * 24bpp) * 30FPS) * 1000 =
Roughly 2.21184e+11 bits per second, uncompressed.
Hmmm.
Why not just erect a satellite dish and stream the data into space? The cost of having an alien intelligence beam it back to you would probably be less than conventional storage costs.
I would especially favor this plan if it's some kind of face-recognition software or other privacy-deprivation program.
Another plan would be to hire 1000 people to sit and swap tapes on their home VCRs every few hours - just hire the people who currently plan to make a living stuffing envelopes at home.
Yet, something tells me that you're not doing anything that I would want to encourage...
Cheers,
Jim, hopefully out of your Jurisdiction
-- My Weblog.
What you described there is nothing else than the core concepts behind pretty much any recent video codec, such as MPEG (even MPEG 1 did that), Real and Sorenson Quicktime, with varying
degrees of emphasis and sophistication.
The illegal we do immediately. The unconstitutional takes a little longer.
--Henry Kissinger
..he's working for *cough* the man *cough*? Since so many of you are asking what it's for... ;-P
Talk to some folks at Fermi Lab about how to
handle REALLY big data streams. Last I
heard they produce about 250GB/s of data
when the beam is on ( after the first stage
hardware filtering has reduced the data set ).
These are the folks to talk to if you have any
really heavy duty data needs.
You may need the input from 1000+ cameras and 90MB/sec storage speed, but that doesn't mean that everything has to flow through just the one final system.
Why not instead split it into many mostly independent segments? If you have, say, 10 machines running 100+ cameras each, then you've instantly reduced the data rates by 10 times - 9MB/sec storage speed. And why not have each segment with its own tape rack backup system? You can get somebody to go around all the racks and collect the tapes at the end of the day. Much cheaper than a monster tape silo - and you can repair/replace a broken one without taking out every single camera.
The issue doesn't seem to be the viability of recording the input from 1000+ cameras, it's really about plowing all that data through one single device. My (admittedly amateur) advice would be: don't. You may even see an increase in the overall system's uptime if you nicely segment everything anyway. After all, a single point through which all data flows means that you have a single point through which everything can fail.
Careful...I've used NetApp boxen, and (I'm assuming you're talking about NAS) they just don't have the sand to handle that kind of load reliably. Granted those that I worked with are now ~18 months old, but we had an Oracle update stream hitting one with a bandwidth of less than 15 MB/s and it started dropping NFS connections. If you try this, be careful to use high speed connections and to multiplex onto as many interfaces as possible!
A hero is someone who knows when to run away. I am a hero. -Trent the Uncatchable
You need to reduce as early as possible as fast as possible. Choose the minimum resolution/bit depth/frames you can live with. Find a compression device (i.e. mpeg2/4) that can handle n cameras realtime and drop res/bits/frames to your specs. These will be expensive, but you can probably make up for it by reducing your storage and network requirements. Funnel n cameras into the devices using analog cabling, then spit the resulting digital stream out a network to m archiving stations with lots of high bandwidth storage to use as buffer. Of course, you'll need spares for each of these devices.
Hire your tape monkeys to run tapes and/or get a disk silo. I suspect you could get away with much less storage than you think by compressing early.
Even better would be motion detection on the compression or camera side so that only the cameras that are sensing movement would be working.
As for your network, you'll need to partition it to meet your bandwidth requirements.
Good luck!
1. Get 3x the amount of disk space required (so 3 buckets of 8TB).
2. Record to one section of 8TB for a day.
3. Switch the data streams over to the next section of 8TB when day 2 starts.
4. Use any standard backup tool to backup day 1 to a library with a bunch of fast tape drives (Several LTO, Exabyte M2, STK 9940, Ampex or Sony DST's).
5. Swap the two volumes after day 2 and start backing up the second day.
You need the 3rd disk set to restore segments to. It can be smaller than 8TB, depending on how much is restored. A SAN comes in handy to allow quick switching of volumes, and to allow multiple systems access to the same disk space.
Even 8TB of disk is relatively cheap (look at Eurologic for instance), the big killer is tape media...
Rob
To Terminate, or not to Terminate, that's the question - SCSIROB
charge it, daisy chained on an iMac. that's what I call a digital hub !
When will I end this grieving ? When will my future begin ?
I find it hard to believe that any company would say, "Hey Herbert, we need $10 million per year of storage. Why don't you go check Slashdot and see what they think?"
This would be the kind of project that most consultants have wet dreams about.
i have a similar problem on a smaller scale. In our house we had some problems with arson (6 times in two years). So installed two WebCams and connected them with my LAN. I wrote a small perl- script on my linux server which gets an image every second and writes it onto the hard disk.
This was quickly done, but even a 30GB hard disk holds only the data of the two cams for two or three days before it is full (JPG 640*480, medium compression). Single images are also very difficult to handle when you're looking for some event.
Encoding it as a MPG reduced the volume significantly but my PI-200 needed more than 6 hours to compress 3600 images to one MPG file.
So i'm looking for some solution. The images are colored right now, but i wouldn't mind to have them greyscaled.
Thanks in advance, Martin
P.S. Privacy disclaimer: all people living in the house were informed and wellcomed them. Up to now, no mom/dad asked me to check when the kids came home at night. All images are deleted automagically after 48h.
P.P.S. No arson since then, but one case of burglary (cam pics where no help because i got informed too late and the images were allready discarded).
Acquisition, Compression, temporary storage.
I don't know what is your source ? Video Camera ? then you need a bunch of computer that will acquire and compress in real time the data received. I'll probably would go something like DIVX with some of the lowest quality setting it is doable to achieve that in real time with a machine that is powerful enought. Althought you will need a LOT of them. But it is doable. Then, in each machine you put something like 4 100Gb Harddrive, which will give you 300Gb of storage if you raid 5 it. According to your specs, you'll need about 90Kb/s, about 77Gb/day. You use that as a temporary storage. With this config, you dont need a gigabit card on each machine, a 100Mbs would be enought (transfer of about 36Gb/hour at max speed). Let's Every day you create a separate directory on each machine, that would make it easier, and you split the files in hourly files in these directories. That would make 24 files/directory, much easier to backup that way.
Backup
you could have something like 1 Quantum ATL P6000 Robotic Ultrium LTO Tape (100MB, uncompressed per tape, $100 normal price per tape, much cheaper in bulk) (http://www.ultriumlto.com/atlp6000.htm), you could have about 32drives, 652 cartdrige, meaning a capacity of about 65TB uncompressed, good for about 10days of backup according to your need. According to your specs, you would need about 70 tapes a day.
It is a fiber library so you need to buy a Fiber switch, something like a Brocade switch, silkworm 3800. (You can put 6 drives per fiber on a P6000), so you need 6 ports. Then you buy several SUN servers that will be use for the backup, something like 4500, 10CPU/10gb of ram, a bunch of gigabit cards inside, plus fiber that will be connected to the Brocade to control drives on the library (6 drives per fiber). You'll need 6 4500s to access the 32 drives of the P6000, using 6 ports on the switch.
You install Veritas Netbackup Server on these babies, the client on each of the PC that would be used for acquisition. Then you have a centralized management server, and you tell each SUN machine to continously backup each invidual pc for the previous day.
You are done. of course it is a management headache. lot of machines to backup, 70 tapes needed a day, but it is doable.
Cost would be a lot. 1000 machines x $2000/machine (need storage, powerful CPUs) -> $2M Library, I don't remember the cost for a P6000 fully loaded with Altrium drives, but probably around $500K.
Then you need 6 SUN 4500s. About $200K list price, you can have a good discount, easely 30%. Netbackup, licenses for 1000 machines, 6 servers. that will be VERY expensive., about $120.000 for licenses for 1000 clients.
You need also the network. Brocade switch, fiber equipment, network switches with 1000 ports, I'll advise something like Summit 48, 48 machines per switches, with gigabit uplinks to several master switches, like extreme switches. 22 Switches, about $5000 each If my memory is right, about $110K then. I don't remember how much is a extreme, but you'll need 2 of them for bandwith.
I would say your total cost would be about $3.5M for a project like that, but you can probably negociate a lot and have it down. But this is going to be costly. Also the daily cost would be 70 tapes x $100, $7000 day, $210K/month.
I hope you are planning to make lot of money with this system !
Of course, this is not complete, I probably got lot of prices wrong, I think overall I might not be too far. I know I forgot lot of things, but this is just a general overview of a solution I would design if I had that money. I only used your specs, I am pretty sure compression could be better, 15 images/seconds, etc. and you could reduce ongoing storage cost by half or more.
with time delay you dont get full frames.
heres a solution :
3 Breece Hill (or SpectraLogic OEMed) 2.75TB AIT-2 autoloaders with 2 sony ait-2 drives (66Mb/s throughput per autoloader) each - $30,000 each x 3 = $90,000
3 x Sun Ultra 2's @ $2000 each = $6,000 (yep they can do 66MB/sec and max out the breece hills and they can handle the compression/video..total throughput will be 66MB/sec x 3 = 198MB/s which should be more than enough to handle the video feed.)
AMANDA tape backup software (free download) and Sun solaris + a shell script.
30 AIT-2 50GB catridges (about the size of a cigarette pack) x 3 = 90 catridges (should fit in a small briefcase) per day for 2.75 x 3 = 8.25TB/day @ $3000/day
So..it can be done for $100,000 and running costs at $3000/day (plus 1 guy 10 minutes to swap the catridges from the autoloaders and cycle em at the end of the day)...which is fairly cheap for this setup. A small standalone sony AIT-2 drive in a PC can read back the tar files of the video if necessary. One large room should be able to hold all the tapes for a couple of months or so. still at $90,000 its not exactly cheap.
No matter how you look at it and what media you use, if you plan on storing this stuff indefinitely you'll have to go for a large (think medium - large shed size) jukebox. Even if you store it on, say, an EMC solution, you'll have to take old data offline at some point for cost and efficiency reasons.
.1TB IDE Hard drives, which would take up less space, and you don't have to worry so much about hardware failure since the drive electronics are switched daily.
That being said, remember that a run-of-the mill 16/10/40 CD-Recorder records at up to 140MB/s, which is well over your goal of 90MB/s. High-capacity CD-Rs can go to 800MB, so you'll end up writing 10,000 per day, and you'll want to use about 100 or more cd-rom drives to do the recording (for reasons of robustness, and to lengthen the average MTBF).
BUT it sets you up for an easy swap in later to switch to DVD-R, or the newest 100GB disc (vapor-hardware) technology, though you can't go much cheaper than CD-R, along with storage reliability of 5 years or so in decent circumstances, 10+ years in climate controlled lockers.
I'd imagine making CD-R cassettes that hold 10,000+ CD-Rs which simply get changed once a day (remember - 100 spindles of 100 CD-Rs is not much space, and can easily be handled by standard warehouse automation equipment). As a bonus, you can store them by day. Need a few hours on Oct 29? Just pull out one cassette. You'd have to trade out a CD-Recorder daily as well, but overall you're going to spend a LOT less than any other storage solution out there.
You can't get much cheaper than this, and the tradoffs may well be worth it to you. But it would likely be a fully custom which would take time to design. It might compare favorably to a cassette of 100
-Adam
I'd recommend finding a solution that works for 200 or so video stations, then split up your cameras/back end systems into chunks of that size.
So find a 200 video solution, and duplicate it five times, situating each site in a geographically convienient manner. A bit of coding glue on the back end might let you have a centralized index.
A single mechanism to handle that much traffic is going to run into many problems and probably won't be satisfactory. bandidth, storage, encoding time, etc for a smaller number of cameras is far more tractable for a smaller number of machines.
This also gives you a way to scale. What happens six months later, when now you have 1300 cameras to deal with? add another site. With a good template, your costs become very predictable, you have a built in mechanism for pilot testing (you will learn ALOT from building the first site), and your project time to scale up becomes easier to estimate.
The original question is basically meaningless without a description of the real project requirements.
You've put some big numbers up there, which all the hardware-heads have been happy to answer for you. But without real information, this is all just big-clock-speed hardware masturbation.
The real question is: what kind of single system/application would produce 24,000 hours of unedited high-quality video per day and storing it until the end of time?
Most respondents seem to assume that you're running a network of security cameras. If so, other posters have indicated that your quality or recording time requirements are probably 1-2 orders of magnitude too high.
If you're producing something where this video is actually going to be watched by people who care about beautiful full-color full-frame-rate production quality, where is your 200 million dollar production staff that will be watching and cataloging this data? And if you can afford that, surely you can afford at least one knowledgable systems engineer who knows how to design a storage system!
My absolute favorite bit of your post: "Reliability should be good enough to not be annoying long term". So how much lossage is "annoying"? Will you save space by randomly culling videos from the previous 7 days? If the whole system breaks down once a month, will that be annoying? If you lose one minute out of each hour, will that be annoying?
This is the sort of problem that can be designed and solved if you've got the need and budget for it. But it's not a turnkey solution. That's why you hire engineers.
Place an hostile offer for all the WD stocks, and you will have cheap disks to spare...
NEOCA - Custom LED Flashlights
Comment removed based on user account deletion
In my datacenter I have a few ADIC Scalar 1000
libraries that house 1000 AIT-2 tapes each.
please tell me where I can buy 100GB server-speed
fiber or SCSI drives for $75.
oh, an can I get those in 4" x 2.5" x 0.5" form factor please??
A year spent in artificial intelligence is enough to make one believe in God.
Using FPGA hardware, Rob McCready, the original poster achieved around an 88% face detection rate in real-time from black and white pictures with a 30 frames per second video camera.h tm
This explains his crazy requirements : looks like he is busy thinking about large scale applications of his work.
http://www.nce.gc.ca/en/success/9920/micronet3_e.
That'd be a storage nightmare.
I don't think so.
Let's assume one camera per VCR, full 30 fps. That's 3 8-hour tapes per day per camera, 3000 tapes a day from 1000 VCRs. 1000 VCR's should cost you $100,000 and take up one
medium sized room (power and AC will need to be enhanced). 3000 tapes per day shouldn't cost more than $3000, or $1 million per year.
You'll only need a few tape monkeys at any given instant, because they'll be around one tape needing changing every 28 seconds. A days's worth of VCR tapes (assuming we pack them in boxes with NO room to spare and stack the boxes in blocks) will take up about 1.5 cubic meters or 50 cubic feet (based on 1x4x8 inches per tape, my rough estimate). That means for a year's worth of tape you need 550 cubic meters or 20,000 cubic feet, which is 3300 square feet if piled six feet high. 3300 square feet is about the floor-size of one big house.
Question to original: Are you still sure you want to do this? If so you might be best off "spreadking the load around". IE: Don't do it all in one place. There are a million convenience store camera's and vcr systems in the world, but they're not all in one place.
Off-hand I can only think of one thing that would handle 3,000 terrabytes per year, and that's if the half million people using Morpheus donated 6 Gigabytes of space each year to your cause.
Our company sells video transaction auditing systems to large retailers, recording all transactions and video in the store, correlating it all in a large database system. We use our own network video appliances that compress the video streams as still images using wavelet-based compression. Image rates are based on the context of the images in the environment - alarms, check cashing, cash drawer open, etc. The compressed images are then stored in a distributed database across multiple servers as necessary, with an aggregate viewpoint for recall. We don't back up to tape in general, because of the inherent problems with nearline storage, and the ever decreasing cost of HD space. Since we have context for all the images, the real value is in the recorded image , and what it's related to.. So you store only what you need to at high frame rates / quality, long term.. Anyway.. email me if you'd like me to expound on this for a few hours.. 8)
Do as linux torvalds, upload all that data somewhere and have everybody mirror it.
:)
Cliff, interesting problem; but I have a solution.
The company I work for, Software Engineering Corporation (www.sencorsoft.com) makes storage software. I won't get into too much detail about it (you can check our website for more information), but I'll just say that it scales well.
One of the finer points of the system is that you can distribute the storage and still query any single point for data stored in any other point. To achieve the throughput you want (and the long-term storage capacity), I would recommend building several systems with a RAID for online storage (enough to cache storage and retrieval operations, would depend on how long each segment of video is) and a tape library for nearline storage (or dvd if you prefer, but for this type of system, I would recommend tape).
Depending on where you wanted to go from there, you could either shelve the media (take the media offline) or purchase additional library units (keep everything nearline). Since the retrieval time is not important for you, and the costs of the latter would be incredible, the former would probably be more palatable.
If you want more information, let me know (either post a reply or send me an email; my email name is gino.canessa, the domain is sencorsoft.com).
Some tips in general also...
1) RAID
--- Using large amounts of RAID is both extremely expensive and unncessary. Since you do not need instant access to everything, I wouldn't even consider it (also, assuming you keep everything for a few months, it would be the largest amount of RAID ever sold by any company =).
2) HSM limitations
--- Most HSM's need a single point of reference. Given that you will be using multiple units for encoding, it would probably not make sense to reduce that to a single point (bottleneck), so you would want to have your storage distributed. In traditional HSM's, each storage system is independant of the others, so when you wanted to retrieve something, you would have to know where it was instead of the archive finding it for you. Also, many HSM's run into problems when you start leaving the terabyte range of data (e.g., petabytes of data).
3) Hardware choice
--- Several other people mentioned this, and I feel it is an important point as well. Many software solutions only work with specific media types or harware vendors. The last thing you want is to be vendor or media-locked a year from now when the next big thing in storage comes along and your chosen vendor doesn't support it.
Hope this helps...
WWW.NETWORK.COM
They make SANS (storage area networks) they are fast, they store 100's of terabytes, lots of government facilities use this. Check them out, and I am sure they are affordable since you are utilizing over 1000 cameras.
Mike H
Okay, mod me as redundant, but I'd like to know what the possible real world application is...
First off, there are commercial solutions that tape 10:1 onto VHS tape... maybe you could take one of these a step further (custom ASIC dev) and do 100:1 for grayscale.
Now, assuming analog is not the way to go (sounds like you want to stay digital), first you'd probably want to do motion detection. Only capture the stream from a camera that is reading changes. Depending on the application, that can cut your storage space down significantly, especially in off-peak hours.
Second, I'd probably rethink 30 fps. 15 fps is nearly as good, and takes half the storage space. ATM cameras can range from 5 fps to 1/5 fps... and give good enough results in most circumstances.
That combination should take you down to between 1 TB and 100 GB per day (maybe less depending on usage)... which is definitely more manageable. At that point, an HSM is the answer... look at any of the other excellent posts regarding this.
I worked at one point for a major US corporation with similar needs, and they have a data backup center... it's basically a huge warehouse with tape drives, robots to change tapes and cart them around the warehouse, and tons of archive space for the said tapes. Perhaps this would be a good solution for you? As I recall, they spent about $5 MM for the hardware, and about the same for the actual installation (which is terrorist resistant, etc.)
Hope that helps!
I am disrespectful to dirt! Can you see that I am serious?!
Might I ask why your requirements include arbitrary retrieval time? I would put it to you that if you have arbitrary retrieval time, you DON'T have to save everything. Are you saying that if I built you a system that saved everything at your highest bandwidth constraints, but took a year to do a retrieval, that is acceptable? I seriously doubt it.
If you are saving that much and care that little about it, you don't need to save that much.
You'd be stuck with the expense of digging through all those terabytes.
I mean, without some assurance he's not talking about surveillance cameras, why make his job easier?
sounds like you are a spy from the fbi... are we starting to map everybodies faces now?
Lots of people are suggesting that you use a large tape robot or some monstrous disk array. Don't. It's expensive and you don't need it.
Use small, discrete, but identical systems. It's more easily scalable, more reliable, and easier to implement. A 1U server with 72GB of internal disk and four-drive, 20-slot rackmount DLT autoloader gets you a well-buffered 12 MB/sec and 800GB of storage for under $30k. You can fit 14 of these systems in two 2.0M racks. For about $400k, you've got 168 MB/sec, 11.2 TB, and if one or two crap out you don't care.
This whole thing smells bad. Probably both a civil liberties nightmare AND a huge "I smell dolllars now" chance to profit from the 9/11 attacks. Tell us why you need to do this. Be specific. Another case of slashdot editorial silliness.
They stab it with their steely knives,
But they just can't kill the beast.
Wouldn't it be cheaper and easier to get an exculsive contract with a doughnut bakery, then 'hire' 1000's of 'cops' to stand on every corner. I think it could work...
Just pay everyone on slashdot $10 to run a special version of morpheus. Then just send everyone feed from one camera. When you need to access it, just hope they're online.
Username taken, please choose another one.
This reads like a commercial project and you are in it for the money. So why not hire some consultants?
I'm sure there are a lot of talented folks who could use the work right. If you have that much cash to spend, spread the joy.
I was posed a similar question at my place of employment, and the only thing I could think of that was remotely feasable was to use DVCR's or security vcrs. Thus the bulk of the information, could stay on tape, and we'd only transfer onto a computer, what we actually needed.
Free Techno/Jazz/DNB/MI Music by guys obsessed with monkeys!
The easy way is to go buy 30-50 standard vid storage solutions and stuff em in a security room.
Concidering alot of places have 50 cameras on all the time standard storage solutions are bound to exist and all a 1000 camera system is is alot of 50 camera systems in tandem.
Just make sure its completely automatic.
It MIGHT require some 50 t1 lines or some oddness like that but it should work fine for what you seem to need.
Tape degrades fast
What do people usually do for extended storage? I have some 3M Blackwatch tapes that are 20+ years old and still readable (while several CDs purchased around that same time frame are not), but obviously 9-track isn't really a viable medium for today's storage needs. What are people using for long-term archival storage?
Just junk food for thought...
Hmm, 8TBytes a days isn't going to be cheap. Gonna need lots of tape storage, lots of it.
n de x.html
b li c_manuals/td/TSM/SH26-4083-01/en_US/HTML/cp5ech15. htm
r /
:)
I would recommend an IBM 3494 tape library loaded with about 6 3590 E1A drives (2 for write, 2 for read, 2 for reclamation). The 3494 is very flexible and modular, fully decked out it has a 6,000+ tape cart capacity. This would equate to approx 748TB of total storage capacity.
Now for managing the entire thing, I would use Adstar Storage Manager. ASDM is now sold by Tivoli and not IBM direct, but is the same product. Use the HSM component. Basically you'd have a huge filesystem. When the video data is streamed in it gets written to disk first (requiring about 8TB of disk). The HSM manager will then migrate the data off disk and out to tape, but still leave the file entry in the filesystem.
From that point on if you need to access the video, you open the file that appears to be in the filesystem, but the HSM runs off in the background and mounts the tape and starts streaming the data back to disk for your client to read. Your client will block waiting for the tape to mount, but that would be acceptable considering the amount of data being stored.
This post is incomplete, but you may want to follow up on the 3494's at
http://www.storage.ibm.com/hardsoft/tape/3494/i
Also, investigate ADSM and how its HSM works for using a tape library to back a disk subsystem.
http://www.tivoli.com/support/public/Prodman/pu
http://www.tivoli.com/products/index/storage_mg
Out of curiosity I'd like to know what your budget is on this
Ok, here's what you do: Take all of your signals (don't even bother compressing them... as other people have pointed out, it looks like 9 GB/sec might be hard to compress) and beam them straight out into outer space. This solves both problems - the massive amount of data, and the high bandwidth.
Now, current technlogy offers few options for playback, but since you don't mind waiting for the playback (you did say arbitrarily slow), I'm sure solutions developed in the future will be both feasible and cheap. (if not, just wait a little longer). Here are a few possibilities:
Develop ultra-focused and ultra-sensitive receivers to detect the signal bouncing off of a distant object.
Teleport a big mirror ahead of the signal to bounce it back. Listen for it to return.
Develop a faster-than-light vehicle to pass the signal, and then use future technology to record it on some futuristic ultra-dense medium. (Note: this can be done obeying Einstein's theory of relativity. Space isn't quite a vacuum, so the radio signal is travelling at a speed less than the speed of light in a vacuum. You could evacuate an area of space for your vehicle to travel so that it is less dense than the space the radio signal is travelling through, and thus has a higher speed of light. You'll be able to catch up without exceeding the speed of light in your path, but exceeding the speed of light in the radio signals path).
Wait for either the universe to collapse in on itself, or a black hole to devour both the earth and the radio signal. This should, with any luck, warp space such that the signal hits the earth again.
Direct the signal at a black hole such that it orbits it. The signal will be trapped in orbit. You could send a vehicle to this area to read the signal, and then retransmit it with more energy to escape the black hole. Of course, you'll only get a brief snippet of data before the vehicle is sucked in, so many may be required. (I'm not too good at black hole theory; hope it doesn't show!)
Anyway, there are lots of possibilites!
-- morcheeba
founding member, literalist society.
HIV Crosses Species Barrier... into Muppets
I have dim memories from those lame Discovery channel 'Inside Las Vegas' of shelves of neatly arranged VCR like devices, with a couple of spare tapes next to each one.
They probably use custom long record hardware and rotate the tapes, keeping them only as long as necessary.
Bleh!
I have the solution: use the face-recognition hardware on the camera signals. Then you wouldn't have to store the images themselves, but only the names of the people in front of the camera! Store this in an easy-to-read log format:
20011129 18:30 John Doe at location 23
20011129 18:31 John Doe at location 298
20011129 18:31 KNOWN TERRORIST at location 13
20011129 18:32 John Doe leaves location 23
...
Very easy!
An archiving workstation would have four DVD-RAM drives. The drives would be written to one at a time at a speed around 3GB/hr. At 90KB/s per camera would come out to be 324MB/hr per camera. This means that an archiving workstation/DVD-RAM drive could handle the streaming data from nine cameras. To handle 999 cameras would require 111 archiving workstations. These would record approx. 1.6 hours of the nine cameras per 4.7GB DVD-RAM media. It would go through the drives in round-robin such that it will have written to three of the four drives in approx. 4.8 hours and started writting to the fourth DVD-RAM drive. Assuming there is at least two operations staff available 24x7, then they would have to replace the 3 used media with fresh media at a rate of 5 minutes per archiving workstation.
Hardware costs: Since DVD-RAM drives have come down to about $500 a drive, I figure a nine camera archiving workstation could be built for around $4000 each. For the 111 archive workstation farm plus a spare it would total to about $448,000. DVD-RAM media costs about $25 each. An archiving workstation would go through 15 DVD-RAM disks a day for a cost of $375 per day for one archiving workstation's work. Across the 111 workstation farm this would be $41,625 per day in DVD-RAM media. The first year of operation would cost $15.7 million in hardware and media alone. There would still be the cost of two operations staff available 24x7 and storage of 1665 DVD-RAM media disks per day or 607,725 DVD-RAM media disks per year!
Who the hell modded this as insightful?
Broadcast TV is 752x480, which shows that this poster is an idiot.
Tip for the moderators: If you don't know that the information is correct, DON'T MOD IT!
so the motion/change detecting algorhithms may be of no use. Face-recognition in a crowded walkway in an airport would require 30fps or close to it. The compression can be done at the camrea so that is not an issue.
There seems to be (2) issues at hand:
1) How to store this data given his original paramters (I would want to work with worst-case numbers myself anyways)
2) SHOULD we be helping without knowing wjat we are helping with (we already have in some ways)
My personal view is that if we are helping someone bid on the recently beta-tested face-recognition systems (in Miami I think it was) that will be part of the new anti-terrorism work we should know that up-front before we help..
That being said, remember that a run-of-the mill 16/10/40 CD-Recorder records at up to 140MB/s
Funny, I have access to a 16x cdr and it takes me quite a bit longer than 5 seconds to burn a 700MB cd. Additionally I have had my cd software return a perfect burn report only to find the cd horribly mangled the next time I actually start to pull something off it, so a run-of-the-mill CDR is probably not the way to go. Unless of course you intend to make X copies of each disk for redundancy's sake or fashion some kind of parity check on incoming data and the finished cdr (RAID cd-recording?).
Green-voting, republican-registered, socialist-libertarian.
Ok what you need is not easy to implement or going to be cheap.
It'll also depend on your availability requirements. EMC has 2 lines (symmetrix and clariion). Clariion would be cheaper and actually faster than a symmetrix. Symmetrix has some more features but costs quite a bit more. You should go with mid sized Sun machines like a 4800-6800. Use Vertias filesystem. What about a database? SAN? For longterm storage I would off-load to tape daily and develop a database for cataloging what's where etc etc to ease retrieval. What about clustering? I'd recommend Netbackup and use Flashbackup to do raw filesystem backups if there's no database. You've also have disk choices from Sun (T3), and hitachi. You probably want to implement a SAN as well.
We have extensive EMC, SUN, Oracle and Veritas expertise so drop me an email if you'd like to discuss further. We have some of the top oracle people in the nation if you need oracle help as well. This is a very complex subject to outline on a slashdot forum. There's too many it just depends situations.
jjohnsonNOSPAM@tenure.com
Recently at a trade conference (Crestech) I saw a demo of a system some grad students had put together which paired a panoramic low-rez camera with a moveable hi-res camera. The panoramic would observe an area, and software would recognize areas of activity and focus the other camera on it. But this isn't really what the Ask Slashdot'er was looking for, obviously.
Freedom: "I won't!"
$5 / month hosted VPS on linux = awesome!
Check out Loronix's solution. They've developed hardware and software specifically for this application. They handle more than 1800 cameras @ 30fps with archival for 30 days for the casino industry. I'm sure they'd be able to put together something that'd fit your needs.
so the motion/change detecting algorhithms may be of no use. Face-recognition in a crowded walkway in an airport would require 30fps or close to it. The compression can be done at the camrea so that is not an issue.
There seems to be (2) issues at hand:
1) How to store this data given his original paramters (I would want to work with worst-case numbers myself anyways)
2) SHOULD we be helping without knowing what we are helping with (we already have in some ways)
My personal view is that if we are helping someone bid on the recently beta-tested face-recognition systems (in Miami I think it was) that will be part of the new anti-terrorism work we should know that up-front before we help..
What technophobic hubris! It's almost too good to be true.
If all you ever think of are the bad uses of new tools, you will never develop new tools. This is technophobia, pure and simple. Every tool, right down to nuclear explosives (see project plowshears), has it's good uses.
It is a mistake to think that things will not be developed because you deny their positive uses. Someone, somewhere is working on everything. Even if you are the best, you are never the only.
DMCA, Hollings, Palladium. What might have sounded like paranoia is now common sense.
Rob McCready, an electrical and computer engineering grad student at the University of Toronto has developed the first face-detection program that uses programmable hardware - which is much faster and more accurate at discerning faces than any existing software programs http://www.nce.gc.ca/en/success/9920/micronet3_e.h tm
dude, a 16x cd-r takes about 5 minutes to burn a 700MB disc.
The main application for this (that I can see) would be for a casino. In the casinos I have worked at (which are pretty small by Las Vegas standards) I did pick up a few things after talking to surveillance technicians. What I heard surprised me somewhat.
Most places use analog taping systems.
The ability to stop one recorder (or a group of recorders) out of several hundred, rewind and playback frame by frame in a very short ammount of time isn't easy to do. Analog systems in this area have been around for a long time, and have been developing constantly. Digital technology in this area is still relatively new, and has a long way to go before it achieves wide acceptance in the industry. Having said that, there are digital systems out there, but they are *very* expensive. Analog systems are just as effective, and cost a fraction of the ammount.
Why would you want to store the footage indefinately? For a system such as that in a casino, the ability to replay footage a couple of months old isn't that important. Why should it be in your case?
Back to the earlier question: why full motion? Even in casinos, not all cameras are full motion - There are alot of recorders that capture a wide range of frame rates. One or two FPS is perfectly adequate for watching traffic down a corridor, for instance. Even in a slightly different application, the principal is just the same. For your higher security (risk) areas use recorders with a higher frame rate, and lower frame rate recorders for your lower security areas.
Basically there are hundreds of systems/solutions out there. Each with their own advantages and disadvantages. What you have to do is look at the solution that best fits you. Digital may be nice, but it is expensive and the standards are still changing. Analog is much cheaper, but archiving for long periods is difficult.... For a better suggestion, a bit more information is needed.
First you are talking about a very large storage
system. Of course, you are talking about alot of storage. It does not, however, need to be all one type. If you use a product such as SAMFS you can get the benefit of extending that to other media. As mentioned before, silo's would be a good adjunct to this system.
Calling SAMFS HSM software short changes it. It
is far more then that. With it you can have multiple systems read the same volume. It has HSM parts wich will migrate data from one stoage medium to the next leaving a part of it on the original media (as set by an admin). It also makes backups of those archives making it also a backup product.
It has the added benefit of using QFS as its file system. It stores the metadata seperately. Meaning that you can restore the entire system if you have one of its archive copies and a copy of its metadata file. The metadata file is of course much smaller then the data tislef so you can effectivly restore the system in a matter of minutes where it normaly would have taken hours and days.
SAMFS is also one of the few procuts (maybe even the only one) that can realy drive the storage media at drive rated speeds. Ask anyone what they realy get off their drives and you will find it is usualy about 2/3 of its throughput.
Finaly keep in mind you can use things other then tape. You could put it on optical storage as a longer term solution.
Finaly, while you will get some good information from slashdot you realy should enguage a storage specialist for this. Many of the storage companies have them. If you have any doubts about doing this, then get help. Because what you set in your data center today will come back to you in the future. It is much easier to impliment then to migrate.
Only record movement - it comes standard through a number of security vendors. This can save a lot of space, especially since you don't want to record 12 hours of a stair well nobody is using between quitin' time and business open.
Ctimes2
My cube. My friend. My solace. My prison.
why doesn't anyone wonder about the possible freedoms that systems such as these have the potentional to decimate? 1000+ cameras = a lot of surveilance. on the other hand, why not ask the uk, who already has quite a few security cams covering most of the public square meterage?
Everyone is busy trying to do the sums for any number of various technologies, and as uninformed as I am, most of them seem more or less accurate.
But assuming these are fixed cameras... why can he not hope to get better compression ratios? If this is in a warehouse, or factory... then only 8 hours of the day will show very much difference at all (per camera). Even assuming that daylight/nighttime is visible from the camera pov... one nighttime frame will hardly be different from one night to the next. I would think that 25 terabytes of storage could easily hold the mundane video of the place for years, and that another 25 terabytes would hold many years worth of the "difference" frames. The compression you might manage on a fixed location grayscale camera over a period of months should be something phenomenal.
For any given season/hour, the system would have an average frame, and the computer would check to see how different it is. If its below a certain threshold, say 5% over all the pixels, or 2% concentrated in any given area of the frame, store the difference. We're talking 256 possible values for a pixel, and likely pictures that are easily compressed by any number of algorithms.
A more relevant question is what kind of cpu power would it take to do this on the fly...
Just get one of these: http://www.sun.com/storage/L700/
I was wrong in my hasty calculations... 1 second != 1 minute...
So 140MB/minute = slower than 90MB/s. But the number of discs per day is correct.
-Adam
I Hope its very expensive to do this.
Get a thousand computers running Linux and a NIC, nothing too fancy (~$200 each) a nice Promise SuperTrakSX 6000 (~$200 each) 10 160GB ATA drives (six attached to the RAID controller, four attached to the motherboard, ~300 each.) Total price, a little more than three million, and you get 1.6 PETAbytes of data storage (less if you take advantage of the built in RAID.) That's about 200 days worth of capturing. Assuming you don't start for a few more months, I wouldn't be surprised if there were 250 GB drives available, which would give you about a years worth of storage. You can probably assume costs/PetaByte goes down at least 50% a year, so this might be a surprisingly good long term strategy--just keep adding boxes to the network...
The first thing I thought of was six and a half thousand PC's (that's six per camera basically) each with four 120GB hard disks.
Total cost is around £6 million (which is mostly hard disks) and gives you a year of storage. Admittedly that isn't cheap, but it does give you total redundancy and is better than having to buy a warehouse load of tapes every week.
Besides, you get QUICK retrieval into the bargain, although you say you don't need it, I bet someone will say "Quick, we want to review now!". That many PC's could also cut distribution costs, by only requiring a VERY slow (modem?) link to whereever controlled them for sending back video when requested and controlling things, and even then - one at most is going to be needed to send stuff back from the sound of your post...
No doubt someone will shout me down for being stupid, but if you set up the first 2 months of PC's first, then when they start to near maximum capacity you set up the next 2 months worth you get to also add the latest in cheap but enormous hard disks to them, maybe you can cut the years data storage down to £4 million.
The added extra of being able to erase a set of PC's at will and start recording again is also useful. I doubt you need to record that much video over a huge timeframe and with tapes you have to move them around, etc. to reuse them.
Just did a lookup on Google for the name given in his email-address, and this came up:
e .h tm
http://www.nce.gc.ca/en/success/9920/micronet3_
He's obviously a Big Brother wannabe:)
Seriously. If you want to know how this kind of thing can be pulled off, talk to the heads of security at some of the major casinos.
I think some sort of BEOWULF CLUSTER may solve this problem.
I don't think that there are any realistic solutions to your problem.
It is cost prohibitive, and quite possibly technically infeasible.
Such a system would be very large and fragile. Forget building in redundancy... you are over budget before you even begin. I seriously doubt you could ever achieve even 90% operational status (camera, workstation, network, mass storage... too many failure points).
Why won't you tell us the application you have in mind?
I am very small, utmostly microscopic.
Let the Feds drown in their own problems, please. I am not going to help anybody building such a structure so that thousands of faces can be analyzed in realtime, to provide yet another excuse for poor to no investigative skills within the agency.
Forget it.
VCR's are cheap these days
We use an optical jukebox with worm media here
at work for a rather large image database (45
million records).
The jukebox has 16 drives and can hold 1000 platters. Each platter will store 5.2 gigs of data, but our jukebox is a couple of years old, the current drives will do 9.1 gigs per platter.
This'll give you around 9.1 tarabytes total per jukebox, you'd probably want to have a couple of jukeboxes driving this and then rotate the platters out the jukeboxes for storage.
Only probable is that the platters are around 100 USD a piece, at least in the volume that we purchase. For your application you could get them cheaper.
While this is a large scale project, it's far from impossible.
First, it's important to convert the video to a good lossy interframe codec. If I had to do this project today, and wasn't concerned by free software issues, I'd build something like this
Athlon XP SP machine
Osprey 500 card
Windows Media Encoder
The 500 card will go from the composite input of the security cameras, and do a hardware conversion to 640x480 progressive YUV 4:2:0, which is the native input mode of Windows Media. The Windows Media encoder will then have enough extra CPU power on on the SP XP machine to do a 640x480 29.97 fps encode in real time. For archival quality, you'll need something less than 1 Mbps (125 KBytes/sec) to store. You could set them up to use a qulaity-based instead of a datarate based encoding mode, so the data rate usage would drop enormously when nothing is going on.
This is the stream you'd transmit to the storage facility, which would then have to store 1 Gbps at peak (or 125 MBytes/sec). However, real-world needs would be substantially less than that, since most of the cameras aren't showing anything most of the time (although they'll probably have peaks at similar times, like rush hour).
125 MBytes/sec is way to much for a single mechanism, but it's certainly in the realm of some real-world applications. Among other things, it's very easy to parallelize this.
For those concerned about putting long-term archival data to a Microsoft propritary format, there are some other options. Today, MPEG-2 is likely to be the best bet, although it's require peak data rates more like 3 Mbps to achieve the same quality. Typically the hardware will cost more as well.
Within six months or so, MPEG-4 should be a viable option, which combines quality in the ballpark of WMV8 with the openess of an ISO standard.
My video compression blog
If sending all data into 1 point is too hard.
Split it. perhaps 100 cams for 1 node not bad idea.
You can easly maintain and backup these nodes.
I believe this aproach can solve many problems.And cuts costs.
[My english is better than most other people's Turkish, so please point out mistakes politely. Thank you.]
HPSS:
http://www4.clearlake.ibm.com/hpss/index.jsp
It was built to do stuff like that.
Would you shut up already?
when you resort to a perl script when 2 men and a truck would do!
don't hit whitey
So he has to buy some trucks and a warehouse and hire some day laborers to move and stack the tapes. That's his problem.
Now that we've given him an answer, let him figure out the details. He's not paying us, you know.
Post it on a bunch of public web sites,
and let google do the archiving...
Yeah really, like we're that smart to know how to build something like this ourselves. Dude, we're just as stupid as you - that's why we're playing Quake3, instead of writing Quake3.
First, is 30 FPS absolutely ESSENTIAL? Many security applications can get by on 10 or 15 frames per second, which in itself would cut your storage needs down by 50-66%.
Again, is 640x480 essential, or could you get by with 400x300? Again, cutting small corners will save immense space in the long run.
Secondly, forget uncompressed... Use something like the Sorensen streaming encoder (i forget which company markets it... perhaps apple?). Pair it with a G4, and you've got a live video compression station. If the camera's are recording the same scene, the compression should be great, probably figure 80-90% compression...
Lastly, forget recording to disk... instead investigate recording straight to DLT. Otherwise you'll just have to redouble your efforts, first recording to disk when the ultimate desination is probably going to be DLT anyhow...
That's it for now... i STRONGLY disagree with anyone saying that 1000 VCR's is the ideal way to go... But who am i to say?
I question why you would need to have recordings that are 20 years old that are as good as those in the last day so you could consider scaling and degrading the video quality with time.
Example: Record everything in color with high framerates and high resolution on hard disks. After the data is a week old (or whatever), compress it to B&W with half the frame rate and resolution. Repeat till you have a minimal desired quality then move it to long term storage (tape, etc).
Feel free to count me as a co-inventor- I have no application for this algorithm at this time.
-BxT
I am an Engineer for a company that does only storage, so I might be able to offer some suggestions. The best solution would probably be SamFS, which is a Hierarchical Storage Management product developed by LSC software, now part of Sun. SamFS runs only on Solaris Sparc, so that means a Sun box. Your reqs. would max out an E450, so you should look at a 4500 or 4800 at the minimum. For disk, avoid Sun T3's like the plague. They suck. For your needs, a Clariion FC4700 running RAID 3 is perfect. So perfect, that Sony just signed an OEM agreement to sell Clariions with their video editing solutions. For tape, I would suggest LTO drives in a StorageTek L700 library. SDLT is too new to be trusted. Also look at AIT-3 in SpectraLogic Gator 64000 libraries. If you have the cash, the ultimate tape solution would be STK T9940 or 9840B drives in a StorageTek 9310 powderhorn (as seen in the movie Eraser). Unfortunately, a powderhorn with no drives is about $200k, T9940 drives are $35k each, and 9840B drives are about $30k. Good luck.
first off, when i do find out were your trying to put this cameras in use, i will put on my $5 face mask and break as many of your cameras as i can.
o rage/DisplayPages/supplies.htm?DataPage=2200mx-juk ebox
secondly, cd burning silo's. im not familar with who makes them, but they do exist. your looking at 129 cd's a day so your looking at around $75 MAX to store all your data on cd-r.
this wasnt quite what i was thinking of but it might do the trick for you http://www.products.storage.hp.com/eprise/main/st
however the cost for storage is way up from using plane cd's
-- botsex is {grep;touch;strip;unzip;head;mount}
You could contact this company:
http://www.c-3d.net/
These folks are the makers of the FMD (Fluorescent Multi-layer Disc) that will have storage amounts of 20 - 100 Gigabytes of data on a regular sized CD (120mm). With retrieval speeds of up to 1gigabit/second, this might be an option for you.
Although they have not yet released a product for the public to buy, they might be interested in your problem. They are currently working on being the new standard for Video disks and HDTV disks.
Goran
Carpe Scrotum - The only way to deal with your competition.
Sounds like you are going for one of the BIDS (https://www.bids.tswg.gov/) projects.
Based on other posts, I would guess your project involves face-rec in an airport. As others have noted, I think your requirements are way too high, even for the app you may have in mind.
Your res is OK, and is probably necessary - no need to change that - but I would drop the greyscale a bit, from 8 bit greyscale to 4-6 bit greyscale. If you went with 6 bit greyscale, you would end up with 230KB/frame - with 4 bit greyscale, it would be 153KB/frame.
Drop the frame rate to 15 frame/s, and you would end up with 3.5 MB/s per camera (6 bit) and 2.3 MB/s (4 bit). Do your vid compression and you would be looking at ~35 KB/s (35 MB/s) and ~23 KB/s (23 MB/s) respectively, or 2-3 TB/day - which is much more reasonable.
Even this can be improved - only record extreme movement - if you are doing face rec, you don't care if one pixel changes, or a few pixels change - even rapidly. But if a whole lot of pixels change, record that. In fact, you could even use a variable frame rate system to up the frame rate recorded, from say 0-1 FPS (little to no movement), up to 30 FPS (lots of movement, subject nearly fills camera FOV). Doing this will probably save a ton off of the bandwidth and storage requirements for the whole system - but this would be difficult to give numbers for without knowing the exact use of the system, camera placement, amount of traffic in camera path, etc. This would have to be known to decide whether to put such a system in place.
If making these changes to the requirements is not acceptable (I have seen a posting claiming you are a guy doing face-rec with an above 80% success rate, but that you need 8 bit greyscale, at 30 FPS) - you may be looking at a near impossible to implement system, without the ability to spend a lot of money for storage and processing needs. Something a government would be willing to do a few times for certain installations, but not something commercially viable, except for extreme economic situations (perhaps casinos and such).
Hope this helps...
Reason is the Path to God - Anon
Instead of trying to store 8TB a day of video, why not look at ways to reduce the video. If the video is for security reasons(that is my assumption) then why do you need 30fps? Why do you even need one frame every second?
:)
Another solution may be to take an initial image and then simply record the changes, similar to what CVS does. It's much more efficient to store just a few changes than an entire image*30 every second. This solution would probably require a lot more computing power, then it's easier to add computing power than infinate storage space.
MPEG-4 may do some of these things, if so, you already have a solution. If not, get cracking a "fun" algorthim
Sometimes I feel like a nut... Ok so it's most of the time
So, I understand that your customer wants 24x7 30fps. Part of the job of the engineer is to extract what the real requirements are from the "wishes." I'd argue that your customer likely wouldn't mind 15fps, for example. And, I'd argue that (assuming this is a surveillance system) you could add motion sensors or delete the scenes that aren't any different from the background, vastly decreasing the amount of data. Also, the 'stored indefinately' thing is a bit silly -- can't you put a 10 year window on it? Otherwise, your data will eventually fill every warehouse in the world, and your customer won't be able to afford the rent. It seems to me that this is really your initial job -- shrinking the amount of data that actually needs to be stored.
The question implied some sort of centralized storage area. I think this is a dumb idea, because you're going to spend huge bucks on the centralized equipment for storing it, compression, decompression and so on. Instead, I'd locate storage with each camera or some small (3-5 camera) subset of the cameras on something manageable (like a few tape jukeboxes or similar), and pay somebody to go around swapping tapes every few days or week or whatever. If you want to run a network, make it cheap -- low-speed ethernet or serial. Sure, it'll take a while to download any given thing, but so what? If you really want it quickly, you can always go get the tapes.
A SAN is probably 1 option, but it's way $$$ our IBM SAN SHARK only has 1.2 terrabytes, and a tape unit that backs up most of that daily cost about $1.7 million. You would probably only need 1 SAN disk device(can be scaled quite large 10+terrabytes), but multiple backup units, as the tape changer units only do about 20 Gigabytes/s split across 2 drives, so the Maximum backup speed can hit 40 Gig/sec and that's only after the data is cached to the tape controller units internal drives.
Pipe camera output to /dev/null.
/dev/random, it may take a while but read access wasn't crucial. ;)
Read out images from
//Humming
I'm too stupid to preview.
It's the only thing that I can think of that fits. Seahaven supposedly had 5,000 cameras, not 1,000, but maybe 80% of them were off-line most of the time. And you just know that Christof would've wanted every bit of video kept forever, because you never know when an event from Truman's childhood might have significance that isn't realized until years later. But only keeping B&W is OK, because it signals to the audience that they're watching archival footage.
Nothing for 6-digit uids?
30 frames per second is the problem. Most security cameras work taking 1-3 frames per second using a timelapse VCR. At 1 fram a second, you can get a lot of hours on one VHS tape. This is enough for the police to use in an investigation.
That would be one amazing multi-angle DVD porn!
30% off web hosting. Coupon code "SLASHDOT".
Sure, but I meant as a complete solution for security cams (and e.g. optimized for surveillance with the higher idle degree)
If you have to program and implement those algoritms yourself, you'll be occupied for a while
Get yourself a few human brains. The storage is emense (so they say), and the retreival time is instantanious.....unlesss it has a brain fart, then you're in trouble....at least this storage you can talk the data out of it....
Store it all on cdr's. You can most likely fit atleast 1-10, 24 hour day video's on one cdr, maybe even more, especially if you use compression such as divx. This would require some automation to burn the cdr's and do the compression either real-time or after the video has been recorded. Sticking to the issue though, it'll cost only about $.20 per cd, with 1-10 video's per cdr and a total of 3000 camera's that is only about $60-$600 a day for the cdr's. Pretty low compared to tapes, which will cost more than $.20 each, and take up a lot more room.
As automation goes, cd's are easier and less expensive.
Question everything.
http://www.ampexdata.com/Products/Mass_Storage/mas s_storage.html
Used for storage of digitized movie frames (24 fps @ 3000 x 4000 pixels x 120 minutes x how many composite layers?) during post operations.
Yeah, but 1000 cameras to generate the data wont be cheap either.
It seems like you could reduce the cost by distributing the load a bit more - find a solution for 50 cameras, then buy 20 of those solutions. This would allow you to use cheaper components and not be constrained by throughput performance. The calculated network throughput was 90MB/s, which puts it in the realm of gigabit ethernet if you try and push all the data to a single location.
You could probably divide things into tiers - you have 1000 cameras connected to 20 machines on 50 seperate 100baseT networks. The 20 machines operate as a pre-tape cache, so they should have fast disk access and a reasonable amount of space. On each of these machines, have a second interface to a gigabit backbone, where there are 2 or 3 machines that archive to tape.
It's not going to be cheap, but you're talking about 8TB/day of storage with 1000 cameras. There are no cheap solutions for that kind of problem.
-- Patience is a virtue, but impatience is an art.
Look at any of the posts proposing a solution and you'll notice that they describe (or assume) a 3 step process: Capture the video, encode/compress the video, store the encoded video.
:o)
This is not a new problem. In fact, the problem has been around since before TV. Consider a parallel 3 part system:
1) gather information
2) convert the information to a more convienent/succint data format (this would also strip out miscellaneous info.)
3) store this data.
This describes the problems of inventoring, chronicling, and surveying people.
Once we have described the necessary system, we must apply constraints. You have already defined the first 2 parts of the system . . . or have you?
Consider that the 90MB that you want to store each second as the information that you are gathering (not the video) as the input to the system. That is the real system you are looking for. The first constraint is how will the system gather the information (or in what ways can you provide the 90MB/s?)
Fiber optic cable?
Printed on paper at high resoultion?
Tapped out like Morse code?
The second constraint is how acccurate must the data be when it comes back out of the system. Since you've already compressed the data and many compression algorithms cannot deal with even one bit of inaccurate data, we'll assume that the data is as succint as possible and nothing can be removed. That will garuntee that this part of the system has a 1 to 1 mapping and therefore the system is invertible (it can be undone.) However, this does not mean the data cannot be converted to a more convient format.
There are a couple constraints on the final storage. There is a maximum amount of space it can consume per amount of data and whatever the storage element is made of must be in great supply. (The ideal storage element would take up zero space and be in infinite supply, and you aren't gonna find that . . . at least not on a limited budget.)
So let's say we were provided with definite values for all these constraints. All you would need to do is design a conversion mechanism, with the input being the 90MB/s stream of data, that perhaps applys some transform to the data and then impresses the transformed data upon the storage element. If you can design all that within your given budget, congratulations!
If not then there is one final solution: shutdown all the TV networks, movie studios, and security companies. Then go around and destory every camera, burn every videotape, smash every DVD and delete all that porn lying around on your computer. No more video means no more video to archive
In many present and (not so far) future experiments in HEP we deal with this kind of data rate. A nice overview can be found here here.
On page 14 you can find the data valume. It is at about 100 TB for present experiments (I am with BaBar).
page 25 gives some overview on the hardware we use at BaBar/SLAC (e.g. farms of STK Powderhorn tape silos with 6000 tapes each, etc..).
page 95 gives an overview on data rates. ATLAS records at 100Hz and 1MB per event, i.e. 100MB/s
Page 99 gives overview of the (estimated) costs of hardware and tapes for LHC experiments. They are in the order of 20 MCHF (Mega Swiss Franks ~ 0.8 Mega Dollar) initial + 10 MCHF per year. We use a mixture of large RAID farms and tape silos. Everything is managed by HPSS (High Performance Storage System). From my experience at BaBar I can tell you that these numbers are underestimated by at least a factor of 2.
KdenLive/PIAVE - non-linear video editing
There is a company that specializes in this form of archival storage and retrieval. Nice (http://www.nice.com/nicevision/index.html) is a company that markets "Nicevision" and they do this all the time. In this day and age, I'm afraid VCR tapes and tape monkeys don't cut it for evidence retrieval and don't hold up to FBI and court standards.
Many companies that use this very expensive technology include major retailers, airports, hospitals and casinos. I work in the security products industry and see this equipment at shows and have participated in selling it to a couple of major airports.
One of the reasons to have this vs the old fashioned way is that security managers now can tie specific security incidents coded in the access-control software to specific time frames in the video. The video of a security breach can be played back "on-demand" for the FBI. It is also more admissable in court and some insurance agencies are beginning to demand this level of surveilence.
It also ties in well with facial recognition.
Follow me here for a miniute.....
Having 1000 Cameras all feed into a box is a problem I don't wish to be saddled with solving. However why not just scale up a system I use for a few cameras to fit the task:
What you need:
A roomful of rackmount hardware, 100-Base-T switches, and air conditioning.
1000 2U ATX Cases, with matching motherboards, 1.2ghz+ CPU (Intel or AMD), Bt8x8 PCI VidCap Card, and a 100baseT ethernet card.
4000 100gb UDMA hard drives.
Make 1000 2u rackmount computers with 4 100gb Hard Drives in each unit, with a RAID 1 of 2 RAID 0 sets (IE make 2 200gb Volumes, and mirror). Vidcap at 640x480x30fps with decent codec: you now have about 500mb/hour. This yeilds about 16 days of data on each unit. Make the system delete oldest files when drive full, keeping a 16 day log.
Need to grab some suspicious video? Browse to the machine that is attached to the camera you are intersted in over your cheap 100baseT network, and pull down the video. Archive it to CDR if necessary.
This is a reasonably cheap (about $1500 per camera max) and easy solution. And it would be pretty easy to implement. Also, any failures will take out only one or two cameras, not all of them. There are a few caveats tho...
*Unknown long term reliability of these inexpenisve hard drives - I'd guess that they will be fine, but we've raided just in case. Mind you, swapping out ide hard drives on a daily basis in 2006 won't be fun if there are problems down the road.
*Same as above for using commodoity PC hardware.
*Only 16 day archival length.
I've implemented this (in a much smaller scale of course) sucessfully, and if the caveats don't break your requirements, this is a wondefully simple way to go.
The interesting thing about this question (to me, anyway) is that it's quite like the video on demand problem, but backward.
With video on demand, you take a bunch of high powered servers, load 'em up with all the movies the world has ever produced, and then let thousands of clients each request their own video streams from the servers.
Here, you've got a thousand or more clients each acting as the source of a stream, and you want to record all those streams. You've still got the same issues with a large number of independant, high-bandwidth streams, though, and you appear to want to save all those streams onto a centralized server or group of servers. Get yourself a vid-on-demand system and press the 'record' button instead of the 'play' button, and you should be all set.
Given Moore's Law, if you wait ten years, I will then lend you my palm top. It will probably be a bit overkill, but hey!
Just pipe to a file, append the famous "Eschelon blocker" sig file and let the NSA/CIA/FBI do the archiving for you!
Retrieval is no further than a FOIA (Freedom of Information Act) request!
--
Spaz!
Been done, and our company is already doing it.. The security industry has to deal with this sort of thing already..
Melkor had a similar situation on his hands in _The_Silmarillion_. I find his solution practical and elegant: Simply enthrall one of your mortal enemies and afix them to the top of your building, where her unblinking eyes will see all.
If the ACLU gives you heck, simply get your lawyers to amend the DMCA and/or the constitution as necessary. Better still, give 'em some rings of power. It's the american way.
- undoware.ca
Maybe it's a new Google site:
http://pRon.google.com!!
Have you looked into Panasonic CCTV (http://cctv.panasonic.com)...they have large scale analog and hard disk based recording components, that are very flexible and extensible.
Then there's the issue of loading the drives. With small tape changers, you could load them up with a days worth at a time, or you can pay someone to physically change tapes every couple hours.
Of course, I work at a place where we used to run 32 Exabyte tapedrives in parallel to take data for experiments. But we had Graduate Students (read Slave Labor) to change tapes every two hours on shift...
- "History shows again and again how nature points out the folly of men" -- Blue Oyster Cult, 'Godzilla'
System Parts (1): StorageWorks Modular Storage Solutions
Quantity Part Number Description Estimated
Retail Price*
(USD)
252 232432-B22 72.8 GB Pluggable Ultra3 10K Drive (1") $ 515,340.00
12 Fibre to Hub/Switch Connection from Storage to Fibre Hub/Switch
3 135820-B21 M2200 Controller Shelf $ 18,183.00
6 176622-B21 HSG80 Controller - 256 MB Cache $ 98,976.00
18 190209-001 StorageWorks Enclosure Model 4314R - Rack $ 53,190.00
6 380674-B21 RA8000 256 MB Cache Upgrade $ 23,040.00
6 135823-B21 Cache Battery $ 3,408.00
6 222318-B21 ACS V8.6-1F Controller Software FC-SW software $ 42,000.00
18 119826-B21 StorageWorks Enclosure 4200 Redundant Power Supply $ 4,086.00
12 380561-B21 FC Optical GBIC $ 4,440.00
18 168256-B21 1 Meter VHDCI to VHDCI SCSI cable $ 3,834.00
12 234457-B22 5m Multi-mode Cable Kit $ 1,368.00
Estimated SubTotal $ 767,865.00
Storage InterConnect
Quantity Part Number Description Estimated
Retail Price*
(USD)
4 Existing System Slot Requires an open System slot in server
4 Fibre Host Connection Connection from Hub/Switch to Server
4 176479-B21 Compaq StorageWorks 64-Bit/33 MHz PCI-to-FC HBA $ 8,180.00
2 158222-B21 FC Switch 8 port $ 37,940.00
1 380551-001 RA/MA8000 Platform Kit Windows NT Intel/PCI $ 750.00
2 167365-B21 Compaq Storage Switch Universal Rack Mount Kit $ 370.00
8 380561-B21 FC Optical GBIC $ 2,960.00
4 234457-B22 5m Multi-mode Cable Kit $ 456.00
Estimated SubTotal $ 50,656.00
Rack Parts
Quantity Part Number Description Estimated
Retail Price*
(USD)
2 120663-B21 Compaq Rack Model 9142 (42U height) $ 2,704.00
5 207590-D71 Power Distribution Unit - Low Voltage $ 1,640.00
2 169940-B21 Rack Blanking Panel Kit (15U) $ 92.00
1 120670-B21 42U Side Panel Kit $ 212.00
1 120669-B21 Baying Kit $ 85.00
Estimated SubTotal $ 4,733.00
Estimated Total $ 823,254.00
Either a motion sensor or a program that will compare the last image taken to the one before that, if they are the same, it's just a pointer, or something like that ... 90kb down to 4 bytes should save plenty of space.... just a thought.
__________________________________________
Take comfort in your ignorance.
Grandmaster Plague
OK, with regards to all the data needs of 1000+ cameras grayscale 640 30fps....
Hardware compression on the fly X 1000streams...not too too bad these days though. Come on, It's grayscale. 256 palette at the most. But regardless, without touching your original specs (i think 50KBps is more like it for the streams with cool-guy compression tricks):
Why must there be a need to have 1 system that can write the 90MBps AND back up the TB's by itself?
Cell-based, it's much more realistic with OffTheShelf equipment already. If you break it down to a 10-cell system, 9MBps and DLT Backup (or whatever new tape-daddy systems are rolling now) make a perfect solution. I mean, last time i checked, which was over a yeah ago, DLT 40/80 drives were writing at 15MBps.
I'm assuming that cost is not as much of an object, given the scope....but manageability is. Cell-based is perhaps the only way to go, And will eliminate Downtime into the specific "Failure Zone" of the cell that temporarily goes down. But WTF, You can easily have 1 or 2 backup "Cell Systems" that can be routed to and fired up within a minute or 2 of failure.
The only cost that builds is the tape media. Reduce this by using the best match of price/storesize/performance....and then scale your older tapes to optical if need be, for physical storage constraints. Trickle your older tapes to the new on-the-horizon High capacity Optical DVD's and recycle your tape stock.
Everyone might bitch that a huge floating DLT library might get expensive after, um, a few days or weeks....But this is keeping with manageability AND the "implied written in stone" requirement of 640x480x30fps with compression. This way takes the central failure points away from most of the aforementioned systems, along with their ludicrous initial prices.
whew....got me really thinkin' on this one.
TeraGlobal can do 100:1 wavelet-based compression in real time on any of the G4 macs. IIRC, they were actually able to compress one stream and decompress another simultaneously.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
First, a disclaimer...
I work for IBM SAN Support, so I am obviously most familiar with our products, which are indeed quite good, but there are several sources for most of this stuff (except the tape drives).
Here's what you are going to need...
1) A big honkin' network and fast video processing boxes (This is the most odious requirement, the rest is quite easy)... You are going to need some serious bandwidth to pipe this stuff. Unless your compression is onboard, Gigabit ethernet is your only option for multiple cameras. (You would be pushing the bandwidth for Fast Ethernet on an uncompressed stream) Even then, you aren't going to be able to pipe 1GB/s through the box, because you are probably not going to get an adapter to go at line speed. Also, unless video compression is a lot easier than I thought, you are going to need an damn fast machine to handle that many compression streams simultaneously.
There may be some professional boxes with an embedded MPEG encoder so you don't have to try and to that heavy lifting on a server. (I know the chips exist, as IBM sells them (they are used in the TiVo)) Where would you get such boxes? I don't know.
2) I am no sizing expert, but you are going to probably need, oh, about 3 impressive servers (read: plenty of 64-bit PCI slots, perferably on separate busses) to handle data "buffering" and tape backup chores. These servers would need Fibre Channel boards for the I/O, and of course your network adapters. Two of each.
3) Tape drives: Piece of cake. IBM happens to sell an LTO (Linear Tape Open) library, the 3584, that will do nicely. Fully configured, it has a half-petabyte capacity using 150GB carts. The max. theoretical spindle speed w/o compression (which would be useless here) is 15 MB/sec. My actual observed field speeds are around 10-12 MB/sec. (Backing up the uncompressible Win2k SP2.) Each drive frame can accommodate 12 drives, and you can bolt together up to six frames to form a library. (There can also be drive-less "expansion frames") The tapes aren't cheap ($150/each, IIRC) but still aren't bad. The LTO drives are currently the fastest and largest drives on the market, and I don't think anyone other LTO vendor sells a big honkin' library like the 3584. As far as your automation needs, the robotics can easily handle the puny number of daily changes you need, although you are going to be making a LOT of use of the 10-slot "I/O" door to insert and remove tapes.
4) It sounds like you need a highly avail. box for disk storage. A Symmetrix is a popular, but expensive, choice. IBM of course has the 2105 "Shark" box, which is just as good, and not nearly as pricey. I have heard Hitachi also makes a pretty decent storage machine. You don't need a very large one, because you only need to hold the data long enough to pipe it to tape, however, I'd suggest at least a day's worth. I think you will need more than one though, as 180 MB/sec of I/O is probably going to be rough on any box. (I really don't know what the I/O bandwidth of those boxes are, though. That's what sales drones are for.)
5) Infrastructure equipment. I reccommend the InRange FC-9000 fibre channel switch for your SAN. Non-blocking and quite reliable. I know nothing about network equipment.
5) Someone that knows real backup software cold... Veritas NetBackup or Tivoli Storage Manager (TSM Rocks!) is what you need, but it is not trivial to configure.
SirWired
I'll arbitrarily assume a per-feed post-compression datarate of 180 kilobits per second is adequate for a low-framerate delta compressed surveillance camera application. (At 5fps that's almost 4.5 kilobytes per deltaframe) - For 1000 feeds that's a datarate of 175megabits/sec, or 1.8 terabytes/day. Let's break down those numbers...
If you have one 1U dedicated hardware compressor station for each 8 feeds, 125 of those are needed (you can place them in racks of 8-10 units each around the monitored facility, to shorten the lossy analog signal paths from the different camera feeds) If each hardware compressor station produces a 180kbps stream for each of its 8 streams, that's 1440kbps or 1.4mbps which alone won't choke a 10mbps ethernet segment, though 125 times that will make a 100mbps ethernet segment barf. So split whole data capture network into 5 x 100mbit segments - very manageable, well within sustained load tolerance on each. (125/5=25) - 25 x 1.4mbps feeds = 35mbps each segment handling 200 feeds.
On the receiving end you need to dump to something. Why not tape? It's relatively inexpensive and if retrieval can be arbitrarily slow then it is not a bad solution - spinning disk arrays sound like overkill to me. Modern tapedecks have high capacity, high reliability and high data rates.
Assuming 32 gig capacity tapes and 40 tape stations recording in parallel (eight for each of the five network segments) let each tape station handle 25 feeds. That's 25 x 180kbps = 4.39 megabits per second, which is a reasonable buffered datarate, that gives you a per-tape capacity of 994 minutes, or about 16.5 hours between tape changes.
So - about every 16.5 hours some monkey^H^H^H^H^H^H archival lab technician has to change 40 tapes. (for 25 tape stations that's 10.3 hours between 25 tape changes, or 30 tapes every 12.2 hours, or 50 tapes every 20.7 hours, or whatever...)
To prevent tape underruns, the tape stations could each have two disk buffers each equal in size to the tape capacity and a tape write session would then just be dumping from one buffer, which would be deleted as the tape write session concludes succesfully, while the other session buffer is being filled by incoming data. Very likely the tapes can be written and even verified much quicker than it takes to accumulate data for one session, so in case of tape verify errors you could have a panic mode beeper-alerting a technician to put in a new blank tape because the previous one was a dud, and still have time to complete a second write-verify cycle well within the time of one session period. Harddisks are cheap, so consider putting as much as 3 or 4 buffers on each tape station to increase the headroom tolerance for human error and physical tapedrive breakdowns, cleaning and maintenance.
However you choose to do it, it seems like a manageable workload, and there's robotic systems that can do all this and much more. As for physical storage, you're looking at something like 58 tapes a day on average, or over 21,000 tapes a year. It sounds like a lot, but you won't need 'warehouses' or huge hosted server arrays to store all this data. All this assumes that offline storage and slow data retrieval is acceptable.
For redundancy, place a couple of spare tape stations on hot standby on each of the network segments, and have these spares with extended disk capacity receive and accumulate buffer copies of some of the feed streams in parallel with the master tape stations, so they can dump a master station's worth of buffers to tape if a master station fails. This gives you time to spin up a replacement preconfigured tape server to assume the role of the failed master tape station.
If anyone wants the spreadsheet I used to play with these numbers, contact me. kobot at kobotica dot com
/dev/null is fast to write to and never fills up, you can stream all your data into it! Sure it's not all that easy to get data back out of but you said that wasn't a priority.
-Lewis
You could consider the following:
:-)
StorageTek Powerhorn 9310 which holds 6000 tapes.
In this Powerhorn you can have 40 9940 tape drives which hold 60GB (Native) per tape. This will net you ~360000GB or ~360TB of storage capacity. The 9940 is a really fast drive too! (10MB/Sec) With 40 drives you should be able to get 1.44TB an hour. This is yours for the low low price of ~1 million plus the price of tapes (~80$ each!). I have done something like this but with 9840 drives and it works great! BTW: I am not a StorageTek salesman!
One of the most important things overlooked by all 4-point and 5-point posts in this thread is that you want to store it indefinitely. This means that you have to retrieve data after some time and rewrite it, since no media lasts forever.
When it comes to storage in these amounts, there is only one thing to do: make a spreadsheet with different product's total costs during the system lifetime, and remember to include the costs of regenerating data on outdated media in the future. And then choose the technology that has the lowest total cost of ownership.
November 28-29, 2001 - Boston Park Plaza, Boston, MA
Also, Exabyte has the X200 Tape Library with:
My instincts tell me you're not going to WANT to point ALL 1000+ cameras at one central point. Rather, indeed, build the system from building blocks of around 50-100 cameras. Storage and management systems for this sort of thing are very common already, so you can run several of them in parallel quite easilly.
This also has the 'durability' factor of avoiding a single-point-of-failure if you point ALL the cameras into one storage system.
If you're truly wanting to go fully digital, I'd still say keep it modular, and buy some of the DVD-R 'automated silos' that can hold a hundred DVD-R's or so, which gives you ~850-900Gb of storage per silo. The media lasts long, takes up very little space, and each 'silo' can have usually around 4-8 DVD-R drives installed in it, which gives you a cummulative recording speed of 12-24Mb/sec per silo, assuming a 3Mb/sec recording speed per DVD-R.
I'd also assume you'd spend the money to install cameras that have the compression built into the hardware, so the DVD-R silo's can simply be handed the 'raw compression' in 3-hour chunks from a hundred cameras per silo per DVD-R, which would still give you a healthy 'margin of error' to deal with the management of fresh DVD-R's in the silo's, and recording drives possibly failing, without having to worry about actually losing any of your data.
I'm specifically preaching a modular, broken-down format since you made such a point of data being archived forever. That meant, to me, that you NEED to keep the data, above and beyond most else, so who cares if you can record all that data, if something fails and you lose 3 hours because of that? This way, you have a lot of 'fallback' points, without having to worry so much about any single component going belly-up on you.
Place a frontend of your choosing (I'd use a pair of redundant sun E450's) to a EMC SAN. For backup use SDLT's in a StorageTek ® L700e tape library. Specs are 800GB/hour, 74.6TB per silo, with 2 silos able to be hooked to one robot. This will provide for plenty of redundancy should one or two SDLT drives go out (its a little more than 2X the requirements). It would require changing all the tapes out every 18 days, so half the tapes once a week roughly. Tape cost is going to eat you alive though. 1400 SDLT's/week is going to cost ~$180,000/week.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
While many of us are a little too wary of astroturfing and other nefarious plots by Big Brother to ironically get our help in plotting our own demise (pauses for breath), the reason why a lot of Slashdotters are so darned interested in what you're doing is because they would also like to try reframing the original problem such that it doesn't require 1000 cameras running 24/7. Forgive us for wanting to think outside the box.
Black holes are where the Matrix raised SIGFPE
my 2 cents, namely:
It is obvious you need a storage area network. A distributed, multihomed storage area network. You need to partition your cameras into logical groups and build a SAN to support each smaller, managable group. Then allow for connectivity between the groups and any clients wishing to retrieve video from the archives. It would be dangerous to put of all your storage in one place (even if it is offsite), and too impractical to have duplicates of all the video at each site. So build a few offsite datacenters (use whatever COTS you find cost-effective, it's not like EMC doesn't have what you want), and put in a few redundant land lines between facilities and the actual site.
Of course, the closest I've ever come to planning anything of this magnitude is a RAID-5 array and a tape carousel, so take my ideas with a grain of salt.
Black holes are where the Matrix raised SIGFPE
They make such systems using racks and racks of video digitizers that save to hard disk that is groomed to tape on large tape library systems. My my company Qualstar makes the libraries. Casinos typically need to keep 7 days of video from ALL cameras to satisfy legal requirements (gaming commissions). Most often this is still being done with VCR but lately the move to digital is on.
I believe they typically digitize at 15fps and often the video is multiplexed four inputs to a frame. I've been in several of the casino installations. They use enough CAT5 to rival a co-location facility (the camera feeds are CAT5 also).
Perhaps this might ruin it for you but the OS is NT. They use dual CPU (that is two motherboards) rack-mount boxes for the digitizers. Single CPUs for the tape controllers (interfaced SCSI to Sony AIT-2 drives). And a bunch of camera switch and control stuff all over.
The cool thing is their software that lets them see a bad guy and then queue up the back history following him/her camera to camera (with branching to follow accomplices). They call this sort of thing "investigations" and they used to take hours and hours. Now they can do it in 10s of minutes and dupe off a VCR tape for the police/whatever.
Most Target stores across the U.S. use the same system on a smaller scale.
I was the night admin at an insurance firm for a little while. They also hired a pretty young communications major to come in at night and do the back ups.
...er supervising. I told her she'd been doing a good job these last few weeks and asked, "shouldnt you be needing more tapes soon?"
all she had to do was wait for the beep, change all the tapes, then press any key.
one night i went into the machine room to see if she needed any
she blinks and looks at me, "tapes?"
i have never run down the hall that fast, nor have i ever sweated as much during a backup.
--------------------- Turn evil by smiling.
Hey,
Ok, say you have 1000+ cameras emitting 30 frames/second worth of 640x480 grayscale video...and you have to store it indefinitely. What do you do?
You cut down on data. First of all, you don't want 30fps. PAL video is 25fps; You don't need that sort of quality. 15fps would be more than enough, probably even less.
Second, you don't record stuff that doesn't change. You need a codec that supports 'Map identical pixels to transparent'. That way, only changed data need be recorded, cutting down on file size substantially.
Thirdly, you delete all-identical frames. For all-still areas like stairwells, this will cut right down on files.
Fourth, use a good CODEC. With greyscale and not too much data to store, you can get VERY small files.
With all these measures, you won't need as much storage space as you estimate. It would be variable though; high-activity areas with all-day traffic would still produce a lot of data. I'll assume it'll output 30 kilobytes per second, being on the pessimistic side.
To store all the resultant data, I'd use a two-tier system of PC computers. I note you havn't mentioned how you plan to digitise all this data, but that's out of the scope of this post.
Let's connect 10 cameras to each PC, and we'll have 100 PCs. 30KB/s times ten cameras is 300 kilobytes every second. That would be *easy* with Fibre Channel (2 Gbits/sec!!). 300kbps for an hour would be 1055MB/hour, about a gigabyte every hour. If we can get an IBM 73 Gig drive, that'll give us just over 3 days of storage per computer.
I'd have the computer accepting the telemetry, compressing it and writing it to the drive. Every 24 hours, we'll open a new file. Each computer connects by 100Mbps ethernet to another computer. Every day, this will copy down the 24GB file from the last day, and record it to a DAT tape. Say we get 120 Gigabyte tapes, that's a tape every five days, times 100 computers, that's 20 tapes per day.
After this, decide on a set archive time. Keep the tapes for, say, 5 years. You'll need a good indexing system and a warehouse for tape storage (You can get automated, robotic systems), but if you have 1000 cameras, I expect you have a fair budget.
That's what I'd do, anyway.
Michael
"Goodness me, how unlike the FBI to abuse the trust of the American public." -- The Onion
Yes, it's called video tape. You can buy it at KMart.
Any video compression algorithms worth using for this kind of application do comparisons from one frame to the next, and only compress the differences (except for occasional reference frames.) Some of them do substantial motion compensation to model the differences, others don't. Many of them let you tweak the frequency of reference frames - is it every 10? Every 100? Do you need the ability to go backwards, or is smooth forward and clunky backwards good enough?
Very few locations actually generate much motion on a 24-hour basis, except for road traffic cameras, and I'd be extremely surprised to see an application need to store those on a long-term basis (as opposed to storing for a week or so in case there are traffic accidents - anything you need longer than that should probably be handled by license-plate recognizers.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Just wait 10 years. By then we will have 10 terabyte drives. We might also have ten gigabit networks by then. Asking this question was like asking how the how do I build a multi-terabyte database 10 years ago.
on a side note, your probably going to want to develop a database application, probably with a perl/php/etc front end to organize all the file names-to-date-camera etc.
If you really need 30fps, and that much daily storage; the only way to go is either Tape or CD/DVD.
2 00 00/20113.99999.product.BPC.html
There are CD/DVD "jukeboxes" out there, but none that I know of that will fit your needs. You need to check out Sony's solution.(my old co)
go here:
http://bpgprod.sel.sony.com/bpcnav/app/99999/4/
If that doesn't work, go here: http://bpgprod.sel.sony.com/
and look under Products / Data Storage / Automated Solutions,, click on DMS8800- PetaSite(TM) System
"PetaSite(TM) System
$151,200.00 U.S. List
The flagship for Sony's large scale digital archive and enterprise data backup solutions. Scalable and highly flexible in design for a storage applications up to 29.0 Petabytes (compressed) or 11.2 PetaBytes (native)"
If this amount of storage isn't enough, ask about the "T-Junction".
Model#DMS8800J
For every T-junction you use, you almost double the original spec.. i.e. 11.2=22.4=33.6 and so on.
The original unit has one Tape Deck Rack that is all Tape Decks attached to 27 Tape Storage Racks with the menacing robotic arm.
Every other Tape Storage Rack can be a T-Junction for an addtional 27 Racks of Tape Storage.
The T-Junction is basically a drop off point for one Robotic arm system (27 racks) to another robotic arm.
If you are good at Tetris, you could probably come up with the Theoretical max Rack layout!
While I was there, there was talk of additional Tape Deck Racks(DMS8800D) to speed up access.
Call them up and ask for SIC (Systems Integration Center) located here in San Jose,Ca.
They can configure one up for you and give you a quote.
Good Luck!
ip cameras or the like on their own dedicated network attached to multiple servers would be good. the servers suck the data from the cameras and throw it out to a SAN. the SAN would have to be quite large initially, but should eventually shrink as time passes. A standard 42U rack can hold 14 3U RAID boxes + JBOD units. Something like the IBM Profibre DF1100 can control 7 JBOD units, or 80 disks in total. Use multiple of these to get 140 disks into the 42U racks. Use 73GB LP FC-AL drives. This will give you a mirrored capacity of around 5124GB of space per rack. 8TB/day = about 2922TB/year, or 4383TB per 18 months. A years worth of storage is 584 42U racks, while 18 months is 877 racks. After approximately 17 months, buy 122,780 new LP FC-AL disks(140 disks per rack). The capacity has to be at least double. break the mirror and replace one mirror set with the newer drives and rebuild. Then repeat with the other mirror set. You now have another 18 months of storage in the same amount of space. Repeat this process. Obviously, you would need disk monkeys on hand to continually change failed disks. You would also need multiple arbitrated loops, possibly connected by a switched fabric.
Smile, aliens might be watching.
If you are going to buy 6,000 Terabytes of data per year for the next X years, please just do me one favor....let me know what company so I can buy their stock!
Kind thoughts do not change the world
You see? You see? Your stupid minds! Stupid! Stupid!
*Obviously* you do the compression out near the cameras; anything else would be silly, with the possible exception of doing lightweight compression at the cameras and heavier-duty compression at a centralized server, if you're in a LAN / Metro area network where bandwidth is cheaper than distributing lots of hardware. But you still need to figure out what level of data transmission is realistic - "90KB/sec" is 720 kbps, which is half a T1 line. (As a telecom vendor, I'd be happy to sell you 1000 T1s or equivalent Internet or FrameRelay/ATM bandwidth :-), at least assuming you're in the US where I've got bandwidth to sell instead of subcontracting. But your problems are much different if you're trying to camerafy every FooBar Retail Store in North America vs. trying to camerafy every street corner in Toronto.) But do you really need this much? Most business video-conferencing is 384kbps or 128kbps,and even 64kbps can be surprisingly good, especially since you don't mind grayscale. This lets you use DSL or ISDN to connect your remote sites (or if your cameras are in groups, lets you put more cameras per T1 from each group going back to your central locations.) This also means that your central location (or locations, depending on your disaster recovery needs and your bandwidth-vs-operations needs) can get by with a much more realistic data connection - if there's any long-distance component to the communications, you'll be much happier buying an OC3 than an OC48 or GigE, though in a metro area, if there's dark fiber, it doesn't much matter what speed you run at.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
How about interleaving the images from all the cameras. Is it possible to put these into a multidimensional space and then compress it using dimensional compression formulas. Also the more images the more possible blocks of similarity and possibly better compression.
When shit hits the fan get some of these https://youtu.be/pY-GncsZ-UE
Where I work, we have been having problems of people busting the windows and grabing whatever they can before the security people show up.
B/W cameras are very cheap (it cost more for the box and wire than the cameras) but recording the video is very expensive. Since I don't live in the land of low cost NTSC, I'm forced to go with PAL and the result is that a single VCR that can record for 24 hrs cost about as much as a well equiped PC. I can get a switch box for 4x the cost of the cameras that will switch between 4 cameras.
So whats the best option? How many vidoe capture cards can I put in one cheap PC? What compression can I use? I suspect if I use a multplexor then the compression will be very bad. How stable is video4linux with 4 cameras going at once?
Excellent- then contact me about my royalties since I have patents on this. I knew I could get at least one sucker... :) :)
Seriously- I had thought of this as well years ago (also for a security system) that I was contemplating development on (I bagged the whole thing though). I thought this would suit Robs system well since it is also for security purposes and no one is really going to care about 20 year old data.
As for my "co-inventor" commit- I think it's important for everyone to remember the MORAL obligation to credit those that provide you with direct assistance (even if their ideas are not completely original).
-BxT
I've been thinking lately about massive video storage for entirely different purposes.
Lets suppose you have a TiVo with broadband internet access (DSL/Cable/Fixed Wireless, whatever).
You want to watch something that aired yesterday. You did not tell your Tivo to record it. Instead, you request the show through your broadband internet connection. It streams to your Tivo in the background (doesn't matter if it's slower than real-time, the Tivo has become a streaming video cache).
Now consider the storage and bandwidth requirements for the back-end of this 'service'. You would need incoming feeds from every supported channel. To leave room for expansion we'll call it 1000 channels. If you're clever with your algorithms you wouldn't need to re-capture reruns (you would need to properly handle commercials which would significantly increase the overall complexity of the problem). So each unique 'episode' is stored, then each scheduled viewing indexes the locations of commericals as well as which commercials to insert. A little complex but I would estimate 80%+ of what is broadcast is rerun material. That reduces the number of effective 'new' material to 200 channels.
In a perfect world we'd store the complete history of television. But if we just stored the last week it would still be a great feature. So 200 channels of mpeg2, 24 hours a day for 7 days. 33,600 hours of video. It will be randomly accessed by N simultanious customers (where N should be as large as economically possible) which effectivly cancels out the option of using tape.
(warning: shameless abuse of Moores Law follows)
100 GB ~= 30 hours, so we'd need 112 TB of storage (ouch!) for one week. Scarry stuff huh? If hard drives continue to improve at current rates, we can reasonably expect to have 1 TB consumer level drives in three or four years. We can also reasonably expect gig ether in the server farm, 5-10Ghz processing with 3-6GB of RAM and 20-50Mbit consumer broadband transfer speeds. Our 112 TB isn't nearly so difficult in that environment. With a farm of 256 boxes with 8 consumer level 1TB drives each we'd have 2048TB or 18 weeks of programming. Of course we'd want to have hdtv resolutions in that timeframe...
So my question: Is this reasonable? In five years will we be able to randomly access television?
Specifically:
l
r dw are.html
m l
http://www.sgi.com/products/media/broadcast.htm
http://www.sgi.com/origin/3000/digital_media/ha
http://www.sgi.com/products/storage/9400.html
http://www.sgi.com/products/storage/software.ht
Ok I don't know alot about this kind storage amount but what about using 10fps the human eye can only catch about 12fps so it don't really matter, that would it down to approx 2,7 TB/day. Then do the only motion thing and save 100:1 and that'll be appox. 27,3 GB/day or appox. 28MB/day/cam. Then use CDs for storing a 700MB cd would be able to take 25days/cam and could be easily stored.
If this is for face recognition technology then the most video you really need to store is pictures of individuals that aren't recognized by your system. Once someone is logged by the system then all you need to store is a message like:
"Subject N passed camera Z at HH:MM:SS"
Everyone would be numbered as they entered the facility and then tracked by that number.
-BxT
This is an interesting problem I think because of the data longevity requirements.
w ar e/Storage/)
It will be extraordinary and expensive, but you can certainly get 100+ terrabyte solutions that are fully automated, both disk and tape. There is a small (almost entirely North American-based) industry and these things are used by a variety of people for a variety of uses - commercial, scientific, defense...
You can certainly drop cash over a fairly large range and to play with various speed/reliability/support tradeoffs; your budget will dictate how quickly you can get data back at a given age. One thing is certain: you're going to end up with some large, impressive looking pieces of equipment suitable for display in a harshly lit, well air conditioned space.
People have discussed brands and preferences - there is some competition in the marketplace. You will want to make a big project of shopping around. Have a good read...
(http://directory.google.com/Top/Computers/Hard
Call people up. They will certainly lavish attention on you if you give them the same spiel you gave us. Ask vendors who their competitors are - it's not taboo, and they will tell you, if they're confident their own solutions are better.
Moving right along. Depending on how important retrieval speed is at various ages of the data, you may opt for a very, very big hard drive array (think EMC). Regardless, I expect you'll ultimately be thinking tape output, and a very, very big tape library (or three, depending your data's popularity in relation to its age) (think StorageTek), and develop a regimen for emptying it and storing the tapes - probably in a very expensive safe place.
All this you can do. The relative price/performance to existing analog systems and "package deals" by existing security vendors (there's a google category for them, too; exercise for the reader) is up in the air, but you're not landing on the moon, and my guess is there are already some (<50?) systems like yours running as we speak.
BUT... and now it gets interesting: your system has one very unusual requirement. "Indefinitely" is a long time. And the system you chose to archive your data with has been designed for redundancy, not archival storage. Tapes tend to have a high rate of failure, and their rated lifetimes tend to be _short_ (generally 3-5 years, as it is, for instance, with DAT or AIT or DLT). If indefinitely is really "3-5 years" then fine, all good. However, if it's not...
Interestingly, tape capactities are losing the race against hard drive capacities quite spectacularly, so, if you just want to buy yourself a "little" more time (maybe 5-10 years instead of 3-5) you can (arguably) consider archiving the hard drives directly. This will cost more, of course, both in drives (my intuition says it will about double your ongoing media costs vs tape, unless you can find a way to use EIDE drives, in which case it might even be cheaper than tape - !) and in making sure your storage facility has a well-conditioned environment. But the fact remains, select your drive vendor well (and you will be able to develop an excellent relationship with a manufacturer, I'm sure), drives will last longer than tapes under the right conditions. Maybe twice as long. Maybe longer.
Of course, you will get HD failures too. And this is ultimately a troubling regimen because drives are complicated and have many moving parts - stiction, fragility, etc... Because of vagaries of the manufacturing process, some drives will last 50 years, and others 50 days. And anyway, maybe for you, indefinite is like unto the next generation...
In which case, you pretty much have to go optical. DVDs might prove very attractive. Given factors of size/weight/storage capacity even the commercial stuff available now doesn't look _so_ bad, and I would personally think about calling Sony (for instance) and chatting with them about it. There might be higher capacity formats you can look into (trading off vs. price). I expect you can find someone who makes a DVD-R robot that would fit the bill.
Your DVD-like media has exceptional durability and will last... well, a long time. I say this because I spent some time a while ago attempting to find a longevity rating for DVD/DVD-R/DVD-RAM media (when we were considering an archival project), calling all over hill and dale, and I finally got the answer that there really isn't a rating, because really nothing has failed yet due to "age," and based on the materials they don't have an "expectation of failre." Take that with a grain of salt - it's the manufacturer talking - but still... plastic lasting forever has some kind of upside at least.
Ideally in such a situation you want to be able to skip the tape step and go right from hard drives to optical. Your application's needs will dictate whether or not you can get away with that. Still, interesting to think about it. The time capsule people in Denmark were talking about using specially treated plastic paper to hit a 1,000 year lifetime, so it all comes full circle eventually...
We're on the road to Tycho.
Upsides:
Has four decks capable of recording video, digital video, or digital data. 255 slots for tapes, so plenty of storage. And is built for this kind of usage.
Downside:
Think megabucks...
-- ted russ http://www.arach.net.au/~ted/mydynes/ http://www.arach.net.au/~ted/myblogs/
Okay, I'm really pulling this one out of my [thin air], but given the budget involved, you might be able to contract advanced-access to prototypes of those 50GB/side "super DVD" writers. 160 discs per day would be easy to store.
On a related note, 1000 x 3KB per second can be handled by 3 8x CDR drives, thought you'd need to store 1280 discs per day.
-B
Not that this wasn't entirely predictable.
Note: Not affiliated with any of these people.
.5TB native/1.4TB compressed on AIT tape. $8,000.00 * 16 = $128,000.00
1. APS Technology: Medea VideoRack. 600GB at $8,000.00 * 100 = $800,000.00.
2. APS Spectra 2000. Up to
Plus computers to handle the incoming information, backup units in case of failure, UPSs, cables, and storage site usage. The extra units above are also in case of failure. With #1 & #2 above you can store 100 * 600GB = 60TB a day. You only need 8TB so that gives you a bit more than seven days worth of storage. As others have said - you get software which allows you to move the older information off of the disk array.
Unlike others - I suggest you should use the DAT tapes as they are less than 1/8 the size of a VCR tape, stack easily, and would only take up a reasonable sized area on a daily basis (eight tapes maximum per camera or about a four foot cube). Also, if you used MPEG4 as someone else suggested then you might even be able to get away with a greatly reduced footprint for storage. Depends upon how busy the camera is.
Also, on the computer end - nothing greater than a 733mhz system is needed to handle the streaming video if that is all it is going to handle. Thus, you can buy a rack system with 1000 systems if you wanted, make it a Beowulf cluster, and it would only cost you about $400,000.00 for the entire set-up. So in other words - you are looking at about $2M dollars for everything. Including the people to set everything up and automate the entire system.
Later.
Mark (not to be confused with the other Marks who have already posted)
markem@ev1.net
I was at O'Hare international airport (Chicago-ORD) last night at 11pm for picking someone up from a flight arriving at 10:52pm. I would guess that more than 50% of the security camera probably had very little to no moment in front of them. Even being able to remove storage of just one hour on 50% of 1000 cameras can save ALOT of storage/cost.
http://www.divatv.com/
These companies should be able to provide a solution. Big, fast VOD servers are no big deal anymore.
"Well, good luck finding a judge that doesn't run a bestiality site."
On most Linux systems, there is a device called /dev/null where you can store an infinite amount of data. Yet it never uses any disk space!
Maybe you could store your video feeds there!
// Alan Porter
I'm not giving any kind of insight on this. Think of the implications of "anybody" easily storing massive amounts of video. I'm not helping bring in a police-world 1 second sooner. To the poster: Fuck you.
but didn't tell the application.
;-) (don't get ruffled on that, I can't administer it alone either!:P)
... beyond ya
Sorry, but I've chatted with people about hospitals and gone thru the proofs on the load and distribution and backups (can you imagine what happens in an ER if the xray sent up for soft display goes down because some idiot cut the fibre?)
There's alot of room left to wiggle- you yourself have punched the numbers in and know what it takes. The first thing I thought of when I read the post is realtime face detection / catalog, database building, surveiliance for a small city / Olympic Games.
With that much video feed coming in you either have to have 1000 dedicated operators or some sort of computer assisted recognition. Or come to face the tape at a later time.
Very few cameras on the market will give you 30 fps non-interlaced. If you are willing to spend 5% of the budget of these 4K+ cameras... well, guess that gives an overall idea how much the budget is.
Getting ideas on Slash dot is a great step, but it is in no way the a substitute for a thorough analysis of your bandwidth, backup, routing, and storage requirements. Period. Don't think you can administer this alone
If you can have a decentralized storage, that helps. That also approximately doubles your cost as redundant arrays are needed for each location.
Anyway, get that analysis done- please! You may learn that the technology you have to use is priced
One of my friends works in satellite imaging. They have huge tape backup systems that handle around 4 gigabit/s of image backup. I don't know exactly how it works but the system is fast, reliable but probably very expensive. You can probably setup a scaled down version of this system along with some encoding mechanism (since satellite pictures are stored raw), maybe divx ;-) ?
Imperium et libertas
Autocracy and freedom
Sounds like Harrah's or MGM properties are getting serious about security.
Here's how to reduce security spending:
1. Reduce your storage needs by discarding every 3rd frame. Just make sure you put more than 3 cameras (sync'ed, of course) on the perp. Less storage needed = less money.
2. Put the goons on the floor. A few $10/hr thugs are a lot cheaper than a $500MM security system. Or do a combination of both.
3. Intense screening of applicants. Make HR do more than just fill our paperwork all day. Put an ex-FBI/CIA/NSA HR monkey in charge. ALL your floor personnel should be able to get a SEC or TOPSEC security clearance. Hell, those malooks out at Area 51 need something to do in their off hours, anyway...
4. Facial recognition systems. Install on EVERY entrance. USE them. When you see an undesirable, show'em the back exit a la 'Casino...' Plus, collect and hold any fugitives and turn'em in for the reward money.
5. Put Cowboy Neal at the door with a fungo bat. Tell him to beat the hell out of any 'suspicious looking' characters.
Price per GB, tape is cheap.
Streaming to Tape is "extremely" FAST.
Get the right Tape technology, STK9840B FC 19MB/s native speed, ~9sec seek time. Fastest drive in
the industry today.
Get the right Robot, STK9310 capable of holding
~5700 tapes nearline, capable of controlling 20-80 tape drives. Also, capable of joining many
STK9310's together. Largest robot in the industry
today.
All in all, you could store a "massive" amount of
data nearline. With 1 STK9310 you could store in
order of 150TB nearline.
Couple the robot/tape technology with a good HSM application like ASM, a good clustered filesystem like QFS, a few Sun servers, some fast disk like STK9176, a few SAN switches and you have yourself a solution.
Fiasco! was gpl'd if you could get the author
and his employer to agree to re release it
(copyright problem) it can store great detail.
The pix you mention would compress with as good quality as 75%jpg in under 5k.
Put 4 realtime MPEG-2 encoders hooked up to cameras in a machine and call that a node.
You will need 250 of these machines to take care of your 1000 camera requirement.
Each node will require a RAID array capable of about 6MB/s sustained write.
If a typical MPEG-2 stream is about 900KB/s, then youre looking at around 3 gigabytes an hour per camera. That works out to about 2TB per week per node.
You are probably best just to archive the drives, and replace them weekly, as backing them up to tape will be too slow.
2TB per node x 250 means 500TB of disk drive to be archived and swapped into RAID enclosures per week.
My math may be off - i just threw this together, but this will be a total logistical nightmare, no matter which way you look at it.
Lets say you can use IDE disks - 100GB for US$300. Thats 20 x 250 disks per week, which is $US 1.5 million per week.
Over a year, youre looking at $US 78 million in drives alone, though thte real figure might even be sub $US50 million if you consider the rate at which drive prices plummet.
i would guess $US6000 per node including cameras, so you'd be looking at $US 1.5 million in setup hardware.
Space to archive the disks and staff to shuffle the drives is not included.
All in all, it would be expensive, but you could do this for about a century for less than the cost of a B2 bomber.
I gots ta ding a ding dang my dang a long ling long
2 IBM 3494 Tape Libraries will give you 2 weeks of
online storage along with a backup system to handle
recovery. The tapes can be unloaded by frames with
1 days data taking about 150 60GB tapes.
Each 3590 tape drive can record a frame of 10 60GB
tapes at 14MB/sec so 150 cameras can be recorded
on 1 tape drive. 8 Tape drives can handle your
data rate (16 drives with full redundancy).
No special automation or Silo is needed. A Human
Tape Operator can unload a full frame from each
drive once each 8 hour shift and file it away in a
library.
You would be need to archive 150-200 tapes a day.
A library to hold 7 years of tapes would take
90x20 foot long by 6 foot high, double sided,
sliding tape racks. You would need a 30x145 foot
tape library room.
Your access time to get to any particular frame,
recorded on any previous day would be about 15
minutes.
3590 Tapes have a guaranteed 7 year archival life
and can be extended indefinitely by re-copying them
every 5 years.
Thanx Doug...
You obviously have a fairly large budget. Contract a larger company to build a special data recorder that suits your needs (unless that is what your customer is looking for).
example: D-6 which is uncompressed HDTV records 128MB/sec to tape with each tape holding around 450GB. Philips has had this for a year and it's called 'Philips VooDoo Media Recorder'. There's no reason why the tape mechanism from this couldn't be adapted for what you are trying to do. That is, after all, if you have the budget to have someone like Philips as one of YOUR customers.
Sometimes it better to design a system from the ground up, rather than using bits a pieces that 'kinda' do the job.
How about building a flexible magnetic tape media that can be spooled on gigantic rolls like kite string? With the right flexibility, it could be any width or length to carry any number of compressed video feeds for any length of time. Changes would be staggered and be done by fork-lift..
The tape would only need to be as reliable as the unspec'd specs deem. (Maybe write-once, read a dozen times is ok...)
A good optical storage with high density optical cartridges will keep your data pretty indefinitely and cheaply. Maybe even a DVD solution is availible..
Cool! Amazing Toys.
one big ass beowulf cluster of these.
:)
mass storage servers, fiber lines, hell lot of physical space and more money than i know what to do with.
hehe, i should know, i work with a 1.5 terabyes worth of data.
Lizard "Never let them set limits on your mind!"
The only place I can think of that would have this many cameras is in a casino. A casino is not going to want to compress the images if it costs them quality. Often, what they are looking for is the little nuances in the handling of the cards or dice, or even body language. If they lose some of the image due to compression or due to cutting every other frame to save space this is going to be relativly useless to them. Remember that while the human eye/brain "sees" at a rate that is in the range of 30 frames a second, slow motion videography will require a substatially higher rate.
You can also compress down to 4:1 on each frame with 4 different cameras. That would reduce space by a factor of 4. And also you would be able to watch 4 cams at a time to save you viewing time.
See what Jim Gray is doing!!!
http://www.research.microsoft.com/~Gray/
So many channels and still nothing worth watching.