Slashdot Mirror
Digital Video Capture and High Frame Rates?
Jeff asks: "So the folks at a place called Conniption Films (great name) developed a camera called the Millisecond Camera which can shoot 12,000 frames of film a second. I read the article and thought 'Hmm that's neat' but then realized they were still using an analog process for shooting this highspeed film. Being a geek, not necessarily into the film side of things but curious nonetheless, I wonder, shouldn't a computer be able to do a better job of such a thing? They say the film runs around a spindle going 500 mph (!). Wouldn't that be prone to failure and use alot of energy? Wouldn't it be more appropriate, easier, and overall cheaper to just hook up a high res CCD to a beowulf </duck> cluster of 2 ghz+ machines and capture high speed images that way? Why hasn't it been done yet? Or has it and I haven't seen it yet?" I did a double-take, when I first read this question, and then got curious and did a little digging. Turns out, high frame rates are not exclusive to the analog photography world, and to illustrate my point, I provide this link. It's woefully short on details, and the explanations as to why a camera that can record 1M frames per second is limited to a playback of only 103 frames, but the technology is out there. Has anyone seen any other digital cameras out there with high frame-rates? What visual mischief could you aspiring photographers get into with such a camera?
are often a bit delayed. they're slow. that's it.
The explanation as to why it can only play back 103 frames is QUITE clear... the chip has 103 "on-chip" memory buffers per sensor, and they get cyclicly overwritten with the last 103 frames.
This overcomes the bottleneck of trying to transfer data off the CCD at such high frame rates in real time, but limits you to "downloading" the last 103 frames after-the fact from the chip.
MadCow.
I used to have a sig, but I set it free and it never came back.
CCD simply needs a few milliseconds to regain their 0-volt signal level again before they can emit a new pulse. This recoverytime makes it unsuitable for high speed filming. Helas.
I have a Sony DSC-F707. It takes beautiful pictures but only has enough buffer memory for 3 burst pictures. With higher resolution images (akin to film level quality) you'd need way more memory and throughput than can be supported with traditional flash memory. A external drive mechanism won't work either (i.e. bluetooth) because of the throughput necessary to sustain something at the rate discussed. I mean camera's today can't even do mpeg compression decently.
There are a lot of limits when it comes to cameras connected to PC's. I've worked in a lab where we used cameras that generated 640x480x4 (32 bit color) frames at 60 Hz. Guess what. You can't even buy a HD that can sustain that kind of transfer rate for any period of time. Good thing those computers had about a gig of ram each ;-)
There is actually a few limitations. Bandwidth is the most important one. Here you're looking at the connection between camera and the computer. We used special frame grabber boards, fireware or USB - well... nothing that I know of can handle 12,000 Hz. Next, somewhat smaller limitation is the bandwidth to memory. When you're talking about 12,000 Hz - that will become a factor. And of course - unless you've got about 40 Gigs of Ram (at least) you would want to save the stream. There are Video Vaults which are basically raid arrays, but again - they can't handle this kind of data stream.
Technology is coming along though. The new CMOS based cameras can have fairly high frame rates. You can actually select between resolution and framerate. Last time I checked the fastest they could go was about 500fps (at low resolution), the limit being again the link between the camera and PC. I believe the theoretical limit of the CMOS type camera is at either 5000 or 8000 fps (I don't really remember which - sorry).
There is another form of image sensor called a CMOS (complementary metal oxide semiconductor) sensor, and it is usually used in cheaper devices like web cams, or digital cameras under $100, since it's much cheaper to manufacture. As a result CMOS images generally worse.
To-do List: Receive telemarketing call during a tornado warning. Check.
On the pure digital front, there are units that can record 1000 FPS continuous at 512 x 512 pixels. The system is data-rate limited. The imager can go much faster; if you cut the image size down to 32x128 pixels, you can get 32K frames/sec. At 128 x 128, you can get 11.2K frames/sec. The data goes into a buffer in the control unit (1 GB, typically), and is read out via FireWire. So this system can take a lot more frames than the device described in the article, which stores the images in memory within the imager and can only store 100 images or so.
That's not so out of reach with today's technology. There's certainly no reason to use a cluster, since it could be done internally with the proper (custom, expensive) hardware. I believe the highest bandwidth consumer dram is PC1066 RDRAM, which has a bandwidth of approx. 3.2Gbytes/s. You'd need eleven RDRAM channels to reach 35Gbytes/s, so you get one second of video for each GB of RDRAM per channel that you throw at it.
Number of required channels can be reduced if higher bandwidth DRAM is used, which I'm sure exists somewhere.
Yes, it would be frighteningly expensive, but these high speed film cameras aren't exactly cheap either.
Carpe Cerevisi - Seize the Beer
You're right in the last part of your analysis: it's the latency that kills CCD detectors. The light sensitivity (i.e., the quantum efficiency) of CCDs is definitely not the problem, though: CCDs are as close to ideal photodetectors as you can get. They capture virtually 100% of the light that falls on them (their QE is >90%: film is usually quoted at 20%).
:)
This is actually a point in favor of high speed CCDs : in order to get the same level of contrast, you need about 5 times less light than a normal high-speed camera. Remember that the same argument you made for light sensitivity/light levels also applies to film. They'd need a light source 4000 times brighter as well, as the film is only exposed for a small fraction of the time.
You might be able to do something cool that mixes film and CCDs: have a film made of CCDs that are then read out after being exposed to light. This solves the bandwidth problem as well, because you could have multiple systems reading out the data from multiple CCDs - it's not hard to aggregate GB/s worth of bandwidth from slower sources. The main problem, of course, would be flexible silicon. That'd take some work.
Are you sure about this? CCDs have near perfect QEs, so they capture all of the light that falls on them (as opposed to 20% or so for film, I think). The process for generating the charge is the photoelectric effect, which is basically instant. I think you're more talking about the latency of CCD, rather than the response time. That is, the amount of time it takes to readout the actual frame and let everything settle back to zero.
This problem is solvable: after all, film has the same problem, much much worse: the settling time for film is millions of years (heh)! They solve this by placing huge arrays of film on a loop, and exposing them all for a fraction of time. You could do the exact same thing with a CCD (if you could make flexible silicon, or something like that) that would solve all of these problems.
CCD most distinctly does not suck: you can prove this by looking at astrophotography, which is without a doubt one of the hardest photographic problems that exists: extremely low light levels, and moving targets. Astrophotography is completely dominated by CCDs, because the sensitivity is just so much better, so you can get far more light in a shorter time.
This is the same problem that film would have, though. You can't get sub-photon resolution of light, because it doesn't exist.
For high speed photography, you need lots of light. This is just generically true. But the quantum efficiency of CCDs is virtually 100%, as opposed to film which is much lower. So, this is a strong point in favor of a CCD system - you'd need less light. There's a poster above talking about how in car crash tests they need massive lighting systems to be able to see things. This'd cut down on their power bill quite a lot.
Another option is to wait for 10gigE (along with the rest of the supercomputing world) or go with Myrinet, which has recently broken the 1 gigabit barrier.
Recently? Myrinet has been doing 2 gigabits full-duplex since May 2001 when it started using fiber. Not to mention that full link utilization only uses a few percent of the host CPU. What's the point of fast cluster interconnect when you use half your CPU sending packets through the TCP/IP stack?
Well, For one thing nobody records at that resolution. As another reply stated, DV is 720x480.
Another problem with your simple calculation is that video is never stored as 32-bit color. That's totally unrealistic. The common way to store video is not RGB, but YUV. Because of the way the human visual system works, the color components (U,V) are typically stored at 1/4 the resolution of the luminance (Y), meaning that an 3*X pixel RGB image would be stored as a X+X/4+X/4=1.5X image in YUV, half the number of pixels.
More significant, though, is that fact that just about every digital image recording mechanism stores information compressed onto the storage media. This is true from consumer digital cameras to DV cameras to the Sony HDTV cameras Lucas used for Star Wars.
Consider what it means to take 12,000 frames per second. You're probably recording a single nearly-instantaneous event, or getting many images of a very fast event. In the former case, there will be a series of frames before the event in which nothing is going on, and the difference between the frames is close to zero, which compresses extremeley well with MPEG-style compression. Your data rate could be 1/100th of the uncompressed rate. When the event occurs, the instantaneous data rate goes up, but buffering can solve this, since it probably lasts a few frames.
In the latter case, recording a fast event at a fast framerate, is essentially the same as recording a normal-speed event at normal frame rates. In this domain as well MPEG-style compression is extremeley effective. At the maximum you would need 1/5th or 1/10th the uncompressed rate, but 1/100th is a pretty reasonable number given current technology.
The only challenge with realtime compression at this speed, of course, is sufficiently fast hardware. I think it could be done in parallel -- capture several GOPs worth of data (15-45 frames perhaps) and send it to a compressor, and then switch the buffer output to a new compressor, round-robin style.
In any case, video is usually stored at rates many factors smaller than the uncompressed rate. So if you change the variables of your equation to a more realistic resolution and color depth, then divide that number by 10 or 100, you'll have a more realistic data rate.
720(w)*480(h)*1.5(color)*12000(fps)= 6.2GB/s, divide by 100 for agressive compression but reasonable results = 600MB/s
Still too fast, but not completely unrealistic if you've got a healthy budget. ;-)
You were probably correct a few years ago, but you need to get with the times.
There is no film in the world that can outshoot a high-sensitivity CCD nowadays. Cameras like the Kodak 760x can shoot at ISO 6400 with reasonable quality, which film is utterly incapable of matching with any sort of quality, and CCDs are only getting better.
Yes, crappy consumer digicams suck at anything over ISO 100. But a serious professional digicam beats the pants off of film at high ISOs. (In case you were wondering, my wife is a professional photographer who shoots with a Nikon D1X. I do know a bit about this.)
ZFS: because love is never having to say fsck