Slashdot Mirror
Digital Video Capture and High Frame Rates?
Jeff asks: "So the folks at a place called Conniption Films (great name) developed a camera called the Millisecond Camera which can shoot 12,000 frames of film a second. I read the article and thought 'Hmm that's neat' but then realized they were still using an analog process for shooting this highspeed film. Being a geek, not necessarily into the film side of things but curious nonetheless, I wonder, shouldn't a computer be able to do a better job of such a thing? They say the film runs around a spindle going 500 mph (!). Wouldn't that be prone to failure and use alot of energy? Wouldn't it be more appropriate, easier, and overall cheaper to just hook up a high res CCD to a beowulf </duck> cluster of 2 ghz+ machines and capture high speed images that way? Why hasn't it been done yet? Or has it and I haven't seen it yet?" I did a double-take, when I first read this question, and then got curious and did a little digging. Turns out, high frame rates are not exclusive to the analog photography world, and to illustrate my point, I provide this link. It's woefully short on details, and the explanations as to why a camera that can record 1M frames per second is limited to a playback of only 103 frames, but the technology is out there. Has anyone seen any other digital cameras out there with high frame-rates? What visual mischief could you aspiring photographers get into with such a camera?
You bet. A FOAF(friend of a friend) is working with some folks developing some very cool high speed cams for all kinds of research. They're using CMOS sensors instead of CCD's. These allow you to capture images as fast as you want (tens of thousands of fps) with a corresponding reduction in resolution.
If you could get a hold of a cmos image sensor you could probably rig up something similar but remember those data rates are INCREDIBLY high. Also, that means the length of the shot tends to be fairly short.
A. El Gammal, et al. published a 10,000fps imager with a 352x288 pixel resolution. This guy can maintain the full speed indefinately. Unfortunately is it not a commercial device, but something similar will probably be available within a few years.
Kleinfelder, S. SukHwan Lim Xinqiao Liu El Gamal, A. "A 10000 frames/s CMOS digital pixel sensor", Solid-State Circuits, IEEE Journal of. v38 n12, pp. 2049-2059. Feb. 2001.
The abstact is as follows:
A 352 x 288 pixel CMOS image sensor chip with per-pixel single-slope ADC and dynamic memory in a standard digital 0.18um CMOS process is described. The chip performs "snapshot" image acquisition, parallel 8-bit A/D conversion, and digital readout at continuous rate of 10000 frames/s or 1 Gpixels/s with power consumption of 50 mW. Each pixel consists of a photogate circuit, a three-stage comparator, and an 8-bit 3T dynamic memory comprising a total of 37 transistors in 9.4x9.4 um with a fill factor of 15%. The photogate quantum efficiency is 13.6%, and the sensor conversion gain is 13.1uV/e. At 1000 frames/s, measured integral nonlinearity is 0.22% over a 1-V range, rms temporal noise with digital CDS is 0.15%, and rms FPN with digital CDS is 0.027%. When operated at low frame rates, on-chip power management circuits permit complete powerdown between each frame conversion and readout. The digitized pixel data is read out over a 64-bit (8-pixel) wide bus operating at 167 MHz, i.e., over 1.33 GB/s. The chip is suitable for general high-speed imaging applications as well as for the implementation of several still and standard video rate applications that benefit from high-speed capture, such as dynamic range enhancement, motion estimation and compensation, and image stabilization.
Many people seem to think that digital image quality is superior to analog. This is untrue. So anything you want to analyze by looking at it, you want analog film. Digital advantages are: computer analysis, reproduceability.
Now to your question, the primary digital disadvantage is bandwidth.
Digital images have a very specific size: 1024x768x32 = 3 MegaBytes
Analog images have virtually unlimited sizes (infinite x infinite x infinite). Some people have tried to estimate the resolution of analog images, and they best they come up with is a vertical and horizontal resolution in the thousands, however this is unreasonable. Analog images are more detailed than that.
Now bandwidth calculation:
(size of single frame)(frame rate) = (bandwidth)
(3MB)(12000) = 36000 MBps
So, we are looking at processing and storing about 3.6 Gigabytes per second. I mean processing because we want to use lossless compression, and this would require some very specialized hardware to handle this framerate. This cannot be processed or stored in real time on any modern generalized computer. It should be possible to build a specialized machine to accomplish this.
I have discounted limitations in CCD speed, possibility of using mulitple cameras, high-end hardware I don't know about.
Conclusions: "Digital" is not the panacea. Visual image analysis should always be done with analog film. Digitization is good for reproducing images, and transporting them intact. A camera that does 12k fps is mostly for image analysis of high velocity and high acceleration objects, for analysis in a lab. There are applications of high speed digital imagery, but I don't know any offhand.
Finally, using a computer to process the resulting data takes a substantial amount processing time. So the answer to the question "why not use digital cameras" is "why would you need to?" If you can justify the need, do it. It will require, however, substantial resources which also need to be justified.
For amateur photography, don't worry about a 12k fps camera, stick with the 30fps DV handicams.
Torsten
"why the hell would anybody need 12000 frames per second. The human eye cant process all that plus nothing happens so fast you need 12000 shots of it in a second. This is just plain stupid and its a waste of film."
Because certian events, despite what you might think, *do* occur within 1000ths of a second. (The fireball from a nucelar blast for instance.)
Good cammera's shutter speeds tend to go up to 1/1000th of a second, and can go up to 1/8000th.
As far as the humman eye comment, well, just because you record at 12000 fps, doesn't mean you play it back at 12000 fps...
--Mike
"Not an actor, but he plays one on TV."
Picture an image sensor as a one-inch-square array of pixels. If the frame rate is 30 per second, then 1/30 (3.3%) of the light that falls on the array makes up each frame.
If the frame rate is 12k/second then only 1/12000 (0.0083%) of the light can be used to make each frame. That means that the CCD must be 4000 times more sensitive to light, or you must use a light source that is 4000 times brighter, to get the same results.
And that ignores the fact that solid state light-to-electricity convertors like CCDs have a certain "latency" or "stickiness". Like the effect that the eye sees after a watching a flashbulb, CCDs suffer from after-images, and the brighter the light the worse the problem. Film doesn't have that problem because each frame is exposed on a new "receptor", i.e. a new piece of film.
There's also a different sort of CCD highspeed camera that's used in various types of racing.
That system uses a single row of pixels which can be scanned at extremely high rates - the picture is built from objects moving in front of the pickup row, rather than the camera actually taking a full-resolution image.
Sort of a high-tech slit-camera.
Perhaps not 100% on-topic, but still interesting.
The other factor when talking about extreme high-speed photography (when people are calculating bandwidth):
Most really high-speed cameras shoot in black and white afaik.
If you drop the calculations from 32bpp down to 8bpp for a nic greyscale image, you're starting to get to manageable numbers... Also, adding cheap hardware based compression (RLE or the like), would be able to reduce the data stream to even more manageable levels.
You're not going to be able to shoot 6 megapixel pictures that fast, but 320x240 or 640x480 images should be possible at high framerates. I doubt it would replace film, but it might be handy for quick playback without having to get negs developed.
If you watch the "Bad Boys" DVD (the Will/Martin ver of Bad Boys), they have some very cool high-speed photography of different guns being fired into different objects. They used some sort of kodak high speed imager afaik - around 2000fps.
"Nothing strengthens authority so much as silence." - Charles de Gaulle
There have been many good replies to this thread, though most are talking theory, some experimental at best. A camera module with the ability to capture 12,000 high resolution frames per second is bound to cost a fortune, and I really doubt there will be much competition for a long time. Perhaps a cheaper alternative would be to purchase several currently-available high speed CCD/CMOS camera modules and use a series of mirrors and lenses to allow the cameras to work together in a round-robin fashion to achieve the a much higher framerate. This would certainly keep the project from being locked into proprietary hardware -- be it a single interface type, manufacturer, or other monopolistic attribute.
The idea of "parallel" items is nothing new, we've already seen success with drives, clusters, and even an array of projectors to create a high resolution projected wall.
Just a thought...
anyone else remember eg&g's high speed nuke cameras?
Rapatronic Camera Shots
Treatment, not tyranny. End the drug war and free our American POWs.
See my user info for links.
Let's do some arithmetic:
The wavelength of visible light (in a vacuum) is between 4x10^(-7) and 7x10^(-7) m.
The speed of light is 3x10^8 m/s (in a vacuum). Planck's constant is 6.6x10^(-34) J s.
Put these together, and a single photon of visible light has an energy of between 2.8x10^(-19) and 5x10^(-19) J.
Suppose you want to get 24-bit colour. As an absolute minimum, you'll want to be able to detect 4096 photons per colour per pixel per frame. CCDs are typically 50% efficient, which means you need 256*3*2 incoming photons per pixel. At, say, 1024x1024 pixels and a million frames per second, that means 3*4096*2*1024*1024*1000000 = 2.6x10^16 photons per second, at an average energy of 3.9x10^(-19) J each.
That's an absolute minimum of 1.0x10^(-2) W of incoming radiation.
How much light is available? Well, at "bright sunlight" is approximately 30 W/m^2 of visible light.
That means that you'd need an aperture roughly 28mm across... which isn't impossible, but is certainly not going to be desireable.
So how does ultra-fast photography work? They use really bright flashes of light... which is why you don't want to be filmed for more than a fraction of a second at once.
Tarsnap: Online backups for the truly paranoid
At my work we use the Phantom Cameras from visiblesolutions.com We can get up to 14000 fps (though not at a high res). 1000 fps is possible at 1024x1024. The cameras have 2 to 4 gig of ram on-board with a circular buffer. After the event, you can take your time offloading the data via 1394.
Why? I remember when I first got a digital video camera, the first thing I went out and did was set it up on a tripod, pointing to a full pop bottle, and then shoot it. I ran back to my computer, put it into Premiere, slowed it down, and was vastly dissapointed that I couldn't see every instant of the process. Mostly the bottle's there, then in the next frame, the bottle has a hole in it, then it's up in the air, and it falls. How much fun would it be to see the bullet make its way towards the bottle, see every moment of the impact, and the exit of the bullet in a fine spray of mist? I'd love to shoot in 1000 fps instead of 29.whatever.
Mod my comments down. It'll be fun.
Yep I built a electonic video camera that had megarhertz frame rates 8 years ago. I patented it too. Actually two different designs.
C.E.M. Strauss, "Synthetic Array Heterodyne Detection: A single Element Detector Acts As An Array", Optics Letters, Vol 19, No. 20, 1609(1994)
and
B.J. Cooke, A.E. Galbraith, B.E. Laubscher, C.E.M. Strauss, N.L. Olivias, Grubler, A.W. laser field Imaging through fourier Transform Hetrodyne, proc of SPIE, 3707, 390-408, (1999)
the problem with pixelated detectors is reading out the darn pixels fast enough. Normally this is done by some sort of bucket brigade across the ccd or some sort of serial memory access across a cmos array. very slow. And parallel access to an entire ray is absurdly complicated and expensive
In my approach I solved this problem by multiplexing all of the pixel signals onto the same single wire. Each pixel when activated creates an osciliory signal at a unique frequency. All of these are combined on a single wire out put (amplified by a single fast amplifier) and then the AC signal is digitized by a single fast digitizer and streamed to a hard disk. The frame rate is determined by the frequency separation between the pixels, so if the oscillation frequency is a megahertz then a frame can be resolved every microsecond. This process is continuos and can go on for as long as you have disk space.
the other cool feature is that the chip you do this on is a single pixel chip! not a pixelated array. the pixels come from painting the chip with a rainbow of light. for a 1-D example, imagine red light on the left edge and blue light on the right. when a reference signal comes in it beats with the light. the beat frequency that gets ouput is determined by where (left to right) the incoming beam hit.
of course the good news and the bad news is that this is intended for active remote sensing where one is illumunating a target with a single frequency laser. It does not work with ambient light (note the second articele referenced above will work with polychormatic light) . The good news is that the detection method is hetrodyne detection which has shot noise limited detection sensitivity even on a crappy photo detector. thus the system is capable of detecting a single photon of light.
Another cool feature is that one can do doppler detection with this too since any frequency shift in the target's reflected shifts the pixel frequency. This could be used for example the image bllod flowing in veins, find moving objects in noisy scenes (e.g. submarines, air planes) or any number of flow imaging concepts. The heterodyne detection means its sensitive enough to do at very long distances (say space), or to use it for imaging through very dense media (for example, imaging through the side of a vein or through breast or brain tissue.
A description of how it works in stilted patent language can be read on line here
From an artistic point of view, the problem isn't which medium to develop...it's how to improve both technologies such that cost/energy/latency is not too different. I should have the freedom to choose the technology which best serves the intent of the piece free from those constraints. It could be film, it could be video. It really depends on how I want it to turn out in the end.
So more substance, less rant: here's how I think these technologies would be useful to end users, and thus what we should be thinking about here.
Video Tap: A major video breakthrough in the feature film making process was Jerry Lewis's video tap. This puts a prism or split field diopter in between the lens and the film plane, splitting it in two, one going to the film plane, the other going to a video camera. This is how a director is able to get immediate feedback on how the scene went (instead of waiting for the dailies the next night to see it). A high framerate video tap for high framerate film would be extremely handy. The quality wouldn't have to be great, it would just need fidelity to tell the director and cinematographer how well composed the take was, and making sure all the stuff thats supposed to be in the take are there...and nothing else (like a boom mic).
Internet/NLE: This also would help in modern, internetworked digital non-linear processes. This is where takes are digitized as they are shot (if not already doing initial capture in DV) and dropped into the timeline in a nonlinear edit suite (avid, cinerella, final cut pro) whos project files are shared in an internetworked data store (film crews on other ends of the world, and the CG shop instantly are able to see their shot in the context of the other units shots...in realtime) via a 3 point edit. Even with a film process, the tap could digitize the footage and insert it into the timeline...the print of the footage could be later scanned and conformed to the timeline. Very handy. So this ties into the throughput problem. You have to consider that the bottleneck isn't CCD voltage intervals, cache tomfoolery or writing to a non-volatile medium. It could be a crappy ADSL connection or satellite uplink set up by people who scarcely understand how that stuff works.
Noise and heat: One of the banes of film making and one of the big advantages of digital video is the noise that all those ratchet/crank/shutter type mechanisms in a camera create. A lot of the sound work in a film is dealing with the noise from the camera. Sometimes, the sound is recorded later after discarding the sound from the set wholesale. Now, in order for a cmos imager to be effective at these speeds, we'll need to keep it cool. Heat is more likely to degrade throughput than buffer speed or size. Hence, we're going to need to build hardware to cool the cmos. That hardware is likely going to be more exotic than the cmos, take more energy than the motor for a high speed film device and potentially create a lot of noise on it's own. So the advantages of the high speed DV cam over film are only possible if the apparatus that supports the camera don't reintroduce the same problems on an equal or greater scale than existed in film.
Personally, I feel that the single greatest and most useful application of this technology, from a creative standpoint is the high speed video tap. It would liberate crews from the burden of dailies and integrate high speed footage into modern production processes.
For non-creative uses (scientific/research), this technology can free users from the latent and toxic nature of film processing infrastructure.
"Let him go, Ralph. He knows what he's doing." --Otto Mann (simpsons)
All image capture is analog. It can be electronic (CCDs) or chemical (film), but there's always an element that "charges up" as it's hit by photons.
Compared to film, CCDs are extremely low-res (top quality 35 mm film has resolution equivalent to a 50 megapixel CCD) but, more importantly, they're slow. At very short exposure times, CCDs have so much noise that the final result is useless. The problem isn't the transfer rate, it's the time the CCDs take to "charge up" to meaningful values.
There is one alternative: use very large CCDs. The larger the CCD, the more light hits it, and the faster it can charge. But larger CCDs are more expensive and require special lenses.
Recording directly to digital does have one big advantage: you don't have to pay for the film. But the CCDs simply aren't up to film quality yet (and probably won't be for another 5 years or so). So the solution is simple: shoot on film, then digitise it.
RMN
~~~
I can think of one good reason why this isnt a good idea. As still digital cameras push the limits of how many megapixles they can fit into an image some profesionals are noticing an interesting problem. at a certain point, adding more resolution to a camera actualy decreases the image quality. THat's because when you decrease the size of the sensor which records the data for a pixel in a camera you decrease the amount of light that will hit that sensor.
This means that the signal to noise ratio for each sensor goes up. At 3 megapixels you wont see any degredation from this but as the resolution increases youre going to see more and more loss of acuracy and less acurate images. now this is for a camera which takes photo's at the equivalent of 50 ISO (1/50th of a second) if you want a camera that takes images at 12000 images per second that means each image has to be captured in about 1/36000th of a second, or 720 times faster, or 1/720 of the light will hit each pixel. inorder to maintain acuracy at that speed youre going to have to drop down the resolution to 4000 pixels per image (100x40?) inorder to maintain image quality.
--aiee
The benefit of converting to this Y'CbCr colorspace is that you can get cheap, easy compression by simply subsampling the 2 color-difference channels, Cb and Cr, and humans will probably not notice.
Now lets assume that your "healthy budget" means "professional" and not "consumer"... That 1/4 chroma resolution implies a 4:1:1 or 4:2:0 sampling format which is fine for consumer level products like DV and DVDs, but any professional-level hardware will use at least 4:2:2, bringing you up to 16bpp.
Let's try that calculation again:
720(w)*480(h)*16(color)*12000(fps)= 66.4 gigabits per second.
Now for those of you who want to compress this monsterous stream with a beowulf cluster, I'd like you to show me one that can suck a 66.4 Gbps data stream out of a camera. :) right.
That leaves us with the only option of compressing in the camera hardware as the previous poster suggested with an array of encoders, each working on 1 GOP. Assuming we use realtime hardware encoders, we'll need about 401 of them: 12000(camera fps)/(30000/1001)(NTSC fps) = 400.4. I'd recommend setting the 401 encoders to produce 5Mbps MPEG-2 streams to achieve a decent quality. That gives us about 2 Gbps of MPEG-2 output.
BTW, to hold a 30-frame GOP in memory for each encoder while they encode, we'll need almost 8GB of RAM: 720(w)*480(h)*2(color)*8(GOP)*401(encoders) = 7.74GB (Let's try 8-frame GOPs for 2GB of RAM)
To store that 2Gbps video stream, we'll need a single SCSI Ultra320 hard drive. (not bad)
Now go build it! :)