Slashdot Mirror
Digital Video Capture and High Frame Rates?
Jeff asks: "So the folks at a place called Conniption Films (great name) developed a camera called the Millisecond Camera which can shoot 12,000 frames of film a second. I read the article and thought 'Hmm that's neat' but then realized they were still using an analog process for shooting this highspeed film. Being a geek, not necessarily into the film side of things but curious nonetheless, I wonder, shouldn't a computer be able to do a better job of such a thing? They say the film runs around a spindle going 500 mph (!). Wouldn't that be prone to failure and use alot of energy? Wouldn't it be more appropriate, easier, and overall cheaper to just hook up a high res CCD to a beowulf </duck> cluster of 2 ghz+ machines and capture high speed images that way? Why hasn't it been done yet? Or has it and I haven't seen it yet?" I did a double-take, when I first read this question, and then got curious and did a little digging. Turns out, high frame rates are not exclusive to the analog photography world, and to illustrate my point, I provide this link. It's woefully short on details, and the explanations as to why a camera that can record 1M frames per second is limited to a playback of only 103 frames, but the technology is out there. Has anyone seen any other digital cameras out there with high frame-rates? What visual mischief could you aspiring photographers get into with such a camera?
A. El Gammal, et al. published a 10,000fps imager with a 352x288 pixel resolution. This guy can maintain the full speed indefinately. Unfortunately is it not a commercial device, but something similar will probably be available within a few years.
Kleinfelder, S. SukHwan Lim Xinqiao Liu El Gamal, A. "A 10000 frames/s CMOS digital pixel sensor", Solid-State Circuits, IEEE Journal of. v38 n12, pp. 2049-2059. Feb. 2001.
The abstact is as follows:
A 352 x 288 pixel CMOS image sensor chip with per-pixel single-slope ADC and dynamic memory in a standard digital 0.18um CMOS process is described. The chip performs "snapshot" image acquisition, parallel 8-bit A/D conversion, and digital readout at continuous rate of 10000 frames/s or 1 Gpixels/s with power consumption of 50 mW. Each pixel consists of a photogate circuit, a three-stage comparator, and an 8-bit 3T dynamic memory comprising a total of 37 transistors in 9.4x9.4 um with a fill factor of 15%. The photogate quantum efficiency is 13.6%, and the sensor conversion gain is 13.1uV/e. At 1000 frames/s, measured integral nonlinearity is 0.22% over a 1-V range, rms temporal noise with digital CDS is 0.15%, and rms FPN with digital CDS is 0.027%. When operated at low frame rates, on-chip power management circuits permit complete powerdown between each frame conversion and readout. The digitized pixel data is read out over a 64-bit (8-pixel) wide bus operating at 167 MHz, i.e., over 1.33 GB/s. The chip is suitable for general high-speed imaging applications as well as for the implementation of several still and standard video rate applications that benefit from high-speed capture, such as dynamic range enhancement, motion estimation and compensation, and image stabilization.
"why the hell would anybody need 12000 frames per second. The human eye cant process all that plus nothing happens so fast you need 12000 shots of it in a second. This is just plain stupid and its a waste of film."
Because certian events, despite what you might think, *do* occur within 1000ths of a second. (The fireball from a nucelar blast for instance.)
Good cammera's shutter speeds tend to go up to 1/1000th of a second, and can go up to 1/8000th.
As far as the humman eye comment, well, just because you record at 12000 fps, doesn't mean you play it back at 12000 fps...
--Mike
"Not an actor, but he plays one on TV."
There's also a different sort of CCD highspeed camera that's used in various types of racing.
That system uses a single row of pixels which can be scanned at extremely high rates - the picture is built from objects moving in front of the pickup row, rather than the camera actually taking a full-resolution image.
Sort of a high-tech slit-camera.
Perhaps not 100% on-topic, but still interesting.
The other factor when talking about extreme high-speed photography (when people are calculating bandwidth):
Most really high-speed cameras shoot in black and white afaik.
If you drop the calculations from 32bpp down to 8bpp for a nic greyscale image, you're starting to get to manageable numbers... Also, adding cheap hardware based compression (RLE or the like), would be able to reduce the data stream to even more manageable levels.
You're not going to be able to shoot 6 megapixel pictures that fast, but 320x240 or 640x480 images should be possible at high framerates. I doubt it would replace film, but it might be handy for quick playback without having to get negs developed.
If you watch the "Bad Boys" DVD (the Will/Martin ver of Bad Boys), they have some very cool high-speed photography of different guns being fired into different objects. They used some sort of kodak high speed imager afaik - around 2000fps.
"Nothing strengthens authority so much as silence." - Charles de Gaulle
There have been many good replies to this thread, though most are talking theory, some experimental at best. A camera module with the ability to capture 12,000 high resolution frames per second is bound to cost a fortune, and I really doubt there will be much competition for a long time. Perhaps a cheaper alternative would be to purchase several currently-available high speed CCD/CMOS camera modules and use a series of mirrors and lenses to allow the cameras to work together in a round-robin fashion to achieve the a much higher framerate. This would certainly keep the project from being locked into proprietary hardware -- be it a single interface type, manufacturer, or other monopolistic attribute.
The idea of "parallel" items is nothing new, we've already seen success with drives, clusters, and even an array of projectors to create a high resolution projected wall.
Just a thought...
anyone else remember eg&g's high speed nuke cameras?
Rapatronic Camera Shots
Treatment, not tyranny. End the drug war and free our American POWs.
See my user info for links.
Let's do some arithmetic:
The wavelength of visible light (in a vacuum) is between 4x10^(-7) and 7x10^(-7) m.
The speed of light is 3x10^8 m/s (in a vacuum). Planck's constant is 6.6x10^(-34) J s.
Put these together, and a single photon of visible light has an energy of between 2.8x10^(-19) and 5x10^(-19) J.
Suppose you want to get 24-bit colour. As an absolute minimum, you'll want to be able to detect 4096 photons per colour per pixel per frame. CCDs are typically 50% efficient, which means you need 256*3*2 incoming photons per pixel. At, say, 1024x1024 pixels and a million frames per second, that means 3*4096*2*1024*1024*1000000 = 2.6x10^16 photons per second, at an average energy of 3.9x10^(-19) J each.
That's an absolute minimum of 1.0x10^(-2) W of incoming radiation.
How much light is available? Well, at "bright sunlight" is approximately 30 W/m^2 of visible light.
That means that you'd need an aperture roughly 28mm across... which isn't impossible, but is certainly not going to be desireable.
So how does ultra-fast photography work? They use really bright flashes of light... which is why you don't want to be filmed for more than a fraction of a second at once.
Tarsnap: Online backups for the truly paranoid
Yep I built a electonic video camera that had megarhertz frame rates 8 years ago. I patented it too. Actually two different designs.
C.E.M. Strauss, "Synthetic Array Heterodyne Detection: A single Element Detector Acts As An Array", Optics Letters, Vol 19, No. 20, 1609(1994)
and
B.J. Cooke, A.E. Galbraith, B.E. Laubscher, C.E.M. Strauss, N.L. Olivias, Grubler, A.W. laser field Imaging through fourier Transform Hetrodyne, proc of SPIE, 3707, 390-408, (1999)
the problem with pixelated detectors is reading out the darn pixels fast enough. Normally this is done by some sort of bucket brigade across the ccd or some sort of serial memory access across a cmos array. very slow. And parallel access to an entire ray is absurdly complicated and expensive
In my approach I solved this problem by multiplexing all of the pixel signals onto the same single wire. Each pixel when activated creates an osciliory signal at a unique frequency. All of these are combined on a single wire out put (amplified by a single fast amplifier) and then the AC signal is digitized by a single fast digitizer and streamed to a hard disk. The frame rate is determined by the frequency separation between the pixels, so if the oscillation frequency is a megahertz then a frame can be resolved every microsecond. This process is continuos and can go on for as long as you have disk space.
the other cool feature is that the chip you do this on is a single pixel chip! not a pixelated array. the pixels come from painting the chip with a rainbow of light. for a 1-D example, imagine red light on the left edge and blue light on the right. when a reference signal comes in it beats with the light. the beat frequency that gets ouput is determined by where (left to right) the incoming beam hit.
of course the good news and the bad news is that this is intended for active remote sensing where one is illumunating a target with a single frequency laser. It does not work with ambient light (note the second articele referenced above will work with polychormatic light) . The good news is that the detection method is hetrodyne detection which has shot noise limited detection sensitivity even on a crappy photo detector. thus the system is capable of detecting a single photon of light.
Another cool feature is that one can do doppler detection with this too since any frequency shift in the target's reflected shifts the pixel frequency. This could be used for example the image bllod flowing in veins, find moving objects in noisy scenes (e.g. submarines, air planes) or any number of flow imaging concepts. The heterodyne detection means its sensitive enough to do at very long distances (say space), or to use it for imaging through very dense media (for example, imaging through the side of a vein or through breast or brain tissue.
A description of how it works in stilted patent language can be read on line here
From an artistic point of view, the problem isn't which medium to develop...it's how to improve both technologies such that cost/energy/latency is not too different. I should have the freedom to choose the technology which best serves the intent of the piece free from those constraints. It could be film, it could be video. It really depends on how I want it to turn out in the end.
So more substance, less rant: here's how I think these technologies would be useful to end users, and thus what we should be thinking about here.
Video Tap: A major video breakthrough in the feature film making process was Jerry Lewis's video tap. This puts a prism or split field diopter in between the lens and the film plane, splitting it in two, one going to the film plane, the other going to a video camera. This is how a director is able to get immediate feedback on how the scene went (instead of waiting for the dailies the next night to see it). A high framerate video tap for high framerate film would be extremely handy. The quality wouldn't have to be great, it would just need fidelity to tell the director and cinematographer how well composed the take was, and making sure all the stuff thats supposed to be in the take are there...and nothing else (like a boom mic).
Internet/NLE: This also would help in modern, internetworked digital non-linear processes. This is where takes are digitized as they are shot (if not already doing initial capture in DV) and dropped into the timeline in a nonlinear edit suite (avid, cinerella, final cut pro) whos project files are shared in an internetworked data store (film crews on other ends of the world, and the CG shop instantly are able to see their shot in the context of the other units shots...in realtime) via a 3 point edit. Even with a film process, the tap could digitize the footage and insert it into the timeline...the print of the footage could be later scanned and conformed to the timeline. Very handy. So this ties into the throughput problem. You have to consider that the bottleneck isn't CCD voltage intervals, cache tomfoolery or writing to a non-volatile medium. It could be a crappy ADSL connection or satellite uplink set up by people who scarcely understand how that stuff works.
Noise and heat: One of the banes of film making and one of the big advantages of digital video is the noise that all those ratchet/crank/shutter type mechanisms in a camera create. A lot of the sound work in a film is dealing with the noise from the camera. Sometimes, the sound is recorded later after discarding the sound from the set wholesale. Now, in order for a cmos imager to be effective at these speeds, we'll need to keep it cool. Heat is more likely to degrade throughput than buffer speed or size. Hence, we're going to need to build hardware to cool the cmos. That hardware is likely going to be more exotic than the cmos, take more energy than the motor for a high speed film device and potentially create a lot of noise on it's own. So the advantages of the high speed DV cam over film are only possible if the apparatus that supports the camera don't reintroduce the same problems on an equal or greater scale than existed in film.
Personally, I feel that the single greatest and most useful application of this technology, from a creative standpoint is the high speed video tap. It would liberate crews from the burden of dailies and integrate high speed footage into modern production processes.
For non-creative uses (scientific/research), this technology can free users from the latent and toxic nature of film processing infrastructure.
"Let him go, Ralph. He knows what he's doing." --Otto Mann (simpsons)
All image capture is analog. It can be electronic (CCDs) or chemical (film), but there's always an element that "charges up" as it's hit by photons.
Compared to film, CCDs are extremely low-res (top quality 35 mm film has resolution equivalent to a 50 megapixel CCD) but, more importantly, they're slow. At very short exposure times, CCDs have so much noise that the final result is useless. The problem isn't the transfer rate, it's the time the CCDs take to "charge up" to meaningful values.
There is one alternative: use very large CCDs. The larger the CCD, the more light hits it, and the faster it can charge. But larger CCDs are more expensive and require special lenses.
Recording directly to digital does have one big advantage: you don't have to pay for the film. But the CCDs simply aren't up to film quality yet (and probably won't be for another 5 years or so). So the solution is simple: shoot on film, then digitise it.
RMN
~~~
I can think of one good reason why this isnt a good idea. As still digital cameras push the limits of how many megapixles they can fit into an image some profesionals are noticing an interesting problem. at a certain point, adding more resolution to a camera actualy decreases the image quality. THat's because when you decrease the size of the sensor which records the data for a pixel in a camera you decrease the amount of light that will hit that sensor.
This means that the signal to noise ratio for each sensor goes up. At 3 megapixels you wont see any degredation from this but as the resolution increases youre going to see more and more loss of acuracy and less acurate images. now this is for a camera which takes photo's at the equivalent of 50 ISO (1/50th of a second) if you want a camera that takes images at 12000 images per second that means each image has to be captured in about 1/36000th of a second, or 720 times faster, or 1/720 of the light will hit each pixel. inorder to maintain acuracy at that speed youre going to have to drop down the resolution to 4000 pixels per image (100x40?) inorder to maintain image quality.
--aiee