Slashdot Mirror
Digital Video Capture and High Frame Rates?
Jeff asks: "So the folks at a place called Conniption Films (great name) developed a camera called the Millisecond Camera which can shoot 12,000 frames of film a second. I read the article and thought 'Hmm that's neat' but then realized they were still using an analog process for shooting this highspeed film. Being a geek, not necessarily into the film side of things but curious nonetheless, I wonder, shouldn't a computer be able to do a better job of such a thing? They say the film runs around a spindle going 500 mph (!). Wouldn't that be prone to failure and use alot of energy? Wouldn't it be more appropriate, easier, and overall cheaper to just hook up a high res CCD to a beowulf </duck> cluster of 2 ghz+ machines and capture high speed images that way? Why hasn't it been done yet? Or has it and I haven't seen it yet?" I did a double-take, when I first read this question, and then got curious and did a little digging. Turns out, high frame rates are not exclusive to the analog photography world, and to illustrate my point, I provide this link. It's woefully short on details, and the explanations as to why a camera that can record 1M frames per second is limited to a playback of only 103 frames, but the technology is out there. Has anyone seen any other digital cameras out there with high frame-rates? What visual mischief could you aspiring photographers get into with such a camera?
The explanation as to why it can only play back 103 frames is QUITE clear... the chip has 103 "on-chip" memory buffers per sensor, and they get cyclicly overwritten with the last 103 frames.
This overcomes the bottleneck of trying to transfer data off the CCD at such high frame rates in real time, but limits you to "downloading" the last 103 frames after-the fact from the chip.
MadCow.
I used to have a sig, but I set it free and it never came back.
A quick calculation on the bandwidth of capturing 12000 SVGA-resolution full color frames per second:
1024 (width) * 768 (height) * 4 (32-bit color) * 12000 (fps) = 377,487,360,00 bytes/second (35 Gbytes/s)
So no wonder they use film...
CCD simply needs a few milliseconds to regain their 0-volt signal level again before they can emit a new pulse. This recoverytime makes it unsuitable for high speed filming. Helas.
also consider that most of the time, people that are interested in such frame rates, are also *very* interested in having detailed high-resolution frames of the event at 'interesting' times.
This probably means having to shoot images of around 4-6 megapixels, and I really don't see any way of doing that at the speed needed for this kind of application.
The only way might be exactly what the poster of the topic didn't grasp: have a camera that can take 100-1000 pictures at a 1Mpics/sec frame rate and store them in ultra-fast local memory, and transfer them out at leisure, with a good triggering setup, 100-1000 microseconds worth of data might just be enough for certain applications.
-- the cake is a lie
At 640x240x24, you're talking 7372800 bits per frame. At 1000 frames/second, we'd need to be transferring 7Gbps. That would be a bit hard to handle. You could cut the rate by dropping colors. At B/W, it'd be pretty manageable, but that's probably not what you want. You probably also want higher resolution. No matter what, you wouldn't be able to swallow the stream for long.
Oh, and by the way. The confusion about a million frames/second versus 103 was just poor word choice in the article. What they mean is a 1 microsecond shutter speed - 1 microsecond frames with 9707 microsecond gaps. Great stop-action to cut blurring, but manageable transfer rates.
A. El Gammal, et al. published a 10,000fps imager with a 352x288 pixel resolution. This guy can maintain the full speed indefinately. Unfortunately is it not a commercial device, but something similar will probably be available within a few years.
Kleinfelder, S. SukHwan Lim Xinqiao Liu El Gamal, A. "A 10000 frames/s CMOS digital pixel sensor", Solid-State Circuits, IEEE Journal of. v38 n12, pp. 2049-2059. Feb. 2001.
The abstact is as follows:
A 352 x 288 pixel CMOS image sensor chip with per-pixel single-slope ADC and dynamic memory in a standard digital 0.18um CMOS process is described. The chip performs "snapshot" image acquisition, parallel 8-bit A/D conversion, and digital readout at continuous rate of 10000 frames/s or 1 Gpixels/s with power consumption of 50 mW. Each pixel consists of a photogate circuit, a three-stage comparator, and an 8-bit 3T dynamic memory comprising a total of 37 transistors in 9.4x9.4 um with a fill factor of 15%. The photogate quantum efficiency is 13.6%, and the sensor conversion gain is 13.1uV/e. At 1000 frames/s, measured integral nonlinearity is 0.22% over a 1-V range, rms temporal noise with digital CDS is 0.15%, and rms FPN with digital CDS is 0.027%. When operated at low frame rates, on-chip power management circuits permit complete powerdown between each frame conversion and readout. The digitized pixel data is read out over a 64-bit (8-pixel) wide bus operating at 167 MHz, i.e., over 1.33 GB/s. The chip is suitable for general high-speed imaging applications as well as for the implementation of several still and standard video rate applications that benefit from high-speed capture, such as dynamic range enhancement, motion estimation and compensation, and image stabilization.
There are a lot of limits when it comes to cameras connected to PC's. I've worked in a lab where we used cameras that generated 640x480x4 (32 bit color) frames at 60 Hz. Guess what. You can't even buy a HD that can sustain that kind of transfer rate for any period of time. Good thing those computers had about a gig of ram each ;-)
There is actually a few limitations. Bandwidth is the most important one. Here you're looking at the connection between camera and the computer. We used special frame grabber boards, fireware or USB - well... nothing that I know of can handle 12,000 Hz. Next, somewhat smaller limitation is the bandwidth to memory. When you're talking about 12,000 Hz - that will become a factor. And of course - unless you've got about 40 Gigs of Ram (at least) you would want to save the stream. There are Video Vaults which are basically raid arrays, but again - they can't handle this kind of data stream.
Technology is coming along though. The new CMOS based cameras can have fairly high frame rates. You can actually select between resolution and framerate. Last time I checked the fastest they could go was about 500fps (at low resolution), the limit being again the link between the camera and PC. I believe the theoretical limit of the CMOS type camera is at either 5000 or 8000 fps (I don't really remember which - sorry).
"why the hell would anybody need 12000 frames per second. The human eye cant process all that plus nothing happens so fast you need 12000 shots of it in a second. This is just plain stupid and its a waste of film."
Because certian events, despite what you might think, *do* occur within 1000ths of a second. (The fireball from a nucelar blast for instance.)
Good cammera's shutter speeds tend to go up to 1/1000th of a second, and can go up to 1/8000th.
As far as the humman eye comment, well, just because you record at 12000 fps, doesn't mean you play it back at 12000 fps...
On the pure digital front, there are units that can record 1000 FPS continuous at 512 x 512 pixels. The system is data-rate limited. The imager can go much faster; if you cut the image size down to 32x128 pixels, you can get 32K frames/sec. At 128 x 128, you can get 11.2K frames/sec. The data goes into a buffer in the control unit (1 GB, typically), and is read out via FireWire. So this system can take a lot more frames than the device described in the article, which stores the images in memory within the imager and can only store 100 images or so.
--Mike
"Not an actor, but he plays one on TV."
Shouldn't it be ?
You can get around any bandwidth issues with a sufficiently large amount of cabling. The whole idea of doing this in parellel implies that. Anyway, compare the bandwidth of digital photography with the physical bandwidth of looping film through an eyepiece at 12,000 frames per second and you come up with a very different problem -- you've got to use TINY film, with an effective resolution much lower than what some of you linux numbercrunchers are assuming. "SVGA resolutions?" Think more like 320x240 -- and don't expect more than a few seconds per cannister, high costs, etc.
.000083 s. With low light, you need extremely sensitive equipment to even detect it and even more sensitive equipment to detect the subtle variations in wavelength that make up colors. Today's CCD cameras are very slow to register intensity light -- much slower than film. The chemical reaction in film triggered by exposure can be controlled much better, simply by changing the tolerance of the film -- which is why your high end, high speed shutter digital cameras are so godawful expensive. The $2500 Canon I've been looking at has roughly the same shutter speed as an equivalent $300 film camera. The extra price is NOT a "coolness" tax...it's for the set of three extremely high res CCD sensors and the chips capable of processing their information at that speed. My film prof used to say "digital ain't digital"...there's a quality factor of all digital electronics that can be poiled down to the quality of interpolation, quality of the ADC and of transistors leading up to it.
No, the problem is light itself. You don't get much of it captured with a shutter speed of
CCD kind of sucks, man. For all its glorious promise, the best CCD chipsets aren't all that much better than the wonderful X-10 spycam.
Hey freaks: now you're ju
Because certian events, despite what you might
think, *do* occur within 1000ths of a second
Yes, like when I get fragged playing quake.
Michael
There is no cryptographic solution to the problem where the intended receiver and the attacker are the same entity.
There's also a different sort of CCD highspeed camera that's used in various types of racing.
That system uses a single row of pixels which can be scanned at extremely high rates - the picture is built from objects moving in front of the pickup row, rather than the camera actually taking a full-resolution image.
Sort of a high-tech slit-camera.
Perhaps not 100% on-topic, but still interesting.
The other factor when talking about extreme high-speed photography (when people are calculating bandwidth):
Most really high-speed cameras shoot in black and white afaik.
If you drop the calculations from 32bpp down to 8bpp for a nic greyscale image, you're starting to get to manageable numbers... Also, adding cheap hardware based compression (RLE or the like), would be able to reduce the data stream to even more manageable levels.
You're not going to be able to shoot 6 megapixel pictures that fast, but 320x240 or 640x480 images should be possible at high framerates. I doubt it would replace film, but it might be handy for quick playback without having to get negs developed.
If you watch the "Bad Boys" DVD (the Will/Martin ver of Bad Boys), they have some very cool high-speed photography of different guns being fired into different objects. They used some sort of kodak high speed imager afaik - around 2000fps.
"Nothing strengthens authority so much as silence." - Charles de Gaulle
There have been many good replies to this thread, though most are talking theory, some experimental at best. A camera module with the ability to capture 12,000 high resolution frames per second is bound to cost a fortune, and I really doubt there will be much competition for a long time. Perhaps a cheaper alternative would be to purchase several currently-available high speed CCD/CMOS camera modules and use a series of mirrors and lenses to allow the cameras to work together in a round-robin fashion to achieve the a much higher framerate. This would certainly keep the project from being locked into proprietary hardware -- be it a single interface type, manufacturer, or other monopolistic attribute.
The idea of "parallel" items is nothing new, we've already seen success with drives, clusters, and even an array of projectors to create a high resolution projected wall.
Just a thought...
anyone else remember eg&g's high speed nuke cameras?
Rapatronic Camera Shots
Treatment, not tyranny. End the drug war and free our American POWs.
See my user info for links.
Look at it this way, 12000 / 30 / 60 = 6 2/3, so it would take over 6 1/2 minutes to watch 1 second of video at regular framerate. The events of interest here are likely to be much less than one second. You could fit 0.1 seconds of video into a 32-bit address space. 0.1 seconds doesn't sound like alot, but it's way more than enough time to watch a bullet pierce a playing card.
Let's do some arithmetic:
The wavelength of visible light (in a vacuum) is between 4x10^(-7) and 7x10^(-7) m.
The speed of light is 3x10^8 m/s (in a vacuum). Planck's constant is 6.6x10^(-34) J s.
Put these together, and a single photon of visible light has an energy of between 2.8x10^(-19) and 5x10^(-19) J.
Suppose you want to get 24-bit colour. As an absolute minimum, you'll want to be able to detect 4096 photons per colour per pixel per frame. CCDs are typically 50% efficient, which means you need 256*3*2 incoming photons per pixel. At, say, 1024x1024 pixels and a million frames per second, that means 3*4096*2*1024*1024*1000000 = 2.6x10^16 photons per second, at an average energy of 3.9x10^(-19) J each.
That's an absolute minimum of 1.0x10^(-2) W of incoming radiation.
How much light is available? Well, at "bright sunlight" is approximately 30 W/m^2 of visible light.
That means that you'd need an aperture roughly 28mm across... which isn't impossible, but is certainly not going to be desireable.
So how does ultra-fast photography work? They use really bright flashes of light... which is why you don't want to be filmed for more than a fraction of a second at once.
Tarsnap: Online backups for the truly paranoid
You're right in the last part of your analysis: it's the latency that kills CCD detectors. The light sensitivity (i.e., the quantum efficiency) of CCDs is definitely not the problem, though: CCDs are as close to ideal photodetectors as you can get. They capture virtually 100% of the light that falls on them (their QE is >90%: film is usually quoted at 20%).
:)
This is actually a point in favor of high speed CCDs : in order to get the same level of contrast, you need about 5 times less light than a normal high-speed camera. Remember that the same argument you made for light sensitivity/light levels also applies to film. They'd need a light source 4000 times brighter as well, as the film is only exposed for a small fraction of the time.
You might be able to do something cool that mixes film and CCDs: have a film made of CCDs that are then read out after being exposed to light. This solves the bandwidth problem as well, because you could have multiple systems reading out the data from multiple CCDs - it's not hard to aggregate GB/s worth of bandwidth from slower sources. The main problem, of course, would be flexible silicon. That'd take some work.
This is the same problem that film would have, though. You can't get sub-photon resolution of light, because it doesn't exist.
For high speed photography, you need lots of light. This is just generically true. But the quantum efficiency of CCDs is virtually 100%, as opposed to film which is much lower. So, this is a strong point in favor of a CCD system - you'd need less light. There's a poster above talking about how in car crash tests they need massive lighting systems to be able to see things. This'd cut down on their power bill quite a lot.
Another option is to wait for 10gigE (along with the rest of the supercomputing world) or go with Myrinet, which has recently broken the 1 gigabit barrier.
Recently? Myrinet has been doing 2 gigabits full-duplex since May 2001 when it started using fiber. Not to mention that full link utilization only uses a few percent of the host CPU. What's the point of fast cluster interconnect when you use half your CPU sending packets through the TCP/IP stack?
Well, For one thing nobody records at that resolution. As another reply stated, DV is 720x480.
Another problem with your simple calculation is that video is never stored as 32-bit color. That's totally unrealistic. The common way to store video is not RGB, but YUV. Because of the way the human visual system works, the color components (U,V) are typically stored at 1/4 the resolution of the luminance (Y), meaning that an 3*X pixel RGB image would be stored as a X+X/4+X/4=1.5X image in YUV, half the number of pixels.
More significant, though, is that fact that just about every digital image recording mechanism stores information compressed onto the storage media. This is true from consumer digital cameras to DV cameras to the Sony HDTV cameras Lucas used for Star Wars.
Consider what it means to take 12,000 frames per second. You're probably recording a single nearly-instantaneous event, or getting many images of a very fast event. In the former case, there will be a series of frames before the event in which nothing is going on, and the difference between the frames is close to zero, which compresses extremeley well with MPEG-style compression. Your data rate could be 1/100th of the uncompressed rate. When the event occurs, the instantaneous data rate goes up, but buffering can solve this, since it probably lasts a few frames.
In the latter case, recording a fast event at a fast framerate, is essentially the same as recording a normal-speed event at normal frame rates. In this domain as well MPEG-style compression is extremeley effective. At the maximum you would need 1/5th or 1/10th the uncompressed rate, but 1/100th is a pretty reasonable number given current technology.
The only challenge with realtime compression at this speed, of course, is sufficiently fast hardware. I think it could be done in parallel -- capture several GOPs worth of data (15-45 frames perhaps) and send it to a compressor, and then switch the buffer output to a new compressor, round-robin style.
In any case, video is usually stored at rates many factors smaller than the uncompressed rate. So if you change the variables of your equation to a more realistic resolution and color depth, then divide that number by 10 or 100, you'll have a more realistic data rate.
720(w)*480(h)*1.5(color)*12000(fps)= 6.2GB/s, divide by 100 for agressive compression but reasonable results = 600MB/s
Still too fast, but not completely unrealistic if you've got a healthy budget. ;-)
Yep I built a electonic video camera that had megarhertz frame rates 8 years ago. I patented it too. Actually two different designs.
C.E.M. Strauss, "Synthetic Array Heterodyne Detection: A single Element Detector Acts As An Array", Optics Letters, Vol 19, No. 20, 1609(1994)
and
B.J. Cooke, A.E. Galbraith, B.E. Laubscher, C.E.M. Strauss, N.L. Olivias, Grubler, A.W. laser field Imaging through fourier Transform Hetrodyne, proc of SPIE, 3707, 390-408, (1999)
the problem with pixelated detectors is reading out the darn pixels fast enough. Normally this is done by some sort of bucket brigade across the ccd or some sort of serial memory access across a cmos array. very slow. And parallel access to an entire ray is absurdly complicated and expensive
In my approach I solved this problem by multiplexing all of the pixel signals onto the same single wire. Each pixel when activated creates an osciliory signal at a unique frequency. All of these are combined on a single wire out put (amplified by a single fast amplifier) and then the AC signal is digitized by a single fast digitizer and streamed to a hard disk. The frame rate is determined by the frequency separation between the pixels, so if the oscillation frequency is a megahertz then a frame can be resolved every microsecond. This process is continuos and can go on for as long as you have disk space.
the other cool feature is that the chip you do this on is a single pixel chip! not a pixelated array. the pixels come from painting the chip with a rainbow of light. for a 1-D example, imagine red light on the left edge and blue light on the right. when a reference signal comes in it beats with the light. the beat frequency that gets ouput is determined by where (left to right) the incoming beam hit.
of course the good news and the bad news is that this is intended for active remote sensing where one is illumunating a target with a single frequency laser. It does not work with ambient light (note the second articele referenced above will work with polychormatic light) . The good news is that the detection method is hetrodyne detection which has shot noise limited detection sensitivity even on a crappy photo detector. thus the system is capable of detecting a single photon of light.
Another cool feature is that one can do doppler detection with this too since any frequency shift in the target's reflected shifts the pixel frequency. This could be used for example the image bllod flowing in veins, find moving objects in noisy scenes (e.g. submarines, air planes) or any number of flow imaging concepts. The heterodyne detection means its sensitive enough to do at very long distances (say space), or to use it for imaging through very dense media (for example, imaging through the side of a vein or through breast or brain tissue.
A description of how it works in stilted patent language can be read on line here
From an artistic point of view, the problem isn't which medium to develop...it's how to improve both technologies such that cost/energy/latency is not too different. I should have the freedom to choose the technology which best serves the intent of the piece free from those constraints. It could be film, it could be video. It really depends on how I want it to turn out in the end.
So more substance, less rant: here's how I think these technologies would be useful to end users, and thus what we should be thinking about here.
Video Tap: A major video breakthrough in the feature film making process was Jerry Lewis's video tap. This puts a prism or split field diopter in between the lens and the film plane, splitting it in two, one going to the film plane, the other going to a video camera. This is how a director is able to get immediate feedback on how the scene went (instead of waiting for the dailies the next night to see it). A high framerate video tap for high framerate film would be extremely handy. The quality wouldn't have to be great, it would just need fidelity to tell the director and cinematographer how well composed the take was, and making sure all the stuff thats supposed to be in the take are there...and nothing else (like a boom mic).
Internet/NLE: This also would help in modern, internetworked digital non-linear processes. This is where takes are digitized as they are shot (if not already doing initial capture in DV) and dropped into the timeline in a nonlinear edit suite (avid, cinerella, final cut pro) whos project files are shared in an internetworked data store (film crews on other ends of the world, and the CG shop instantly are able to see their shot in the context of the other units shots...in realtime) via a 3 point edit. Even with a film process, the tap could digitize the footage and insert it into the timeline...the print of the footage could be later scanned and conformed to the timeline. Very handy. So this ties into the throughput problem. You have to consider that the bottleneck isn't CCD voltage intervals, cache tomfoolery or writing to a non-volatile medium. It could be a crappy ADSL connection or satellite uplink set up by people who scarcely understand how that stuff works.
Noise and heat: One of the banes of film making and one of the big advantages of digital video is the noise that all those ratchet/crank/shutter type mechanisms in a camera create. A lot of the sound work in a film is dealing with the noise from the camera. Sometimes, the sound is recorded later after discarding the sound from the set wholesale. Now, in order for a cmos imager to be effective at these speeds, we'll need to keep it cool. Heat is more likely to degrade throughput than buffer speed or size. Hence, we're going to need to build hardware to cool the cmos. That hardware is likely going to be more exotic than the cmos, take more energy than the motor for a high speed film device and potentially create a lot of noise on it's own. So the advantages of the high speed DV cam over film are only possible if the apparatus that supports the camera don't reintroduce the same problems on an equal or greater scale than existed in film.
Personally, I feel that the single greatest and most useful application of this technology, from a creative standpoint is the high speed video tap. It would liberate crews from the burden of dailies and integrate high speed footage into modern production processes.
For non-creative uses (scientific/research), this technology can free users from the latent and toxic nature of film processing infrastructure.
"Let him go, Ralph. He knows what he's doing." --Otto Mann (simpsons)
All image capture is analog. It can be electronic (CCDs) or chemical (film), but there's always an element that "charges up" as it's hit by photons.
Compared to film, CCDs are extremely low-res (top quality 35 mm film has resolution equivalent to a 50 megapixel CCD) but, more importantly, they're slow. At very short exposure times, CCDs have so much noise that the final result is useless. The problem isn't the transfer rate, it's the time the CCDs take to "charge up" to meaningful values.
There is one alternative: use very large CCDs. The larger the CCD, the more light hits it, and the faster it can charge. But larger CCDs are more expensive and require special lenses.
Recording directly to digital does have one big advantage: you don't have to pay for the film. But the CCDs simply aren't up to film quality yet (and probably won't be for another 5 years or so). So the solution is simple: shoot on film, then digitise it.
RMN
~~~
I can think of one good reason why this isnt a good idea. As still digital cameras push the limits of how many megapixles they can fit into an image some profesionals are noticing an interesting problem. at a certain point, adding more resolution to a camera actualy decreases the image quality. THat's because when you decrease the size of the sensor which records the data for a pixel in a camera you decrease the amount of light that will hit that sensor.
This means that the signal to noise ratio for each sensor goes up. At 3 megapixels you wont see any degredation from this but as the resolution increases youre going to see more and more loss of acuracy and less acurate images. now this is for a camera which takes photo's at the equivalent of 50 ISO (1/50th of a second) if you want a camera that takes images at 12000 images per second that means each image has to be captured in about 1/36000th of a second, or 720 times faster, or 1/720 of the light will hit each pixel. inorder to maintain acuracy at that speed youre going to have to drop down the resolution to 4000 pixels per image (100x40?) inorder to maintain image quality.
--aiee
You were probably correct a few years ago, but you need to get with the times.
There is no film in the world that can outshoot a high-sensitivity CCD nowadays. Cameras like the Kodak 760x can shoot at ISO 6400 with reasonable quality, which film is utterly incapable of matching with any sort of quality, and CCDs are only getting better.
Yes, crappy consumer digicams suck at anything over ISO 100. But a serious professional digicam beats the pants off of film at high ISOs. (In case you were wondering, my wife is a professional photographer who shoots with a Nikon D1X. I do know a bit about this.)
ZFS: because love is never having to say fsck
Mechanically inclined.
When was the last time you even heard that phrase? We live in a physical world. A mechanical device is a perfectly acceptable solution to a problem. Not everything needs to be done with software. Just look at the guy's level of disappointment. "But there has to be a way to do this with electronics! Electronics are always better than mechanics, aren't they? It's impossible for mechanics to do something electronics cannot, ins't it? Hello?"
And Cliff's additional writeup is no help either. The reason the video in the example he found can only played back at 103fps is fully explained in the link he provides (and apparently didn't bother to read). Also, the 12,000fps film camera that got everyone talking in the first place not the first of its kind. High-speed film cameras have been around for decades. The real kicker is Cliff's silly statement at the end, which makes it sound as though an electronic high-speed camera would be the first high-speed camera ever. He says, "What visual mischief could you aspiring photographers get into with such a camera?" Gee, I dunno Cliff, how about the exact same things people have been doing with high speed film cameras for the past 50 years, eh?
Sheesh. The world goes beyond the bits in a CPU. Turn off the computer and take a look around at the tangible, physical world.
Free Hans!