High-Speed Video Free With High-Def Photography

bugzappy notes a development out of the University of Oxford, where scientists have developed a technology capable of capturing a high-resolution still image alongside very high-speed video. The researchers started out trying to capture images of biological processes, such as the behavior of heart tissue under various circumstances. They combined off-the-shelf technologies found in standard cameras and digital movie projectors. What's new is that the picture and the video are captured at the same time on the same sensor. This is done by allowing the camera's pixels to act as if they were part of tens, or even hundreds, of individual cameras taking pictures in rapid succession during a single normal exposure. The trick is that the pattern of pixel exposures keeps the high-resolution content of the overall image, which can then be used as-is, to form a regular high-res picture, or be decoded into a high-speed movie. The research is detailed in the journal Nature Methods (abstract only without subscription).

Comments

How long

[Anonymous Coward]

How long before it is used for porn?

I read the title as "High-Def Pornography"...

[HouseOfMisterE]

I think it's past my bedtime.

interlacing

[Anonymous Coward]

Sounds like they have a high resolution image sensor but the timing of the data samples from certain groups of pixels is staggered. Sort of like how one frame of interlaced NTSC DVD video can represent a single "high resolution" 720x480 image, or a series of two 720x240 images 1/60th second apart.

Re:interlacing

[MrNaz]

Yea that's the first thing I thought as well; the principle is similar to video interlacing from back in the day, except that this is more sophisticated, and could conceivably be used to capture extremely high definition, extremely high framerate footage.

If you apply this technology to high grade 50mpix Hasselblad sensors, you could conceivably acheive frame rates of thousands of frames per second in 2k or even 4k resolution using gear that costs under $100k. Currently, that sort of photography is limited to national science bodies and multi-million dollar budgets. Being able to do that sort of thing for under 6 figures would open up HUGE research possibilities for university science labs and other relatively fund-poor institutions.

Re:interlacing

[Rockoon]

I think you've missed the point.

Use the high-frame-rate camera to take a high-frame-rate video, or use it to take a high resolution picture, but you cant take a high-frame-rate high-resolution video.

The idea is that the light sensitive components have a minimum response time that is too large to capture high frame-rate digital data without tricks. So engineers being what they are use seperate groups of them with staggered capture times in order to achieve high frame-rates. In the simplest case there would only be two groups of senors, probably would be called odd and even, which would allow double the frame-rate of that minimum response time.

What these blokes have noted is that the groups of sensors which capture a single frame are stippled across the capture device, so if the capture times were not staggered then the effective resolution is higher. Essentially they are un-staggering the capture times post-capture in order to achieve that high resolution, meaning that you cannot have both at the time time.

The most they can save appears to be 50%, the cost of a regular high resolution capture device which they didnt get with the their high-frame-rate device purchase.

Re:interlacing

[infolation]

The technology they're using, which can derive high resolution frames by comparing several successive frames, or analyzing the rolling shutter effect of CMOS cameras is actually already well established in film visual effects.

Visual effects technology company 'The Foundry' have done quite a lot of research into this area already.

Their Furnace F_SmartZoom tool uses motion estimation techniques to analyse successive film frames to derive single frames of higher resolution than any one of the moving frames. And their Rolling Shutter tool uses local motion estimation algorthithms to analyze the staggered frames output by CMOS cameras to reconstruct them into complete un-staggered frames.

It's very interesting that the scientists in Oxford are exploiting this side effect of CMOS cameras by combining both these technologies to derive high resolution, un-blurred frames from multiple CMOS images.

As a side-note, District 9 was shot on the Red camera (a CMOS camera that exhibits this rolling shutter efffect), and a lot of Image Engine's post-production work that film required this sort of analysis so that staggered frames could be reconstructed to enable 3-D motion tracking for the insertion of CG into live action plates.

Re:interlacing

[Ihmhi]

Hasselblad sensors

There's seriously something called a Hasselblad sensor? That is fucking awesome. That sounds like something off of Babylon 5. "Incoming enemy fighters on the Hasselblad sensors!"

Re:interlacing

[scdeimos]

The idea is that the light sensitive components have a minimum response time that is too large to capture high frame-rate digital data without tricks.

It's not actually a minimum response time issue, at least not from a CCD sensor point of view (as opposed to CMOS sensors you tend to see in consumer-level digital video and photography products).

"Traditional" high-speed photography with CCD sensors usually works by lighting the scene with high-intensity light sources so that the sensors are able to gather enough photons within the short exposure times to be "useful." Have a look around GooTube for things like the "SawStop demo" on the Discovery Time Wrap program for a good example of this.

If you look at a single pixel element on a CCD sensor it's essentially a photon well - it receives photons from the environment and converts them to an electric charge. Assuming the electronics reading charges out of the CCD sensor are good enough, a single photon striking a pixel element would be detectable, thus it's not really a pixel-related minimum response time issue.

The conventional electronics used in the "read out " process of a CCD sensor essentially do the following: they enable a "row" of pixel elements and clock the electric charges across the "columns" by using something akin to a bucket bridge network. The charge from the column getting clocked off the side of the sensor is read by an ADC (analogue-digital convertor) and stored in a digital buffer (RAM) before being sent to the host device. Each row is "clocked out" and read in this fashion, then the whole CCD sensor is shorted to reset any residual charges ready for the next exposure. Any response time issues are in the clocking out process, since the weakest link in the chain will be the time needed by the ADC to capture and convert a single charge.

The proposed technique changes the read out process in several ways, vastly increasing the complexity of the CCD sensor's bucket bridge network and reset electronics in the process. Say, for example, the sensor is setup as an array of 2x2 elements (the article proposes 4x4 elements). The read out process needs to read out pixels in four phases: even columns on even rows, odd columns on even rows, even columns on odd rows, odd rows on odd rows. That sounds complex already, right, but it's worse: because the sensor will essentially be exposed continuously you also need to reset the charges in those groups individually otherwise you'll get residual charge build-up that skews the data over time. If you don't all pixel elements will eventually read as full charges.

Electronics complexity issues aside, I'm wondering how useful this technique will be for high-speed scientific research. When looking at the resultant high-speed video each frame will be offset slightly in both the horizontal and vertical directions (1/2 pixel in a 2x2 network, 1/4 pixel in a 4x4 network). To some degree this will able to be corrected using sub-pixel blending, but this will introduce errors into the frames thus reducing their utility. Nonetheless, it sounds like a very interesting technique.

Representative sample

[LordLucless]

As I read this, there are three comments. Two are about porn. Slashdot in a nutshell.

Re:Representative sample

[Anonymous Coward]

As I read this, there are three comments. Two are about porn. Slashdot in a nutshell.

Actually, only one of those three was about porn. The other two (and this one) are just offtopic. So that makes Slashdot 25% horny, 25% pedantic, and 75%-100% offtopic.

I've actually thought about this...

[pushing-robot]

...and how eventually cameras will not have a "shutter" as we know it but will simply keep track of how each pixel was illuminated at each moment in time. Of course, shutterless sensors are already in widespread use; we call them "eyes", and they have the same benefits that TFA describes: Your brain can observe low-detail fast-moving objects and high-detail static objects at the same time without having to reconfigure anything. Consequentially, shutterless cameras would have the side benefit of better approximating biological vision.

The ultimate dream would be a truly holographic sensor that records exactly where, when, and at what angle each photon hit the sensor, so that the zoom, exposure time, and focus can be changed in post-processing (as well as a lot of other cool stuff).

Re:I've actually thought about this...

[derGoldstein]

This entire field can easily be extrapolated. First, the shutter is a mechanical components that isn't required -- every portable computer has video camera that can take still images. The reason we still have shutters in high-end cameras is because of the way sensors are currently designed, and the fact that modern DSLRs are basically upgraded film SLRs.
And what about the lens? If the sensors are omnidirectional and can simply keep reporting their state at a high frequency, the "lens" (its optical purpose) can be done in software. You just need a high density of sensors and the ability to process the information fast enough.
Obviously, the individual sensors can't be truly omnidirectional, but rather their visibility angle would depend on the geometry of the surface they're placed on -- which could be a hemisphere, or even an almost complete sphere. As you mentioned, the angle of light would still be relevant, but this would be done on an individual sensor basis -- rather than one lens orchestrating the entire image.

There, we solved it. Engineers, get to work!

Re:I've actually thought about this...

[Anonymous Coward]

It depends. In good lighting you don't need to register all photons. However in a dark room or for watching night sky each photon counts. Here is an informative article: http://math.ucr.edu/home/baez/physics/Quantum/see_a_photon.html
Human eye can actually register flash of about 90 photons (10% of them will reach the retina, so about 9 photons is enough to activate receptors). The sensitivity also depends on the wavelength.

Re:I've actually thought about this...

[EdZ]

It's of massive value in astronomy. And it's exactly whatsuperconducting image sensors do.

Re:I've actually thought about this...

[gillbates]

The overwhelming majority of digital cameras do not have a shutter. You do realize that clicking sound comes not from a shutter, but from a small speaker, right?

I'm honestly sorry I didn't patent this technique back in 2005 when I was working with digital image sensors, but suffice to say, it's been known about and used in industry for quite some time. Engineers have always known there was a tradeoff between the image resolution and frame rate, and this appears a rather obvious compromise. An image sensor chip has a limited bandwidth for reading out pixels, so naturally the framerate is a factor of the image pixel count.

Most image sensors can be reconfigured rather quickly, perhaps even between frames. This technique is hardly worth a patent, as it's obvious to anyone who's ever had to make a tradeoff between frame rate and light sensitivity, or frame rate and resolution. For video, there's the standard D1 resolution of 720 by 480. For stills, the whole resolution of the sensor is used. So obvious that it is hard to consider it novel enough to patent.

Re:I've actually thought about this...

[dunkelfalke]

A high resolution optical sensor delivers a shitload of data - 20 and more megabytes for every frame. The processing of the data from the Bayer matrix (we won't take the Foveon into account for the sake of the argument) and resizing also takes time. You need at least 60 fps to get rid of lag while moving. Have fun at processing 1.2 terabytes per second.

Some stills cameras do too, but....

[N Monkey]

There are already shutterless cameras. They're called video cameras...

Some stills cameras, e.g. on phones, are shutterless as well, but often have some interesting artefacts.

In this case it is probably due to the high level of correlation between pixel position and "shutter" time. I'm guessing that, in the paper, (judging only by the abstract) they are using a pseudo-random pattern for the pixel sampling which would trade these weird effects for 'noise' which would be less obvious.