Light Field Photography Is the New Path To 3-D

waderoush writes "In November, Lytro, the maker of the first light field camera for consumers, upgraded its viewer software to enable a feature called 'Perspective Shift.' In addition to refocusing pictures after they've been taken, Lytro audiences can now pivot between different virtual points of view, within a narrow baseline. This 3-D capability was baked into Lytro's technology from the start: 'The light field itself is inherently multidimensional [and] the 2-D refocusable picture that we launched with was just one way to represent that,' says Eric Cheng, Lytro's director of photography. But while Perspective Shift is currently little more than a novelty, the possibilities for future 3-D imaging are startling, especially as Lytro develops future devices with larger sensors — and therefore larger baselines, allowing more dramatic 3-D effects. Cheng says the company is already exploring future versions of its viewer software that would work on 3-D televisions. 'We are moving the power of photography from optics to computation,' he says. 'So when the public really demands 3-D content, we will be ready for it.'"

Comments

Bought a Lytro

[Anonymous Coward]

Returned it.

It was awful, and the resolution wasn't hot

Re:Bought a Lytro

[dfghjk]

An AC calling someone a piece of shit. What a surprise.

I don't recall claiming I was interested in technology, but the fact that I am means I have actually studied what Lytro does. That's why I understand that it will be a failure. I suggest you funnel your anger toward a big, fat investment in Lytro if you feel otherwise.

Re:Bought a Lytro

[dfghjk]

Perhaps you should study more closely the problems Lytro is trying to solve, then speculate on the "new art forms" it may lead to...or perhaps you'd be happier continuing to believe that Lytro could be useful for something.

Here's a hint...depth of field is directly related to the resolving power of a system. Lytro trades resolving power in order to gain variable depth of field after the fact. What it actually does is destroy resolution thereby inflating apparent depth of field, then allow selective degradation of that afterward. The best case result would approach parity with existing technology and no better. Look at the background research of Lytro's founder, you will see that this is true. Now Lytro wants to justify its approach more generically with this computation vs. optics nonsense but that's just BS. Without the right optics and sensors you have nothing. Garbage in, garbage out. Lytro's goal appears to be computing for computing's sake because they don't know any better. Smart people...misguided.

But by all means, believe that Lytro is doing something revolutionary. They've conned plenty of people out of money already, why not you too? They came from Stanford, they have to be good, right?

Re:Bought a Lytro

[pjt33]

Now you're the one throwing around non sequiturs. I'm disputing your description of post-processing refocussing as useless: I'm not claiming anything about the current resolution of any particular product, and I'm certainly not planning to buy a light field camera in the next few years.

My initial idea for a work of art was to combine real-time refocussing with eye tracking to unfocus the area which the viewer is currently looking, focussing instead on something in the peripheral vision, conveying an impression of exclusion from the subject of the work. I'm sure Ai Weiwei could come up with something better. I don't think non-light field cameras could do this except for a subject which remains still enough to take several exposures.

Re:Bought a Lytro

[graphius]

depth of field is directly related to the resolving power of a system

um... no,
Depth of field has to do with the physics of optics. Lytro is using a large array of lenses then combining the collected data in interesting ways. Yes they are sacrificing resolution, but their depth of field "magic" could just as easily be achieved by taking a large number of images taken with a DSLR, but the images would not all be taken at the same time, and the processing power needed would be immense. You might be more correct in saying that Lytro is trading resolution for time...

My first thought

[Anonymous Coward]

Enhance 224 to 176. Enhance, stop. Move in, stop. Pull out, track right, stop. Center in, pull back. Stop. Track 45 right. Stop. Center and stop. Enhance 34 to 36. Pan right and pull back. Stop. Enhance 34 to 46. Pull back. Wait a minute, go right, stop. Enhance 57 to 19. Track 45 left. Stop. Enhance 15 to 23. Give me a hard copy right there.

Lytro's 3-D is inherently limited

[levork]

Wow, TFA is really glossing over an inherent limitation:

the "shiftability" of a Lytro image is a function of the width of the image sensor

If the goal of this is to produce useful stereo content that replicates the parallax seen by humans, then the image sensor needs to be at least as big as the average distance between two human pupils. That's roughly six centimeters. The Lytro's sensor is around six millimeters. Somehow I doubt they're going to increase their form factor by ten times in each dimension, and since the point of a Lytro is to avoid fancy lenses they can't bend the light path to compensate.

Re:Lytro's 3-D is inherently limited

[dgatwood]

No, but they might be able to avoid the lens entirely and do microlensing on a flat surface. For example, I could totally see the entire back of a cell phone be a light field camera, automatically throwing out data from where your fingers overlap the edges from holding it by using capacitive sensors in some way. I mean, we're probably talking twenty or thirty years out here, but that's the direction I see things heading eventually. And that would give you a believable stereo spread, not to mention much more usable resolution.

Re:Lytro's 3-D is inherently limited

[mr_exit]

As long as you've got enough parallax to work out the depth information from your scene, you can push the effect to recreate viewpoints that are wider then you have real data for.

You will end up with tiny slivers of image that you don't have pixel data for when there's a foreground element that diverges more then it did before, but that's easy to recreate. All post converted 3d films have this problem to an even greater extent, there's algorithms out there to clone the surrounding pixels or even use pixels from other frames if the object is moving through the scene

There are lightfield cameras out there that instead of using a single chip, they use an array of small cameras (think cell phone cameras) The adobe one is 500 Megapixels

See the research by Todor Georgiev http://tgeorgiev.net/ The Lytro camera is a nice cheap toy, but there's some stunning results form researchers.

Re:Lytro's 3-D is inherently limited

[Guspaz]

Typical IPD is 54-68mm. IMAX film stock is 70mm wide. This is not an insurmountable problem, at least for professional use.

Imagine a 70mm lightfield motion picture camera. Other than the fact that the data throughput would be positively insane, the requirements for physical size would be substantially less than a current IMAX camera.

I suspect that you can actually get away with less than the typical IPD and still produce a convincing effect. In which case, you can buy the required sensor today; you can get 48mm wide medium format digital sensors, and there's nothing special about the sensor in the Lytro. It's the array of microlenses and software that make it special. So it would be possible today to build a Lytro motion picture camera with a 48mm digital sensor, and I suspect that 48mm is close enough to the typical IPD to produce a convincing effect. Such sensors also have the resolution to make lightfield work for a motion picture (50 MP models turned in up the first page of results on B&H), and the cameras themselves are smaller than most motion picture cameras (or even ENG cameras)...

I suspect that the primary problem would be, again, the data throughput. Uncompressed 24fps 50 megapixel 36-bit images, those would pump out 41 gigabits per second... Compression would be pretty much required. If we use redcode as a benchmark (because apparently motion picture productions are happy with the quality of the compression enough to use it), where the minimum camera-supported compression ratio (on the RED ONE) is 8:1 and the highest is 12:1... This gives us about 5.1 Gbps and 3.4 Gbps... Heck, that's easy to handle. Existing communications tech can handle that, you could have a single 10 Gbps ethernet cable running out the back of your camera to an on-site storage box, and storing that sort of data rates isn't hard. Even a 4TB on-camera SSD module could store 156 minutes of footage... and handle those kinds of write speeds.

voxels

[Anonymous Coward]

Combine the data from 12 or so of these in a matrix and you have a really powerful, accurate, self optimizing point cloud capture device for voxel 3d content.

when the public really demands 3-D content

[Sarusa]

'when the public really demands 3-D content'

When it doesn't require glasses and doesn't give you headaches.

Re:when the public really demands 3-D content

[gmueckl]

A lenticular lens array in front of LCD screens are a nice do-it-yourself solution that almost does the trick. It makes an autostereoscopic display that can display more than 2 images in different directions, making it possible to move around in front of the screen and see a stereo image without glasses. However, there are a couple of limitations. The LCD resolution suffers tremendously and the number of zones that you can create still isn't very high. Maybe it gets better with retina displays, but I'm not sure. Even paper printouts of 20 to 30 images at 600dpi are barely good enough.

Another interesting idea is this proposal: http://gl.ict.usc.edu/Research/LFD/ - replace each pixel on a *huge* screen with a microprojector acting as a directional light source. It is insane in its own special way, but this research group has successfully thrown massive amounts of hardware at problems in the past.

Re:when the public really demands 3-D content

[Dwedit]

There was an article earlier about Tensor Displays (slashdot link), (MIT link), which used a sandwich of three high-refresh-rate LCD screens to simulate a light field by using the screens to selectively block light in multiple directions.

3D imagine that may leads to 3D printing !!

[Taco Cowboy]

I have a Lytro as well. I know that currently its limitations are so severe that have rendered the Lytro cameras to nothing but a novelty.

Its limitations right now are in the computational power --- it does take a whole lot more computational power to make it useful --- and the HORRENDOUS AMOUNT OF DATA to make it any useful.

But, I still have hope in this 3D imaging thing --- I do not see it as mere toy, I see a future link, in between 3D imagine and 3D printing, and beyond.

Currently, to gather data on 3D imagery we use technologies such as MRI, which in itself not really portable.

The concept behind the Lytro 3D camera may offer us a possible alternative.

Re:3D imagine that may leads to 3D printing !!

[ceoyoyo]

"Currently, to gather data on 3D imagery we use technologies such as MRI, which in itself not really portable."

We use things like MRI to gather tomographic data. It's for seeing inside. Lytro doesn't do that, and never will.

Currently if you want to do the kind of 3D we're talking about you can buy a Lytro and get low resolution with a lot of data and processing, or you can buy one of the commonly available compact cameras that include two lenses (like this one) and get instant, high res results.

Re:3D imagine that may leads to 3D printing !!

[cnettel]

Currently if you want to do the kind of 3D we're talking about you can buy a Lytro and get low resolution with a lot of data and processing, or you can buy one of the commonly available compact cameras that include two lenses (like this one) and get instant, high res results.

To be fair, that gives you stereography, not 3D. From stereography, you can compute, tada, something resembling 3D along a short baseline, i.e. not only showing the image through one of the lenses, but a theoretical image from anywhere close to where they were located. The Lytro is a much more solid way of achieveing that, though, if they can create sensors that are both wide enough and carry enough resolution. Currently, they are a long way off. I would even think that you could put to Lytros in kind of a stereographic configuration and get a baseline extension, kind of like astronomical interferometry arrays. You don't need the light field at all locations and having it some distance away is far more valuable than having each and every pixel in between.

Re:The 3D thing is kind of fake

[TheRealMindChild]

Those are awesome. I don't see what you are seeing with the plant image.

Re:The 3D thing is kind of fake

[Dynedain]

But look at the background through the bubbles. The background behind the bubbles doesn't change when you change the psuedo-POV

Look again. Pay attention to the bubble that masks the rear stair column. As you shift the image, the placement of the column within the bubble shifts (the amount of plant displayed varies).

This is beyond stereoscopy, and is a direct result of how they are capturing images. Stereoscopy only works on a single axis of view. This works on multiple axis.

The reason why it looks "fake" to you, is because none of this is done by the camera itself. Everything is done by running computations on the captured image. And as a result, the only way to display it is via an interactive container like Flash. The computations aren't done in realtime, so you only get whatever focus planes the algorithm (or artist) picked out when running the processing. Presumably, you could reprocess for different focal depth points.

Re:The 3D thing is kind of fake

[foniksonik]

Huh? The image with glasses works perfectly. I'm almost certain that you're completely incorrect here. It looks like it loads a full set of pre computed images for all available POV and focal points. Sure there is probably a rough threshold but that's a limitation of bandwidth, not the data.

Medical Applications of the Lytro?

[Guppy]

In November, Lytro, the maker of the first light field camera for consumers, upgraded its viewer software to enable a feature called 'Perspective Shift.' In addition to refocusing pictures after they've been taken, Lytro audiences can now pivot between different virtual points of view, within a narrow baseline.

It sounds like the techniques Lytro uses could make for a really good Borescope/Endoscope. Imagine being able to virtually shift your view to get another perspective (even if only a few millimeters), without moving your scope. If you could process the shifting fast enough, you might use it as a way to compensate for the motion of a beating heart or moving probe. Or upon reviewing a recording, re-focusing on a newly-found item of interest, even after you've pulled your scope out of the patient.

It might also be used to build a compact yet superior type of Fundus Camera -- current cameras are often rather bulky things. The Lytro has a single aperture, yet might be capable of imagine the retina in 3-D (it is a multi-layered structure). The light field info might even allow you to compensate for some kinds of cornea or lens aberration.