Microsoft Research Brings Kinect-Style Depth Perception to Ordinary Cameras

mrspoonsi (2955715) writes "Microsoft has been working on ways to make any regular 2D camera capture depth, meaning it could do some of the same things a Kinect does. As you can see in the video below the team managed to pull this off and we might see this tech all around in the near future. What's really impressive is that this works with many types of cameras. The research team used a smartphone as well as a regular webcam and both managed to achieve some impressive results, the cameras have to be slightly modified but that's only to permit more IR light to hit the sensor." The video is impressive, but note that so are several of the other projects that Microsoft has created for this year's SIGGRAPH, in particular one that makes first-person sports-cam footage more watchable.

Comments

Re:Leapmotion anyone?

[Saint Gerbil]

Leap motion uses two monochromatic IR cameras and three infrared LEDs.

This claims to use one 2d Camera.

Apple and Pears.

Microsoft still can't get the UI right

[mwfischer]

Why isn't this a split screen of without and with?

Re:Microsoft still can't get the UI right

[Thanshin]

So you have no opinion on anything you have no personal experience in?

So that excludes conversation about politics (unless you've managed a country), religion (unless you're a god), the opposite sex of yours, the weather (again, unless god), ...

Re:Microsoft still can't get the UI right

[wonkey_monkey]

Why isn't what a split screen of without and with what?

Re:Microsoft still can't get the UI right

[kryliss]

Yes I know, I'm feeding the troll. Have you ever thought that this may some day be useful for people that have that have disabilities where they can't grab a mouse? My mom takes meds that make her hands shake all over and she has a terrible time with a mouse, she sometimes has to grab her mouse hand with her other hand just so she can click the button on the right spot. How about people who have arthritis so bad that they can't even grab the mouse?

Re:Leapmotion anyone?

[Tx]

It is apples and pears on one hand, however the fact that the camera needs a modification, however small, means that you will still be buying a special bit of hardware to make your gesture control work, so in that sense it is in the same boat as the Leap. Except of course that the piece of hardware in question should be a lot cheaper, and could easily be included in laptops/tablets/monitors at minimal extra cost, if it really works that well and the idea takes off.

Doesn't the kinect use an ordinary camera?

[fuzzyfuzzyfungus]

I thought that the kinect, while nicer than the average cheapie camera in terms of optics and sensor, also used a fairly normal camera(well, one higher resolution visual band one for image and one IR one for depth) and that the real secret sauce was the IR laser device that projected the dot pattern on the environment for the camera to pick up and interpret. Am I remembering incorrectly?

Re:Doesn't the kinect use an ordinary camera?

[axedog]

You are indeed remembering incorrectly. Kinect has a colour camera, an IR camera and an IR Emitter. http://msdn.microsoft.com/en-u...

Re:Doesn't the kinect use an ordinary camera?

[axedog]

Sorry, I read your post a tad hurriedly. Kinect has the three components that you mentioned, but that make it something other than a "fairly normal camera".

Re:Doesn't the kinect use an ordinary camera?

[plover]

You are correct. The IR laser and IR camera are used to measure depth, while the visual light camera only picks up the image.

The cool thing about the Kinect's IR pair is that it senses depth in the same way a pair of eyes does, in that the delta between left and right eyes provides the depth info. But instead of using two eyes, it projects a grid from the location where one eye would be, and the camera in the other location measures the deltas of "where the dot is expected - where the dot is detected". The grid is slightly randomized so that straight edges can be detected. If you've ever stared into one of those Magic Eye random dot stereogram posters, you're doing pretty much the same thing the Kinect does.

This system is very different. The Kinect has a deep field of view, but all the demos show this working in a very short range. I haven't yet read the paper, but I'm wondering if that's the point of the IR.

Re:Doesn't the kinect use an ordinary camera?

[Anonymous Coward]

I thought that the kinect, while nicer than the average cheapie camera in terms of optics and sensor, also used a fairly normal camera(well, one higher resolution visual band one for image and one IR one for depth) and that the real secret sauce was the IR laser device that projected the dot pattern on the environment for the camera to pick up and interpret. Am I remembering incorrectly?

Yes and no.

It's correct for Kinect 1. It uses a "structured light" approach (developed by PrimeSense), which projects a magic pattern and has a (regular) IR cam observing the distortion in the pattern.

Kinect 2, on the other hand, uses real Time-of-Flight (or rather, it measures the phase difference between the modulated IR signal and the reflected IR light) imaging, very similar to laser distance meters, just 2D instead of a single point. (Versus Kinect 1, it provides a better resolution and less noise.)

Re:Doesn't the kinect use an ordinary camera?

[OzPeter]

This system is very different. The Kinect has a deep field of view, but all the demos show this working in a very short range. I haven't yet read the paper, but I'm wondering if that's the point of the IR.

From watching the video my understanding is that they illuminate the subject with a fixed IR source and map the drop off of the reflected IR in 2D space and then interpret that drop off as a depth map of the object they are looking at. Which looks surpassingly accurate for the sort of use cases they demonstrate. They also point out that this technique is not a general purpose 3D system.

Re:Doesn't the kinect use an ordinary camera?

[ab8ten]

That's how the Kinect 1 works. It projects structured light and then reconstructs the world based on deviations from the expected pattern. It's built from off-the-shelf parts. The Kinect 2 measures the time it takes for an emitted laser light to be reflected back to the sensor. It's much more accurate and reliable, but requires purpose-made sensors, thus increasing the cost. Here's a good article with technical descriptions of the two methods: http://www.gamasutra.com/blogs...

Re:Doesn't the kinect use an ordinary camera?

[qpqp]

I believe, the question was, whether there's been an advance in the depth-sensing algorithm in the sense that you don't need a specific IR pattern (i.e. a grid) like in the Kinect anymore, but that just a couple of IR emitters are enough.

As predicted in Better Off Ted

[OzPeter]

At the very end of the video it describes how the system is tuned to skin albedo. The only problem with this is that various races around the world have different albedos - which does have a real world effect in photography when trying to expose correctly for skin. In the video they mentioned training the system on the user, but all users shown in the video were white - so I can't say how well it would work for non-whites. But in general I am impressed with what they have done.

Back in 2009, in Better Off Ted episode 4 "Racial Sensitivity", they developed a security system that had issues with skin albedo and not detecting (from memory) dark skinned people - which resulted in all sorts of hijinks for the African American employees

Re:As predicted in Better Off Ted

[Anonymous Coward]

That itself is probably based on the fact that there are biometric systems that do have issues with certain racial types. Iris scanners are one example, as the extra melatonin means there's too little contrast to pick up enough detail to be as reliable without tuning for that type of iris. That's been known since way before 2009.

Re:Leapmotion anyone?

[bloodhawk]

It is completely and utterly different from how leap motion works, they are not even vaguely similar. this is about using a single 2D camera not multiple cameras and LED's

Hyperlapse

[fgouget]

They should rename HyperLapse to SmoothLapse, StableLapse or CleanLapse.

Re:Hyperlapse

[Anonymous Coward]

ProLapse. (with apologies to RockStar)

Re:Leapmotion anyone?

[Anonymous Coward]

As fast as the price of camera hardware has been dropping, adding a 2nd camera doesn't seem like much of a price to pay to add a dimension to the resulting product....

yeah

[ClumsyNinjaCheats]

It's hard to disagree. Microsoft is power!