Automatic 3D Reconstruction of Scenes

Neil Halelamien writes "New Scientist reports on a piece of software by MDRobotics called instant Scene modeler (iSM), which automatically generates 3D reconstructions of scenes, using a few hundred frames from a pair of ordinary video cameras. The software uses David Lowe's SIFT vision algorithm to quickly locate common features between sequential images, for use in the reconstruction; SIFT has also been useful for generating panoramas and object recognition. MDRobotics has a demo page showing the software being used for crime scene reconstruction, along with animated GIFs of input video and the resulting 3D model."

Comments

open source implementation?

[snooo53]

Yes, I would love to see an open source implementation of this program. Does anyone know of anything like this, or similar software?

Sounds Bogus

[menace3society]

This reminds me of the bit in Enemy of the State where the government operatives take the lingerie store security footage, and then use their computer to "rotate the camer 90 degrees." And on top of that, they then see something in the bag that wouldn't have been at all detectable from the actual camera's angle.

It's pretty silly to suppose that this thing will be able to generate a 3-D representation of a scene without without getting highly-detailed footage of everything from every angle. Otherwise, it would just be a completely bogus modelling system to pull a fast one on people who don't know any better. "Ladies and gentlemen of the jury, although the original crime scene evidence photos don't show it, when you look rotate the angle and look at the far side of the desk, the defendent's fingerprints are clearly visible"

Re:Sounds Bogus

[noidentity]

This reminds me of the bit in Enemy of the State where the government operatives take the lingerie store security footage, and then use their computer to "rotate the camer 90 degrees." And on top of that, they then see something in the bag that wouldn't have been at all detectable from the actual camera's angle.

The technology is basically giving a computer the same information you are able to get by looking at a scene and moving slightly. Unless it's something really subtle, if you can't pick up the information the program can't either.

Re:Sounds Bogus

[drinkypoo]

People have been working on this particular problem for a lot longer than you'd think and there have been methods for creating 3d meshes from 2d video for quite some time now, shown at siggraph, and so on. There are at least tens of books on this and related subjects. Anyway you are exaggerating the problem. Obviously it will not be able to get information from anything it can't see, and without moving around and through a scene all you're going to have is a bunch of contour maps with empty back sides. This is still a huge improvement over having no 3d data at all. It will immediately be a gigantic boon to compositing animators, who will be able to get depth cues from 2d video.

Re:Sounds Bogus

[Solder Fumes]

Actually, you're using an example which I usually think of as the least ridiculous from Hollywood. If you remember the scene correctly, you will notice that what they did was observe a small shift in the portion of the bag that was visible, and then had the computer project what kind of deformation would have caused that shift. And resolution enhancement is funny when they show it being done to a still image, but with video it is possible to analyze successive frames and use time-domain information to filter out pixelation and noise effects.

Home video

[Lord_Dweomer]

You know, I've seen all these tools come out that will be a great boon to the home movie producer. This seems like the next step to letting them use more CG in their movies. As that improves, the limits on what home movie producers can create will be as big as their imagination.

Powerful Technology, some ideas on how to use it..

[shapr]

This is not limited to static scenes as one comment says. It could be used to reconstruct moving objects just as well, with a bit more software.
You could very accurately construct physical models of criminals from security tapes.
You could also construct an accurate model of how they walk. Since every person has a unique walk, that would be more difficult to disguise than physical appearance alone.
You could discover identifying details of the cars they drive, like a small dent in the fender.
This would be perfect for eBay, you could send them a short film of the object you're selling and they would post a 3D model of the item.
This heralds the end of both motion capture and the existing hours long '3D scanning' of clay models used in films like LoTR. Instead of requiring a mechanical stylus to touch every point of a model, you just film it.
Once the software has the ability to turn multiple 2D viewpoints into a single 3D image, this will be the perfect replacement for VR gloves as well. You could have a cameras on either shoulder and your hands would be your 3D mice. That sounds like a nicely intuitive interface.
Moving companies could find this useful. They could film the objects, the moving truck, and in return get an optimal packing order. You could also film the stairway up to an apartment and the software could figure out how to get through any of the particularly tight spots, if it's possible.
This would be good for the sort of augmented reality that washington.edu has researched. When the software can regognize the separate parts in a machine it can display directions for disassembly on a heads up display.
Oh, I can think of lots more uses, but better to get hold of the code and try to implement some of the random ideas above.

Re:That was amazingly cool.

[wheany]

So they are not as much editors as they are approvers.