Building 3D Models On the Fly With a Webcam

blee37 writes "Here is an excellent video demonstration of a new program developed by Qi Pan, a graduate student, and other researchers at the University of Cambridge. The 'ProFORMA' software constructs a 3D model of an object in real time from (commodity) webcam video. The user can watch the program deduce more pieces of the 3D model as the object is moved and rotated. The resulting graphics are of high quality."

Comments

The Death of Hollywood

[headkase]

With open-source rendering images already well established and continually improving that only leaves the content areas under-developed. This method will allow anyone with an object to digitize it. This will enable people to take that content and then mix it in virtual environments. Throw in some voice-synthesis software, some directing software, and a million monkeys hammering away at plots then Hollywood as an institution is dead. This is another piece, the others will fall into line as well. It is ironic in that in one of the Civilization games, discovering the Internet invalidates the Hollywood Wonder.

Re:The Death of Hollywood

[Idiomatick]

Awesome, All I need to recreate Starwars now is to visit the nearest starport and record a video of a TIE fighter with my phone and I'll be good... Oh wait!

Re:The Death of Hollywood

[Anonymous Coward]

Actually while you laugh it could allow all sorts of fun stuff because you can model that tie fighter or spaceport by hand if necessary, then visualize and convert it to a 3d model. I don't know about you, but there's lots of things I can do easier with raw materials and my hands that would look good as a model that if I was to model in 3d would take a lot longer to produce...

Re:The Death of Hollywood

[Idiomatick]

For sure! I mean this is clearly useful tech. BUUUUT saying death of Hollywood is horribly wrong. I mean... other countries hardly even bite into Hollywood. While budgets might be overblown... You can't do for >1000$ what studios do with 200million dollars PLUS. That's a pretty fucking big gap.

Re:The Death of Hollywood

[garg0yle]

There's one more key we're missing - the ability to render humans realistically. We can manage just about everything else, but until we can make a virtual John Wayne that looks like John Wayne and not a wax mannequin, we're not going to see Hollywood abandon "talent".

Of course, once we can do so, the next step will be to "improve" the stars - start with a virtual Natalie Portman (for example), and then "tweak" her for further fanboy appeal.

Re:The Death of Hollywood

[Idiomatick]

Hollywood != B-films

Re:The Death of Hollywood

[McNihil]

Or why not let the viewer choose who plays that part... Angelina Jolie with those perky ones from the Tomb Raider movies for instance. How about watching Cassablanca as yourself as Bogart? Now how about being Dekkard in Blade Runner? The only thing that is needed is the motion capture of believable performances that's all.

Re:The Death of Hollywood

[Chyeld]

I take it you aren't used to using Poser or Blender, or any other related 3-d software and thus don't know the joy of: "You STUPID PROGRAM! I just want her to walk down the stairs! Why are her arms doing that! NO! NO! NOO!!!! Stop floating down the stairs and walk! Why is your hair clipping through the wall, why is your hair even moving that way! STOP IT!"

Hollywood's death knell might be sounding. But it's got a few more good decades in it left before we need to morn for it.

Re:The Death of Hollywood

[Requiem18th]

Ah but in the near future you will need to get a copyright license to make a picture with a model taken from a real object. Soon you won't be able to make a movie without getting a "RAND" license for every object that appears in your movies.

Re:The Death of Hollywood

[RAMMS EIN]

In theory, we can make good computer games, too.

But how many open-source games can you name that have great graphics? And how many closed-source games with great graphics are there?

I don't think Hollywood is dying just yet.

Re:The Death of Hollywood

[Quantumstate]

I think the OP was implying that this new technique might be useful for making the graphics for these things. Since it has only just been created it would indeed be very surprising if open source games had used it to make great graphics already.

Re:The Death of Hollywood

[Jarik C-Bol]

dear god, why did you mention blender. that is not a 3D modeling program, its a psychological torture device.

Re:The Death of Hollywood

[westlake]

With open-source rendering images already well established...that only leaves the content areas under-developed.
Throw in some voice-synthesis software, some directing software, and a million monkeys hammering away at plots then Hollywood as an institution is dead.

The geek needs a million monkeys.

Hollywood gets by with a handful of men men like John Lasseter, Andrew Stanton and Brad Bird. In sound design, a Ben Burtt.

Digitizing the prop is trivial.

Knowing which prop to use - and how to use it is not.

There are around 400 individually designed objects in the "bubble wrap" scene in Wall-E.

For 10% of your final grade, your mission is to explain how these props are used to create a mood, comic or tragic, advance the story, reveal character.

Re:The Death of Hollywood

[mad_minstrel]

You're right. I haven't seen anything that good from Hollywood in years.

Re:The Death of Hollywood

[Thansal]

That actually is a rather awesome idea. Reminiscent of the Sprawl trilogy's Simstims, but more likely to happen some time soon (good 3d scanning/modeling of a human vs wetware).

Re:The Death of Hollywood

[Telvin_3d]

To hell with Blender, why did he mention Poser? It is the photoshop-lensflare-filter of the 3D world. Something which has it's place but which is used 99% of the time to let amateurs run around shouting "look at me I'm an artist too!"

Re:The Death of Hollywood

[Telvin_3d]

Really? Where do you think that people are magically going to get real life versions of the wanted 3D characters? Besides, anyone good enough to clean up and make these kinds of models usable is good enough to just be able to model it in the first place. Faster too.

Re:The Death of Hollywood

[vikstar]

That's because Hollywood is just crap rendered in good graphics. There are international studios that produce high quality cinema which will live on due to the content, and not a glossy wrapper.

Re:The Death of Hollywood

[Ambiguous Coward]

Yes, you're right because the only 3D models used in games are fantastic monstrous characters. You never see things like chairs, tables, lamps, cars, and refrigerators. There's simply no need for them! And being able to easily create models of such things would be equally useless, especially when anyone in the world would suddenly have the ability to contribute large numbers of everyday objects to some sort of global repository.

Sarcasm aside, at the very least maybe this will spell the end of the infamous Crate. Probably not, though.

Re:The Death of Hollywood

[Telvin_3d]

Yeah, but those objects are easy. Like, dumb easy. It takes far, far less time to make a fridge the traditional modelling way than it takes to clean up data like this. And you can make any fridge you want instead of having to find a real fridge and haul it out, light it and scan it.

Object scanning systems have been around for over twenty years in various forms and costs. The fact that even the people who have been able to afford it don't use them should tell you a lot about how applicable this tech actually is. If scanning was cheaper or faster or easier than modelling you would see people like Blizzard and WETA all over it. But you don't.

Re:The Death of Hollywood

[mldi]

I take it you aren't used to using Poser or Blender, or any other related 3-d software and thus don't know the joy of: "You STUPID PROGRAM! I just want her to walk down the stairs! Why are her arms doing that! NO! NO! NOO!!!! Stop floating down the stairs and walk! Why is your hair clipping through the wall, why is your hair even moving that way! STOP IT!"

Hollywood's death knell might be sounding. But it's got a few more good decades in it left before we need to morn for it.

Hilarious you bring that up. Did you ever look at the model of Big Buck Bunny? It's like his face is sucked in down his throat, just so that when it renders, it looks like they way they want it to. It's horribly fucked up for anything more complex than snowmen or giant walking stick men.

Re:The Death of Hollywood

[Ambiguous Coward]

Okay, how about instead of using a live camera view, you record footage of a fly-by over a city?

And what do you want to bet Google would pee themselves if they were able to use this to, say, replace streetview with 3D city models generated through such technology? Instead of paying someone to drive a car around which takes pictures every so many feet, they'll pay someone to drive a car around which generates a fully-textured 3D model of any city around. Suddenly, streetview graduates from Myst-style clickthroughs to something much much more immersive.

That is to say, which is cheaper: paying some group of people to model (including photographic textures!) all major (and many minor) cities around the world, or just to drive a car around and do it automatically? To use your own argument: if falsifying (painting, modeling, etc.) that data was cheaper or faster or easier than driving a car around, you would see people like Google all over it. But you don't.

You're just not thinking big enough. :)

Any open source software?

[gnalle]

I s there any open source software that can generate a 3D model from photos? As far as I can see the source code of proforma is closed. http://mi.eng.cam.ac.uk/~qp202/my_papers/BMVC09/

Re:Any open source software?

[ivanwyc]

Well, if the paper is well written, implementing it as a GPL project won't be too difficult.

Wait..wait... on the FLY with a WEBcam?

[Anonymous Coward]

Did the Slashdot contributor discover TFA with a spider?

Re:Wait..wait... on the FLY with a WEBcam?

[Footsienabackyard]

I did using Stumble...

Re:Download link?

[lbalbalba]

From the website:

In a few months time when code porting (to new library versions) is complete, a Linux-based demo of ProFORMA should be released.

Star Wreck: In the Pirkinning

[Animaether]

Are you referring to Star Wreck: In the Pirkinning?
http://www.imdb.com/title/tt0472566/
( yes, it has an imdb entry ) /nokarma

Hello 'Likeness Theft'

[kbob88]

I can just see it now -- anyone who can get a bit of video of you can create a 3-D models of your face and body, and then do anything with the likeness. When rendering gets really good, this could be a bit embarrassing. Instead of 2D retouched photos of celebrities and politicians, we'll be seeing hacked up 'animated' (but realistic) video of them doing all sorts of wild stuff. Well, it might be a boon to the porn industry, at least in the short-term before the rendering software becomes available to consumers.

Re:Hello 'Likeness Theft'

[Quantumstate]

It is still very difficult to animate the model in a realistic way so you would need a bit more than this program. I can see it happening in combination with other footage which you could capture movement from then change the models to the celebrities.

Re:Hello 'Likeness Theft'

[hitmark]

have a chat with hollywood, they have worked on this for a while.

like say 1.000.000 agent smiths ;)

Re:Hello 'Likeness Theft'

[Telvin_3d]

Yeah, that only took a team of dozens of people plus a crapload of body double actors and months of work. And, despite the fact that it was almost 10 years ago now, it wouldn't be that much easier to do today. Well, a scene of a bunch of duplicates standing around would be fairly simple but they have been doing those for decades. The only significant difference between making the opening fight scene from Matrix2 then and now is that advances in hardware would cut down the final render time.

Re:Hello 'Likeness Theft'

[sw155kn1f3]

If you're old enough you can remember "The Running Man" starring governator which did exactly that to show his character as having lost the battle.

Re:Hello 'Likeness Theft'

[Footsienabackyard]

Yes, the export format is important to this...but I only use gmax.

Needs Chroma Keying

[KalvinB]

It may help the modeling process if chroma keying could be used to make the camera ignore supporting objects like a turn table or a hand that's holding the object. Another improvement could be to automatically cut out excess vectors and triangles. It wouldn't be too difficult (for someone who can make this type of software) to determine that the plane that makes up the side of the demo building is virtually flat and reduce the complexity to two triangles.

One of the key limiting factors to amateurs making mods for popular games is the expense and complexity of 3D modeling software. Few good coders are also artists. And few good artists are also coders.

Back in the day anyone could put Barney in DooM or Wolf3D. Now it takes expensive software and a lot of time to shoot up your favorite character you love to hate.

Re:Download link?

[Anonymous Coward]

That site only has pdf files and videos. Not really useful at all. We want to TRY this software, not watch some fancy demo video.

-XcepticZP

Re:Amazing new mathematics research

[Inda]

Which knob modded this up? C'mon own up.

There's one born every minute.

The future is now

[westlake]

Ah but in the near future you will need to get a copyright license to make a picture with a model taken from a real object.

Try using a glass Coke bottle, a Rubik's Cube or an Igloo cooler in your flick and see what happens.

Re:The future is now

[Requiem18th]

Wow time flies like an arrow. So yeah, it's just like you can't play ancient music for free because the recording was done in the last century or you can't post an ancient text because the scanning/transcription was done within the last century, we live in a cultural deadlock where creating is a risky business where lawyers are advised.

3D vision for robots

[cptnapalm]

I was thinking about robots one day and I was wondering why those who work on computer vision didn't do something like this. Instead of trying to get the machine to understand the analog world, why wouldn't it be better for the machine to have an internal representation of the world by making a 3d map? Quake 3 CoffeeShop, if you will.

The idea I had was that the vision system creates a 3d map with entities, mapped from the vision system as well, inside. The AI works within the 3d representation of the world. If the AI wants to move from A to B, it signals the body controlling subsystem to start walking. When the 3d representation, being informed by the vision system, tells the AI that it is at point B, then the AI signals to stop walking.

Hardware constraints not withstanding, is this model any good?

I'm just a lowly, early middle aged novice C programmer who has never actually done anything with robotics, so if what I said made no sense or is obviously idiotic, I do understand that my ideas are comin' outta my ass.

Re:3D vision for robots

[KalvinB]

What was created here could probably be extended to look at two web cams simultaneously to calculate the 3D space rather then relying on two separate images from one camera.

What this software is doing is looking at the rotation of an object rather than displacement. So it may need some other adjustments besides just using two cameras to get two source images at once.

Re:3D vision for robots

[Rabbitbunny]

That's actually how my idea for a WoW bot worked...

I see a use for this technology.

Re:3D vision for robots

[cptnapalm]

The idea that it could be implemented entirely in software also occurred to me. The vision system would be different, of course, with it taking the video from a game (WoW or Q3A) and translating that into a generic 3d engine which the AI can use. Instead of sending it to an AI-to-Robot system, it sends it to a AI-to-HowThisGameWorks layer which sends the info to the game.

I hope that made sense.

Re:3D vision for robots

[cptnapalm]

Noise I had not even thought about. Which goes back to me not having a clue :)

Re:3D vision for robots

[Monkeedude1212]

There are a few kinks to work out with this model, but essentially, it could work. Specifically the example they showed would model an object, and not the world around it. So the algorithm would have to be reworked to map the world, and like someone else mentioned, would probably use 2 cameras (which means its compiling twice the data, which means "Real time" might not be so fast)

There is also the major issue with dynamic lighting effects. Since lighting is the primary use of how to model something in 3D, you generally just move the object (or yourself through the world) and you assume that the light source is always stationary. Or, for more accuracy, you can move the light source across 129600 (360 by 360) degrees of space, and knowing the lights path, get a more accurate understanding of the depth and shape of an object or world. Now movement and dynamic lighting combined would give you a far more accurate object then just one or the other. (In fact, if you or the object don't move, you'll only be getting half of what is there, essentially the front side of the object or only what is sitting in front of you in your world)

Now, the biggest kink to that would be a dynamic light source where the path isn't predicted. Example, the Robot is walking along, and a Car drives by. The altered light on the ground could mean the robot percieves a bump in the road, when in fact, there is not.

I hope that made sense.

Re:3D vision for robots

[eggnoglatte]

The idea is not stupid, but it also isn't new. It is just turns out to be a little harder to get working in practice than you might think..

Re:3D vision for robots

[cptnapalm]

Thank you! When I did my searching for something like my idea, I couldn't find anything. Obviously my Google-fu needs some work.

With regards to how easy it would be, my estimate was that it was beyond my abilities. So it still is, but just way way beyond my abilities.

Re:3D vision for robots

[hitmark]

thanks for reminding me, as i think i read that research have shown that this is how we humans operate.

as in, we build up a internal model of ourselves and whats around us, and keep refining said model constantly.

thats why we develop habits and preferences, as that means the models do not have to change.

this includes our own body btw, and is the probable cause of phantom limb experiences.

Re:3D vision for robots

[cptnapalm]

Light is also not something I thought too much about; it seems to be above my intellectual pay grade.

Could a basic implementation be done in just software like this: Take a 3d game engine with a map and a model. Attach two "player's" eyes to the face of the model so there are two POVs instead of the usual FPS's one. The output goes to the video processing system (perhaps on a different machine), which creates a new 3d map with entities based on what the video system came up with.

Idea of it being that more could be controlled than in trying to model the real world, like light and shadows. So if a game car drove past, it doesn't cast a shadow, so the programmer can work on getting the basics down first before moving on to dealing with things casting shadows.

Re:3D vision for robots

[nghiaho12]

I actually did this for my thesis. I built a quake like 3D model of the environment using an expensive laser range scanner. The robot has enough geometric and photometric information it needs to perform localisation and path planning. This type of problem where you have the map before hand is generally called "global localisation" and is easier than the Simultanous Localisation and Mapping (SLAM) problem, where you don't have a map prior.

Academic projects versus commercial applications

[Frans Faase]

There seems to be a huge gap between these kind of academic projects and the commercial available programs. I have come across several commenrcial applications that can do these kind of things, but these applications cost at least a 1000 dollars or even more. And then there are all these academic projects (going on for at least two decades), which present nice video's and papers, and sometimes release some software. But when you look at the software, you discover that you first have to download nine other package and compile the whole thing and what you get is some kind of script you have to run, with all sorts of command line options. But sofar, I have never found an application with a solid interface on the level of the Gimp or Blender for the matter of the fact. I find this rather strange. I am almost getting the impression that some of the results are sold to the developers of the commercial packages.

Re:Academic projects versus commercial application

[Telvin_3d]

The major reason that these types of programs don't get expanded into commercial products or bought and integrated into existing products is that they are cute tech demos but not particularly real-world interesting.

Almost without exception anything simple enough for these types of reconstruction programs to handle is too simple to bother with. The paper church in the demo video for instance. The final wire-frame product is, sadly, crap. Neat and interesting crap but still crap. There are at least 3 times the polys that the form needs and almost all of the significant edges are in the wrong place. In the time it would take to clean up the data into something worth using I could build a better model form scratch including textures.

There are perhaps some very niche uses for this in terms of augmented reality. It could be integrated into a game or chat program to give a more realistic version of those make-an-avatar-from-your-webcam gimmicks that seem to gain attention every once and a while. If this guy has developed some very good algorithms he might get the interest of some of the match-moving software companies like Syntheyes.

But the reason this kind of this never shows up in profesional 3D packages is that if you are good enough to be using the software professionally you are good enough not to need these kinds of crutches. It's the 3D equivalent of Dreamweaver's auto-generated spaghetti code.

This is not new

[mapuche]

I haven't read the article yet, but there's already a program doing this with cheap cameras, version 1.0 was free:

http://www.david-laserscanner.com/

Re:This is not new

[mapuche]

BTW, with David you need a laser.

Re:This is not new

[T-Bone-T]

These days who doesn't have a laser level?

Re:Aside from pirating...

[Anonymous Coward]

Have you looked on bittorrent? *ducks from flying legal noise*

Volker Blanz and his 3D Celebrities

[MrSteveSD]

There seems to be a huge gap between these kind of academic projects and the commercial available programs.

Indeed. Check out the work of Volker Blanz . He was producing amazing 3D models of celebrities from photographs a decade ago. Yet you can't get anything remotely that good today (I've used Facegen etc).

Simultaneous location and mapping

[Animats]

That's called "simultaneous location and mapping", and in the last five years, good algorithms have been developed and quite a few systems are more or less working. Search for "Visual SLAM".

The Samsung Hauzen vacuum cleaner uses Visual SLAM. There's a video. This is way ahead of the blundering Roomba.

Re:Academic projects versus commercial application

[laddiebuck]

Sadly anything useful from academia (not implying that this is) is spun off into private companies, 9 times out of 10. This despite the fact that it's mostly developed on public money.

Re:Academic projects versus commercial application

[snadrus]

Simplification is easy: answer "if this point was missing, what's the angular difference lost?" and if it's below a threshold then do it.
Then area thresholds could be set, or more logic to do it for you.

Motion Capture - Yahoo group

[sonamchauhan]

People interested in this area may find the 'Motion Capture' Yahoo group useful.

Its website is located here:
http://movies.groups.yahoo.com/group/motioncapture/

A recent interesting message from the group (edited to evade ./ Junk filter):

---------- Forwarded message ----------
From: Brad Friedman
Date: Sun, Nov 15, 2009 at 9:35 AM
Subject: [motioncapture] releasing some optitrack open source software and code
To: mocap list

Hey all.

Been a while. I've been rather busy with other things.

I'm releasing some OptiTrack related open source software and code.

Two simple modules have been released. One is LGPL and one is BSD.

http://www.fie.us/kinearx

Its all alpha level stuff. But its working well enough for me to move on to other things. I feel its worth open sourcing something that runs, and does useful things. Even if its not feature complete and production tested.

Its really for developers more than end users for now. But this list is about developing tools. So there you go.

My theory is simple: This particular part of working with OT cameras is kinda generic and somewhat dull. I'd rather have an open source backend that we can all maintain together, than have to maintain it completely by myself. The windows app is LGPL. The example client, is BSD. Therefore, it should be good for open source and commercial developers who, like me, also want to collaborate on the dull backend and spend more time on other aspects of mocap systems. The projects are separated by a nice simple binary data stream layer to make sure their licenses don't conflict.

Two main features of interest:

1. Cameras operating on different computers can be synchronized by looping the sync cable through. The existing Arena and PointCloud tools from NP don't do this on their backend.

2. Development is jail broken out of the PC environment by the binary protocol. The example script is written in python and runs on OSX and linux, for example.

Further work to be done:

1. Better support of OT cameras other than the V100r1. That's the only camera I have, so that's what I know is supported. C120 and V100r2 are something I can't really confirm function of. But I'd like to.

2. Occasional sending of a GMT timestamp from the 2d servers, interleaved with the frames, for sanity checking purposes and helping with situations where the sync cable may not be working fully.

3. Switch between the mass marker mode, and COM object mode. This should make the grayscale and masking features of the camera work again (I think being in mass object mode disables them).

Feel free to e-mail me with questions or queries.

Brad Friedman
VFX - Consultant - Mocap
http://www.fie.us/

Re:Academic projects versus commercial application

[HigH5]

"Genius is one percent inspiration and 99 percent perspiration." Thomas Edison
I think that the 99% percent is often the problem with these projects. You come up with something, make a proof of concept, but it takes a lot more work to perfect it.

Re:Academic projects versus commercial application

[Geminii]

If you are not good enough to be using CAD software professionally than you are 99% of the new market this thing just created.