Slashdot Mirror
2D To 3D Object Manipulation Software Lends Depth to Photographs
Iddo Genuth (903542) writes "A group of students from Carnegie Mellon University and the University of California, Berkeley have developed free software which uses regular 2D images and combines them with free 3D models of objects to create unbelievable video results. The group of four students created the software (currently for Mac OS X only) that allows users to perform 3D manipulations, such as rotations, translations, scaling, deformation, and 3D copy-paste, to objects in photographs. However unlike many 3D object manipulation software, the team's approach seamlessly reveals hidden parts of objects in photographs, and produces plausible shadows and shading."
The only way this will really explode.
No longer is it Photoshopped, but instead we say it's been Carnegie Melloned.
isn't this just texture mapping onto a 3d model?
I don't do anything like this for a living, but I must say I'm impressed. I'm fairly certain someone will say this was done back in 1997 though so it's nothing new.
Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
It's "a software" again. I'd like to give the author an information, perhaps while we eat a spaghetti, that in English we have "mass nouns" and thus you cannot have one hardware or one clothing or one software.
How can images be admissible in court in our modern technological age of 3d manipulation of 2d images? Sure, they still have visual artifacts (like in the video presentation for this technology, when the airplanes are turned into 3d, their propellers are not changed, the same image of a propeller is kept for 3d model as was on the original 2d picture) but eventually all of these will go away, it may become impossible to detect that an image in front of you was manipulated at all.
Eventually this will also apply to video footage.
Add the digital augmentation of reality into the mix (Google Glass, etc.) and you can't rely even on the recorded information. We know that people are not good at remembering the details of what they saw, but if cannot be sure of images and video (and obviously audio) either, then this type of data becomes useless in courts. That's an interesting development in itself, never mind the fact that you can now turn a picture into a movie if you want.
You can't handle the truth.
The use existing models to 3D map photographic texture data onto them, remove the 2D object from the photo, and then insert the texture-mapped model into the photograph.
Basically some light photogrammetry, some content-aware filling and some of that sweet, sweet Photoshop CS 3D insertion stuff.
This is NOT a big deal outside the automation aspect.
Still, I expect Adobe to buy it up or do something similar eventually, perhaps even opening a market for 3D objects.
P.S. I'm sure some of you nerds will say I'm wrong, but that's what I saw. No magic, nothing cutting edge. No stupendous breakthrough. About as interesting as SnapChat AFAIC.
IANAL so I don't know how admissible photos are today but I do know that some cameras support adding a digital signature to an image. If the image is altered later then the signature will be invalidated. This feature has been around for quite a while and I believe journalists are the ones who use it.
Pictures and video are used in court but someone testifies that it hasn't been modified. If the defense argues that it has been modified then a jury weighs the merits of that claim.
While those results look impressive, in some of the demos where objects are seamlessly moved around, how are they filling in the original background (or what looks like it)? The video largely explains how the model is textured, lit, environment mapped, rendered with shadow projection with calculated perspective and depth of field, but I didn't hear much about re-filling the background. I assume they're cloning or intelligently filling texture ala photoshop, or perhaps in all cases where they showed something being animated it was a new clone of an existing object into a new area of the photo?
ACADEMIC OR NON-PROFIT ORGANIZATION NONCOMMERCIAL RESEARCH USE ONLY
I am not really here right now.
I would love to know how easy such manipulation is to detect? Is it harder or easier to detect than photo-shop?
At some point, photo-shop type effects will become undetectable.
excitingthingstodo.blogspot.com
The producers of the video used a trick to make their software a little more impressive than it really is.
The item which is to be converted to a 3D model covers part of the image; for instance, the chair covers a part of the carpet, and the origami bird covers the tips of the fingers. However, in the edited form, those covered areas magically appear.
Clearly, this requires two carefully-taken pictures; one with the object, one without. And of course you must be careful that the presence of the item is the ONLY difference between the two, or the effect will not be nearly as impressive.
This is a relatively minor complaint in the face of the impressive combination of technologies on display here, but still, the video would be significantly less impressive if they paused during the origami demo to say, "Now we have to carefully take another, identical picture. A tripod is required for the use of our software to maintain original camera position, and extreme care must be taken not to disturb the scene while removing the object that is to be manipulated."
Of course, there's a chance that they're drawing that hidden space in some other way, but that'd surprise me since they don't even mention it in the video. I guess they could use the same method they use with the 3D objects; interpolate from nearby texture or from a model's stock texture. But that wouldn't work, since the shape of the fingertips is hidden in the origami scene - no way to automatically generate that.
Pretty much this.
A few decades down the road with software like this improving, it will be stupidly hard even for experts to see modifications.
Throw in some basic genetic algorithms and a large cache of images and models, as well as some rules for thousands of visual scenes, and a few million evolutions later, it will be capable of modifying scenes so realistically to the point where an expert would never be able to ID any changes to it.
Genetic algorithms are great for things like this. The more exposure to scenes niche and common, the better.
Still, there will always be the "2nd image" problem. Things like CCTV, if static, will be harder to fake unless a person has access to the original captures.
It is easy to modify a unique picture so it doesn't look changed. If NOBODY knows what is behind that chair, you could put literally anything, as long as it fits the repeating textures at the edges and what you expect to be there. Be it a dead body or a huge dragon dildo.
But if someone hates you enough, that might not be a problem.
No, it seems they are using inpainting :
We compute a mask for the object pixels, and use this mask to inpaint the background using the PatchMatch algorithm [Barnes et al. 2009]. For complex backgrounds, the user may touch up the background image after inpainting.
Thus, only one image is required.
The algorithms and software get better and better.
SIGGRAPH next week in Vancouver.
watching or attending siggraph is like watching an Ubisoft conference.
Everything looks amazing on stage, but when you get your hands on it is another story altogether.
I'll believe this works when I use it. Until then, I might as well go watch the lawnmower man and consider it a documentary.
Why couldn't the algorithm be content aware? similar to content aware fill in photoshop? that is much more likely given the scope of the software.
...informations to better builds the good!
Bad informations with for the good people so making of the understanding isn't!
Blogging because I can...
I ask you!
Comment removed based on user account deletion
And....
It has been cracked.
This site used to be quite good.
Then YouTube put the banhammer on their trolls, and this place went to hell.
The problem is that as technique improves, the theory that the photo/video was altered in a way that can't be detected becomes ever more plausible.
It was easy to take the witness's word for it when the alternative would involve millions in equipment and would likely be trivial to detect.
I thought the point was to create believable video results. Bad, fake-looking 3D out of 2D sources has been done to death, mostly for the cinema...
Violence is the last refuge of the incompetent. Polar Scope Align for iOS
I've done a little bit of work in a related area, so I skimmed the paper (at the bottom of the first link,) and it's nowhere near as impressive and automagical as the video makes it seem. The user has to provide a mask distinguishing the object they are manipulating from the rest of the image, and then the user also has to provide the 3D model for the object! The model is then smoothed to better fit the original using the mask and the inferred illumination, textured using the image, and then popped out to be manipulated in 3D. Not to detract from how cool this all is, but the user is still doing a lot of the heavy lifting.
I bet a combination of the techniques in this paper and the techniques of multiple view geometry (which is where I've actually done a bit of work) would be considerably more impressive and automagical.
That's pretty crashy software. At least it builds. You'd think they would distribute something a little more solid.
- Zav - Imagine a Beowulf cluster of insensitive clods...
This looks pretty cool, but I have a lot of questions.
On it's surface, it looks like a lot of the results they're getting wouldn't currently be outside of the realm of student level work, such as the simple practice of projecting and baking textures into materials from photographs, the innovation seems to be that they're quickly automating a lot of that stuff into a UI with a fast lighting solution. One of the things I find most rewarding about 3d is that you sometimes get this huge burst of increased productivity, as long as you're not too bummed out about things you've spent time and energy learning how to do becoming obsolete. This isn't that different, fundamentally, than setting your viewport background in Maya, 3ds Max, etc. to be a photograph after properly matting your foreground objects and projecting textures with adjusted reflectivity, just without all of the manual tediousness. Also, there's also been other, similar work done on the subject, that I've heard of, but this still looks pretty neat if it's something you can use right now without a billion dollar computer.
One of the big things this tech might be doing is streamlining the process of match lighting. I personally can't wait till the major software packages have integrated solutions for easy lighting from photo sources. Currently the setup for photo matting is a pain, it requires stitching together panoramic photos of reflective chrome spheres - on location - or carefully using observation skills to recreate the lighting by hand (which can be very difficult for glossy surfaces). It would appear, however, that we're on the brink of not needing those things anymore. That being said, this software still has a bit to go, however.
For example, the lighting information baked into the diffuse textures of the objects, in these examples, does not appear to be dynamic - if you watch the taxi-spinning segment you'll notice that the specular highlights on the hood of the car do not properly update as the orientation of the model changes in relationship to the light sources, making the taxi appear to have white paint streaks once rotated out of alignment with the light source. The car falling off the cliff example is probably the most apparent in final results, as the strong baked lighting makes the coloring look off. The way we 3d artists get around this problem is to eliminate the lighting information in our diffuse textures as much as possible before reapplying them as flat color, and then let our lighting rigs take care of the reflections, shadows, and such. As they mention this software doesn't support transparency, and I would guess is rendering everything as matte objects, meaning the renderer probably isn't robust enough to handle anything coming close to complicated reflections/refractions and so on, making this software's usefulness very situational, currently. It would be a great way to quickly populate photos with hordes of smaller objects, for example. However, with a more powerful renderer, feature wise, this tech could be really useful for the Photoshop crowd. I wish Autodesk/Mental Ray would focus on stuff like this instead of the boring crap updates we usually get (Maya's new fluids are pretty cool though, tbh...).
a jury weighs the merits of that claim
Unfortunately, I wouldn't trust the average juror to weigh a head of lettuce.
-
- - You can't take something off the Internet! That's like trying to take pee out of a swimming pool.