Slashdot Mirror
Using Photographs To Enhance Videos
seussman71 writes with a link to some very interesting research out of the University of Washington that employs "a method of using high quality photographs to enhance a video taken of the same subject. The project page gives a good overview of what they are doing and the video on the page gives some really nice examples of how their technology works. Hopefully someone can take the technology and run with it, but one thing's for sure: this could make amateur video-making look even better than it does now." And if adding mustaches would improve your opinion of the people in amateur videos, check out the unwrap-mosaics technique from Microsoft Research.
Why is UW not releasing their source code? If they intend to spin off commercial products, why are they releasing demos? Hell, even *Microsoft* is releasing demos of this stuff.. is Apple and Google the only companies that can ship product these days (even if it is "beta" you can at least freakin' use it).
No more demos. We know you're smart, now make something useful please.
How we know is more important than what we know.
The other cool part of it is that it derives a cloud of points from the video, meaning it can turn a video into a 3D model, apparently. However it seems like their program only uses it internally.
You just got troll'd!
One of the researchers on that Microsoft project is:
Pushmeet Kohli
Which pretty much makes every other name out there a piece of shit.
I always get this feeling when I see a university-styled promotional release that the *software* patents are already pending.
I haven't the time to search just now but I'll bet there's at least one application pertaining to this method which encompasses a hell of a lot more.
Ripping an new rectum in the fabric of spacetime.
Wow, that's got to be some of the coolest tech I've seen in years. I can't wait for some software to come out that uses it. Avisynth plugins, anyone?
Combine this with fractal compression and we could store all the videos we've ever seen on one hard disk.
They ARE out to get you simply because They are in it for themselves and they don't care about you.
Really, the ability for their software to 'unwrap' a 3D object and allow you to fiddle with it as you wish is very cool.
And not limited to a 'static' scene.
And, really, if you're going to go to the effort of videoing a scene, then photographing the scene, then passing the video and the photos through their software. All to get better exposure and resolution.
Um.
Wouldn't it be a far better cost/effort equation to just buy a better HD camera in the damn first place?
The more interesting aspect is that you can tweak those still photos, and then transfer them back. Photoshop some key frames, and you have suddenly created a video with the same manipulation. The video is just a cheap source for spatial data, which you can then texture with your photos.
The easy way would be to use the already calculated depth field from the frame in the video that best matches the photo.
Work Safe Porn
All of these techniques are pretty awesome and will certainly be a boon to home video enthusiasts the world over (plus plenty of commercial places that are on a tight budget), but I've got another idea.
You see it on TV all the time, CCTV footage of robberies and the like, couldn't this technology be used to effectively map out a 3D image of the purpetrator?
I know it wont be perfect and most CCTV is probably too low quality to be used, but it would certainly be pretty cool (and useful) to have a vaugely accurate 3D model of the guy, giving you height, build, etc. and with the help of supplementary images, a really easy way to adjust it's appearance.
+1 IDisagreeSoHeMustBeATrollOrAnAstroturferOrAShill
So my by reckoning, it'll only be a few years before the stuff featured in CSI and similar shows isn't *total* bollocks. A lot of people see photos and stuff being magically enhanced to perfect resolution, revealing the identity of some suspect, and think that can be done in real life.
No not at all. You can get a good resolution out of a video if and only if you have a good picture to go with it, and it doesn't bring out any more data than you have in both your picture and video, it only combines what they have. Infinite zoom with a mere security camera as shown on CSI is theoretically impossible, well, with the exception of aliasing-based super-resolution techniques, but that sort of processing is done on many frames, takes the necessary aliasing and probably a few other conditions like noise level and what not.
You just got troll'd!
All to get better exposure and resolution.
Clearly it's intended for pr0n!
There is no -1 Disagree mod. Slashdot.org/faq defines mod options. USE IT.
This has been posted on Slashdot before.
That would greatly lower the cost of doing special effects, if you didn't have to do them frame by frame.
http://blindscribblings.com - Tasty pop-culture in conceptual fashion.
Wouldn't it be a far better cost/effort equation to just buy a better HD camera in the damn first place?
What about you do buy a HD camera and combine it with a 12 megapixel still camera? Besides just a HD camera doesn't fix the issues you can fix by then adding HDR shots to the mix.
You just got troll'd!
When the ability to deconstruct a video into a 3-d model & skin (the opposite of what a video card does now) is placed into an open-source API, the possibilities are going to be HUGE (and a little frightening).
Anyone want to post a few ideas? I'll give you a few topics to kick things off:
Change detection (Finding lost objects in a room, seeing boxes left in a government office, where's my remote) ...)
Change observation (plant growth, things that change too gradually for us to notice)
Creating 3-d models from humans (extracted from old films, walking down the street)
Weapon systems (Undetectable lasers blinding targets, Unmanned guns with perfect accuracy)
Home interaction (Make a sign with your hand, computer changes the channel, lighting, heat,
Office monitoring (Exactly where each person is any time just by typing "Where's bill" into your PC)
All things that could be done by any hobbiest/hacker with the right API.
(I assume that to get real-time you could use the massively parallel abilities of a video card, making this stuff run on any hardware...)
Also, just storing models and skins is extremely efficient--You could film a room for years in extremely high resolution and use virtually no storage (almost none except when something or someone new enters the room, then just one new high-def skin)
Other ideas?
The whole time they were saying SpaceTime Fusion, I kept refraining myself from shouting "CAPT'N, D'SPACE TYM' FOOSH'N RIF' IS GETT'N MOR' INTENS'!" in a Scottish accent.
:(
I miss Scotty
With most if not all video cameras storing the video digitally, and now with all these new techniques for editing video, why would any court allow for video evidence?
This software, if it actually works as described, could also be used to easily fabricate video "evidence". An average viewer would not be able to tell the difference.
Kinda scary...
--- sig moved for great justice.
I believe IBM did something like this is the 1990s. Obviously not as slick, they didn't have as many CPU cycles.
automatic or not - that was a huge eye-opener if that technology is available at the grad-student level. Available in commercial/comsumer products in 3-5 years.
So much for "video evidence". So much for reality. "Your honor, I have a video of what happened and I wasn't there! ... see?"
Did anyone else notice a very creepy effect in the "enhanced" video with the bust? It made it look like the head was turning to look at you as you moved around it. *shudder*
Then just have the still camera do continuous shooting @ ~1fps while you video. Match them up in this software when you're done and you're good to go...now if I could just get a hold of their software.
I assure you that in all but the most insane cases, doing that frame-by-frame is already not done anymore (high quality rotoscoping, on the other hand.. yipe). You model a quick mock-up in a 3D application, project your painted texture onto that, and composite that with the original footage.
What it does do is remove that whole 'You model a quick mock-up' part in many (not all) cases. Now to see who gets the patents, how much they are to license, and who get(s) to toss it into their editing suite.
Still cameras can pretty much always take higher res pictures than video cameras no matter what price range you're looking at. I wonder if they could combine a still camera into a video camera and have it take high res still frames as the video is shot and then use this software to improve the video. It seems that'd be a cost effective way to squeeze more out of any level of camera.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
This is a 3d track of the shot (which generates a point cloud of 3d points, which can then be used to generate an automatic 3d model of the scene). They then project (a method of texturing that paints a model based on points of projection.. what happens when you stand in front of a projector- you get projected onto) the still photos onto the 3d model, recreating all aspects of the texture and geometry, but instead of SD resolution, you now have gigapixel resolution built into the model.
The reason it looks like a cheap video game is exactly that, they're trying to prove how sharp it is, so instead of anything being anti-aliased etc it's all crisp- which doesn't look like real life.
Solution: get a better video camera, learn how to expose your shots properly.
Oh? And the tree thing? same thing, except instead of projecting the texture on, you just place the texture in the 3d scene where the tree is, and render- it's even easier.
Solution: Don't film a beat up tree. Don't film flowers with a giant sign in the middle of them.
This wasn't at SIGGRAPH this week. As a paper or as a poster- of which there are PLENTY of student posters.
The solution is NOT to fix it in post. The solution is to spend 5 minutes, think it through, and fix it while you're filming.
www.GrenadeHop.com
It's all still a bunch of digital voodoo-black-magic trickery at work.
You want to impress me? Try planning ahead and shooting the video right the first time.
1) Tape myself with cheap whore
2) Combine with picture of Claudia Schiffer
3) Become popular!
Wouldn't it be a far better cost/effort equation to just buy a better HD camera in the damn first place?
I hate to be captain obvious here, but historical footage strikes me as the #1 reason (historical meaning everything up to yesterday). I mean you can't go back and reshoot the millions of hours of footage the world already has, but there's a lot of high resolution photos of the some of the same subjects.
Secondly, the still resolution on most point and shoot cameras is a lot higher than the video resolution, and probably will be for the foreseeable future. Good software plus a $200 camera seems like a better solution for 99% of people than a $2000 camera which is still going to have problems of its own.
Thirdly, about 4 minutes into the video they start getting into the really interesting stuff about artifact removal (which is still going to be a problem no matter how nice a camera you have) and object removal, which is just well, really cool.
The sending of this message pretty much inconveniences everyone involved.
This might be helpful to deal with copyright-protected material that gets into frame, for instance, billboards, logos on T-shirts, posters and art-work on walls. Take a single frame into a photo-editor, replace the unauthorised image with an authorised one, and this technique could potentially replace it throughout the sequence. It could equally be used to replace moving images, for instance on a TV screen, with a "blue screen" (or green), that normal video compositing software can then replace with a desired image.
photo hunt has been doing this four years. let me know when something new comes up regards, mike
Wouldn't it be a far better cost/effort equation to just buy a better HD camera in the damn first place?
Hint: years old amateur/family/etc video meets modern high-res camera.
This blows my mind almost as much as Melodyne version 2 does: http://www.celemony.com/cms/index.php?id=dna Only instead of 'direct note access' it's 'direct video object access'. Or something.
This is getting really close to being something that every straight male porn viewer (which means every straight male on Slashdot...) would pay a lot for.
Combine the ability to remove items from a video, like they showed with the lamppost and sign in the flower shop, with the ability to insert new things into the video, and you could turn a boring man on woman porn movie into a lesbian twin incest movie.
Actually two bad quality images can be used to make a better quality image.
if you're going to go to the effort of videoing a scene
;) you can enhance any video. /. users stop bitching about "this isn't very useful" and "i don't see the point"? It's not useful now, but it will be, otherwise nobody would invest in it and I think people who pay tons of money for this kind of research know a bit more about what research is good for than you but unfortunately they're too busy making money and changing the world to spend their time on /. The fact that you don't see it's point means only that you don't see it, it doesn't mean there is no point. In stead of saying "this isn't useful" why don't you ask "what could this be used for?" Maybe that change in some people would help us progress faster because they will question the applications of certain research which causes debates which lead the faster progress (not at the time of the debates but a couple of years later people draw conclusions and they start to get along and pretend they never asked dumb questions). It would also encourage researches by showing them that if they give you applications for their work you might embrace it. But NOOOOOOOOOOOOOOOO you just say "i can't do anything with this" which basically insults and shuts the trap of anyone who might give you a couple of uses and it just starts a flame war which is basically a debate focused on insults and swear words in stead of what it should be - focused on pros and cons.
"you..videoing..." isn't the only application. This could be used to enhance other videos. Let's say someone else made a great video (captured some really great scenes, focused on some details) and you want to publish it but even if they focused on cool details, they're not enough. You take a few pictures and enhance their video.
Also, this is just the start. They are currently enhancing static videos but I'm sure in the near future, if this is worked on enough, it could be used to enhance any kind of video scene. So you have something interesting happening - by some people's standards, two squirrels fighting over a nut is interesting enough - but the overall quality of your video is just awful. You won't be able to re-take the shot because the squirrels canceled their contract so you'll take some pictures, match them against the video and voila, high-quality video or a forest and two squirrels kicking each other in the nuts over a nut while filmed by a nut.
Do you by any chance remember those huge radios? I mean those REALLY huge radios weighting about 50kg? They weren't very practical and to the final consumer they were cool but they were heavy and incredibly expensive. Now I carry an MP3 player in my pocket that also has an FM radio integrated, just for the hell of it. === POOF === 20 years later === Do you by any chance remember those projects that they started, to enhance videos of static scenes using photographs? There was an article on a site named "Slashdot" which was taken down after it started WW3... I doubt that you'd remember that, but look where we are today: with a couple of high-quality pictures (100 gpixel
This is what this whole project is about. Studying something cool and then enhancing it until it gains practical applications. Why the hell won't
It's not exactly related but Al Jazeera just had a piece about a pedophile who got arrested last year after interpol "unwarped" some picture he had put online.
Maybe those new tech might be used to produce that kind of useful result and not only better pops and moms holiday pictures..
Old article: http://www.guardian.co.uk/uk/2007/oct/19/ukcrime.internationalcrime
It really depends on what you call "bad quality".
You just got troll'd!
These two (article) technologies IMHO are more important than Photosynth
http://www.youtube.com/watch?v=556FvXHLtAo
Lets just say your camera is capable of 10MP, and that an average photo is 3MB(jpeg). Then 3MB x 60(secs) x 60(minutes) = 10800MB per hour I hope your camera can keep up with the write rate (plus the fact that your expecting it to always be in focus for the 3,600 photo's you've just taken).
I made something from the same category as my bachelor thesis. This is same as using refrence shape and creating Virutal studio. But this is beeter using of matchmoving. Because they use informations from the sceen (plus photos). Usage would be great. But I know there is loads of math and coding behind, so it is common that it is not freeware. Aplication like this is not a browser that you can code time after time. This is full time job that can not have a longer breaks. Same possibilities were described in An Invitational to computer vision in epipolar geometry. Idea is quite old but implementations are very rare.
Could this be used as a form of video compression? Shoot your video at high resolution. Extract a few high resolution stills from the video, and then convert the video to low resolution. Save the low res video and the stills. When you want to play the video, use their algorithm, with the stills taking the role of the photos, to enhance the low res video.
The paper states that the scene is restricted to static scenes, specifically it can only process scenes where the video only contains changes with respect to time dealing with camera motion, specular reflections and other camera-angle dependent effects, or scene lighting. Things like scene geometry changes (filming a moving human being) can possibly be done in the future with a better algorithm for depth via motion, etc.
In these specific conditions, it seems like it would work pretty nicely.
I could see a simple laser or ultrasonic depth finder being integrated into cameras to be used as a "ball-park" estimate of depth for that specific frame, relative to where the pointer is located, of course, possibly increasing accuracy if the algorithm could utilize it.
While the techniques for building a 3d model (used at least internally) from a 2d video shot are undoubtedly impressive, they quite possibly parallel SAR techniques that have been around for decades. Are you listening software patent examiners???
It would greatly lower the cost of doing special effects if you did special effects frame by frame.
The problem with most of these technologies is that they never reach photo-realistic visual effect quality results.
Any time you have to do frame by frame VFX you're doing it for the sole purpose of getting a more perfect result. If you need average to crappy results you won't be doing it frame by frame.
This tech has cool potential and will be used by the VFX industry but it won't be automatic and it'll be to augment existing techniques.
Um... geeze... a little impassioned perhaps?
All I was saying was that the Microsoft tech, which got the small billing, looked to be the more interesting and useful now compared to the other tech which got prime billing.
You know, because Microsoft is 'evil', so doesn't get attention, but a University is wholly 'good' so gets top billing.
And also, you're telling me there aren't a heap of research time spent on ways of doing things that are overly complex and impractical just because the researchers want to do so, when there is, in fact, a far simpler approach?
Yes, some of those ridiculous ways of doing things are done for the learnings involved, but that doesn't mean we all have to ooh and ahh over them while they're in their research mode.
I don't think anyone looked at a gigantic radio back in the day and said 'Man, that's useless, that lets me listen to stuff from thin air'. They didn't say that because that was all there was.
I can look at something like this and feel that it's slightly less useful because it's prime two applications are 'better res and exposure' which are indeed already available using a better camera, and 'removing unwanted things from the video'... which is something I quite detest in the world of digital photography and the like today (let's not show the world as it is, warts and all, let's scrub it until it's the sterile version we hoped it would be). And while it has applications in film making, well, they wouldn't be using the consumer cameras this is all aimed at.
Just chill out, I don't fail to see the smarts in this, don't go ape over anyone who doesn't like a bit of tech you do.
And don't think that the views of anyone on Slashdot represent the 'norm' at all.
If they did, we'd all be using command line only interfaces on linux with no driver support for anything and having to code anything we wanted in a new software product ourselves, because, well, if we want it, 'why don't you fix it yourself then?'
Can you buy an HDR video camera? The technology enables effects not possible with current video cameras. Plenty of people don't have the means to "just buy a better HD camera in the damn first place"