Slashdot Mirror
Adobe Demos Photo Unblurring At MAX 2011
karthikmns writes with word of an amazing demo presented last week at Adobe's annual MAX convention. You'll have to watch the video, but the enthusiastic crowd reaction seems genuine (or at least justified), even in an audience full of Photoshop enthusiasts, as photographs are algorithmically deblurred. (Maybe in the future, cameras will keep records of their own motion in metadata to assist such software efforts, rather than relying on in-built anti-shake software.) No word about when this will turn up for consumers in anything besides demo form, but I suspect similar software's already in use at Ft. Meade and Langley.
I'd be able to see the demo!
Did you watch the video? It makes unreadable text readable. That falls into the category of making missing data suddenly appear.
This demo came during the 'Sneak Peaks' portion of the conference. The technology may never make it to market.
That being said, I was at MAX and the demo was as amazing as it looks. Essentially, the software determines the motion/jitter of the camera at the time the photo was taken (i.e. figures out what caused the blur) and then undoes it. I can't imagine why they wouldn't include this in future version of photoshop.
simply the end of photography. bleh.
If you think photography is simply about getting an un-blurry image, you know nothing of photography.
How hard can it be, I mean, they've been doing it in movies since at least the 80s. Hell even the $500 dell desktop on CSI:miami can do it.
"we've got a convenience store video feed of the getaway car, the camera was recording in 480i from 300 yards away"
"can you sharpen it up a little?"
"sure. one moment... ok got it. License plate is california JGL-711. Ok just a bit more... yeah, looks like registration expires march 2012. Wait, let me clean it up some more, yeah it looks like there's a small identifying scratch on the trunk lid about a half inch long shaped like a boomerang. Oh wait, this is the new version of the software, let me zoom in a bit further, yeah I'm pretty sure I'm seeing loose skin cells on the edge of the trunk lid, maybe our missing person is in the trunk!"
"good work, now where's my sunglasses?"
yeeeeeeeeeeeeeeeeeeeeeeeeeeeaaaaahhhh!
I'll think I'll reserve judgement though until I can see it "for real".
Who were the annoying guys off to the side that loved hearing themselves talk? Really kind of ruined the momentum. This isn't MST3k
I think it would be better to say that [most of] the data are already present; the data just happen to be initially in an unwanted form.
This will make things LOOK pretty. It won't make missing data suddenly appear. At best it will make something ugly LOOK a little better. But that's just a computer-generated illusion, not a reflection of reality.
Yes and no - No, you can't magically create information not present in the original image.
You can, however, calculate the most likely "true" value for any point on the image, given a sufficiently accurate model of the distortion present (in this case, an incorrect focal plane). Whether or not that allows any functionally useful interpolation, you can still get a pretty good ID from a few dozen pixels squared, if sufficiently clear (think Slashdot's Gates-Borg icon).
Yes indeed, we can't add a few bytes of accelerometer data to a 10 megapixel image, that would make those images too large!
ASCII stupid question, get a stupid ANSI
Interpolated missing data is still just a fiction
Yes it is. However, in the present case they reuse *existing* data. Requires a lot of processing, but this is not an artifice, they really use computing power to first determine the camera movement, and from this information, they can re-compute each pixel by removing as much as they can the blur resulting from the camera movement, which gives much better results than using a straightforward generic edge enhancement.
This does NOT fix images that are out of focus. This fixes motion blur. The two are entirely unrelated.
Take off every sig. For great justice.
It's not entirely about whether a digitally de-blurred image is more accurate than an inherently sharp image.
It's also about whether the sensory impression made on a human by a digitally de-blurred image is a more accurate a model of the reality than the sensory impressions made by a blurred image. Of course your occipital lobes do plenty of interpolating of their own, so surely the question becomes which system (digital or organic) produces the more reliable interpolations.
Maybe if a person studied the blurred picture long enough they could have reconstructed that telephone number, hard to say from the mediocre quality video of a small display window.
I certainly wouldn't discount software being able to outperform even a trained person in sufficiently narrow fields, and I doubt you would either.
The data here isn't missing, just obfuscated. A common lab for EEs is to unblur a picture in this way. It's much harder in real life, where you need to find what they're calling the blur kernel, but there's no reason for it to be impossible.
We've known about deconvolution forever, the trick figuring out the path of the camera to generate the kernel for the deconvolution. In the TFV, he says we use the custom parameter file (that they probably spent months tweaking for each image), lots of computing power and TADA! unblurred image.
Microsoft had something similar a few year ago, where you have a blurred image and a second underexposed image to do the same thing. see paper here and examples here
Including that makes a cell phone camera just a little larger than desirable, and makes taking a snapshot take just a fraction more time.
Not really. It analyses the photo to determine the movement of the camera at time of shooting, and un-does that movement. Given enough colour resolution, i'm pretty confident that useful detail retrieval could be achieved - not just some artificial generated details to make the photo "look good".
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
No, and really no to everyone else. This is making _obfuscated_ data suddenly because visible.
It characterizes the the motion of the camera from the blur then reverses it: essentially an image stabilization algorithm. It's like making voices audible over loud music by figuring out what the song is and subtracting it from the mix.
It's cool, but not magic. They aren't even pretending to add in missing data like a CSI zoom. Nor does it even seem to take care of simple out of focus situations. So let's not get too excited, well, unless you've got a cheap/slow camera.
We have done this in my image processing class. It isn't CSI bullshit.
It won't make missing data suddenly appear.
The thing is that the data isn't missing, it is just distributed throughout the image. For example consider an unfocused camera. Instead of each point in the image mapping to a single point in the image, it results in a gaussian centered at that point, and these are all summed together. In signal processing terms, you can think of the blurred image as being the convolution of the desired image and a gaussian function (plus some noise):
xb = x # g + n
Take the Fourier transform, an this becomes a simple multiplication.
Xb = X * G + N
Divide it out, and invert the transform and you get the original image (plus noise):
xEst = X + n#(1/g)
The same can be done with motion blur, except now G is is a curve through space, not a gaussian. The hard part is knowing exactly how the image was blurred (what path it followed for motion blur, or the optic properties for unfocused images). I don't know what they are doing to discover the blur kernel, but it is impressive.
The other hard part is that this procedure can really amplify high-frequency noise, (consider dividing the 1/G; the tail of the gaussian is close to zero, which makes the reciprocal very large), and JPEG artifacts in particular are hell to deal with. There are ways to minimize this problem (for example, optimal Weiner filters), but you have to have a fairly high-quality image to begin with.
If you want to learn more the key terms to search for is image deblurring with deconvolution.
If people cared about file size bloat they wouldn't be purchasing the most megapixels possible.
After all, memory is cheap.
Don't know something? Look it up. Still don't know? Then ask.
Microsoft did a similar demonstration one year ago.
Zoom in on that raindrop. Enhance!
Of course. You can't get back more information than that is on the picture. But for a photograph it's enough that it looks good.
Wich reminds me another similar algorithm that worked on human faces. It could restore very lowres images to a sharp, almost perfect face. It's just that face was completely different from the one on the original picture.
It's cool, but not magic.
Right. I did exactly this with at least one ring image from Voyager 1's encounter with Saturn, and that was in 1980 (although I think I didn't get around to writing the code and actually de-blurring the image for two or three years after it was taken). I believe we used a VAX 11/730 to perform the computations.
FYI, Voyager pictures were 800x800 pixels, taken in monochrome with a filter applied in front of the camera. I don't recall whether this particular picture was a single image or a colour image taken with three filters. If the latter then there would have been an interesting twist: the three images would have been taken 48 seconds apart, so the spacecraft would have moved detectably from one colour to the next, so some semi-clever stuff would have been necessary to deblur three individual images and then merge them. But I honestly don't remember after all this time whether we had to do that.
This isn't new. There's a shareware plug-in, "DeblurMyImage", for it.
There are two main cases - focus blur and motion blur. Dealing with focus blur is well understood, because what defocusing does to an image is well understood. Motion blur is harder, because you have to extract a motion estimate first.
> I think it would be better to say that [most of] the data are already present; the data just happen to be initially in an unwanted form.
Not necessarily.
Some techniques of reconstruction use information that is not present. There's a video about reconstructing 3D images of people (with Tom Hanks as an example) which produces a 3D model from data in a picture _AND_ a database of preprocessed perspective angles of known stereotype 3D face models (ggl "morphable", video). I guess this is "thinking outside the box". Literally.
This does NOT fix images that are out of focus. This fixes motion blur. The two are entirely unrelated.
Except that both are examples of convolution and deconvolution. In motion blur, the convolution kernel resembles a straight line in the direction of motion. In unfocused images, the kernel has circular symmetry. I used to write simple deconvolution algorithms about 10 years ago, but only for motion blur, where the kernel was easy to find from the conditions in a well-defined industrial setting. Unfocused images are harder to deal with, because the convolution kernel goes to zero at certain intervals, so information is destroyed.
As mentioned in my other post, here are some examples of more sophisticated image reconstruction from many years ago. When the kernel is unknown, the image can still be reconstructed using statistical techniques (basically because the kernel is the same for all points in the image).
Escher was the first MC and Giger invented the HR department.
But that's just a computer-generated illusion, not a reflection of reality.
Technically, any picture you take with a digital camera is just a "computer-generated illusion", even in raw format. It's not reality. It's a programmer's interpretation of the data from camera's sensors. The camera's sensors detect a different range of light than the human eye, filters are used to try to keep it within the human range. And when you view the picture, what are you going to use to view it with so that the picture is displayed in it's truest form?
Even the images you see with your own eyes are subject to the way your brain handles the data from your organic optical sensors. And then there's the issue with how your brain handles storage of that image. When you view a picture on your monitor, there's probably at least three levels of virtualization separating you from reality.
You'll go crazy trying to record reality, and eventually you'll find it's overrated.
Yeah, this is standard math. A completely out-of-focus picture actually contains nearly as much information as a sharp photo, it's just smeared with a reversible mathematical transform called a point spread function. Reverse it and you get the in-focus image back. There have been third party programs to do this for about a decade. The main problems have been processing speed (it could take a half hour or more a decade ago), determining the point spread function (you have both focus and camera shake, and the former can make figuring out the latter really hard), lens/sensor defects and image format compression (the PSF you calculate for a local region may not work well for the entire picture), and boundary conditions.
Yes. As far as I know the process is called deconvolution.
What is I think new about the thing that Adobe shows here, is that it doesn't just compensate for out of focus or other instrumental effects, but for camera motion. (yes I watched the video). It determines the likely motion the camera made during the exposure. and then uses that as some kind of matrix for deconvolution.
What makes that tricky compared to a classical point spread function that only includes instrumental effects, is that it's probably not uniform across the image. There's not just x,y,z movement, but also yaw, pitch and roll. The video is very poor, but that seems to be what this algorithm is able to detect and then correct for. As such it goes beyond what we've been able to do since before the Hubble telescope needed glasses. Hogbom clean algorithm and such are from the 70's. This goes beyond that.
RogerWilco the Adventurous Janitor
Unblurring without having any additional information has been done by academic software before using a process called blind deconvolution, look it up for some interesting pictures and videos.
It's a rather "expensive" (cpu-intensive) operation, and indeed having sensor data about how the camera has shaken during exposure would significantly help in restoring the image. Interestingly, even cheap smart phones with crappy cameras will often already have movement sensor on-board, so there are some possibilities to improve image quality right after taking a picture; all it takes is a bit of software. How long until someone here whips up an improved Android camera app?
I'm probably under-informed, but I haven't heard of any cameras with full-blown movement sensor, although I know some of them can work out portrait vs landscape by now. Sounds like camera manufacturers have some catching up to do in the hardware department.
Visit http://ringbreak.dnd.utwente.nl/~mrjb/growingbettersoftware to download your free copy of the book
This uses a single image as input, and tries to determine a local prior (L) and a motion kernel (f). It switches between optimization of each in turn, and produces results similar to the demo seen in the video. Given that Aseem works for Adobe, I suspect this work is now close to release.
Cheers,
Toby Haynes
Anything I post is strictly my own thoughts and doesn't necessarily have anything to do with the opinions of IBM.
I attended an interesting presentation in 1990 on transmission electron microscopy being used to determine the structure down to an atomic level of a growing area of a tooth. Calcium and other atoms of interest are far too small to image so you get something blurry. The structure was determined computationally by working out what a series of different structures would look like after being blurred by the limited resolution of the microscope and then comparing that to bitmaps of the captured images.
I know that is a different approach but people have been working on getting information from defocused images for a long time.