Slashdot Mirror
Adobe Demos Photo Unblurring At MAX 2011
karthikmns writes with word of an amazing demo presented last week at Adobe's annual MAX convention. You'll have to watch the video, but the enthusiastic crowd reaction seems genuine (or at least justified), even in an audience full of Photoshop enthusiasts, as photographs are algorithmically deblurred. (Maybe in the future, cameras will keep records of their own motion in metadata to assist such software efforts, rather than relying on in-built anti-shake software.) No word about when this will turn up for consumers in anything besides demo form, but I suspect similar software's already in use at Ft. Meade and Langley.
I'd be able to see the demo!
I suspect similar software's already in use at Ft. Meade and Langley.
I certainly hope not.
This will make things LOOK pretty. It won't make missing data suddenly appear. At best it will make something ugly LOOK a little better. But that's just a computer-generated illusion, not a reflection of reality.
SJW: Someone who has run out of real oppression, and has to fake it.
It's a pretty sharp idea.
Here's the PC Weenies cartoon about this one...
Did you watch the video? It makes unreadable text readable. That falls into the category of making missing data suddenly appear.
Zoom in.. Zoom in.. ENHANCE!
simply the end of photography. bleh.
This demo came during the 'Sneak Peaks' portion of the conference. The technology may never make it to market.
That being said, I was at MAX and the demo was as amazing as it looks. Essentially, the software determines the motion/jitter of the camera at the time the photo was taken (i.e. figures out what caused the blur) and then undoes it. I can't imagine why they wouldn't include this in future version of photoshop.
A staged demo using images that lend themselves to the kind of interpolated guesswork that this uses is one thing. Making it work with real-world forensics is quite another.
http://alternatives.rzero.com/
Maybe in the future, cameras will keep records of their own motion in metadata to assist such software efforts
Because we all could use just a little more file size bloat. After all, memory is cheap, right?
Seven puppies were harmed during the making of this post.
But that's just a computer-generated illusion, not a reflection of reality.
I wouldn't go that far, I'd say it's more like making an educated guess -- and while it's true that a guess is a guess and you should never take it for fact, a guessing tool that is consistently 95% accurate is still incredibly useful, even if just to narrow down the places that humans should then go and investigate by hand.
I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
At best it will make something ugly LOOK a little better.
Or in the case of cosmetics ads, make something that looks good look a little uglier.
Seven puppies were harmed during the making of this post.
It doesn't sound that much harder of a problem to solve than what I learned in EE undergrad about deconvolution. Divide the Fourier transform of the blurred image by the fourier transform of the "motion kernel" as they call it to get the sharpened image. I routinely use a similar method in the lab to correct for visual aberrations in my diffraction spot imaging equipment, but there the problem is much easier as the motion function is exactly traced out by the diffraction spots.
Perhaps getting the "motion kernel" is harder than I suspect it to be in a real life scenario, though.
How hard can it be, I mean, they've been doing it in movies since at least the 80s. Hell even the $500 dell desktop on CSI:miami can do it.
"we've got a convenience store video feed of the getaway car, the camera was recording in 480i from 300 yards away"
"can you sharpen it up a little?"
"sure. one moment... ok got it. License plate is california JGL-711. Ok just a bit more... yeah, looks like registration expires march 2012. Wait, let me clean it up some more, yeah it looks like there's a small identifying scratch on the trunk lid about a half inch long shaped like a boomerang. Oh wait, this is the new version of the software, let me zoom in a bit further, yeah I'm pretty sure I'm seeing loose skin cells on the edge of the trunk lid, maybe our missing person is in the trunk!"
"good work, now where's my sunglasses?"
yeeeeeeeeeeeeeeeeeeeeeeeeeeeaaaaahhhh!
I'll think I'll reserve judgement though until I can see it "for real".
Who were the annoying guys off to the side that loved hearing themselves talk? Really kind of ruined the momentum. This isn't MST3k
I think it would be better to say that [most of] the data are already present; the data just happen to be initially in an unwanted form.
..just came, simultaniously
This will make things LOOK pretty. It won't make missing data suddenly appear. At best it will make something ugly LOOK a little better. But that's just a computer-generated illusion, not a reflection of reality.
Yes and no - No, you can't magically create information not present in the original image.
You can, however, calculate the most likely "true" value for any point on the image, given a sufficiently accurate model of the distortion present (in this case, an incorrect focal plane). Whether or not that allows any functionally useful interpolation, you can still get a pretty good ID from a few dozen pixels squared, if sufficiently clear (think Slashdot's Gates-Borg icon).
The process is called deconvolution, but it does not work always that great...
Here is a freeware http://www.zen147963.zen.co.uk/
Surely you mean Ft. Belvoir.
Its important to realize that this technology removes motion blur but does nothing about an out of focus image. Fixing motion blur is (clearly) entirely feasible by figuring out how light that was supposed to fall on one pixel was spread out across multiple pixels based on the motion of the camera and then removing that extra light data from the wrong pixels and putting it into the correct pixels. I don't know if this is how they do it but it is how I imagine it would be done, and it is amazing that they can get the results after a couple clicks.
Interpolated missing data is still just a fiction
Yes it is. However, in the present case they reuse *existing* data. Requires a lot of processing, but this is not an artifice, they really use computing power to first determine the camera movement, and from this information, they can re-compute each pixel by removing as much as they can the blur resulting from the camera movement, which gives much better results than using a straightforward generic edge enhancement.
He didn't yell ENHANCE!
Enhance 224 to 176. Enhance, stop. Move in, stop. Pull out, track right, stop. Center in, pull back. Stop. Track 45 right. Stop. Center and stop. Enhance 34 to 36. Pan right and pull back. Stop. Enhance 34 to 46. Pull back. Wait a minute, go right, stop. Enhance 57 to 19. Track 45 left. Stop. Enhance 15 to 23. Give me a hard copy right there.
-- I ignore anonymous replies to my comments and postings.
I think they'll use this technology for sure to unblur the movies.
Well if you watch the demo, the presenter makes unreadable text on a poster readable after the filter is applied.
But the point is that in a normal blur situation, very little data is missing (unless the degree of blur is catastrophic). A simple model for blurring is that it happens because the camera was in motion relative to the subject when the photo was taken. All the data is still there, it's just smeared into the wrong pixels due to the motion. If you are able to guess the direction and speed of the motion of the camera at the time the photo was taken, you can put it back into the right place and reconstruct the photo.
Now, there are some obvious problems. If some pixels are saturated, any data that was smeared into them will be permanently lost. Things near the edges of the photo may not be reconstructable. And in photos of very close-by objects, parallax may hurt results, especially near the edges of objects (this seems like a tiny effect to me, but I'm not an expert). But the point remains that almost no data is missing.
The algorithm does not claim to work on photos that are "blurry" because they were taken at too low a resolution.
They show what they call a 'blur kernal' the calculation of which takes up the majority of the processing time. I'm guessing they use that to perform a 2D deconvolution on the image (there is software that already does this). The interesting part is how they calculate the kernal.
Actually, it could quite reasonably make data that was there, but obscured in such a way that a human eye can't make it out, suddenly appear. The demo shows an example of taking a photo of a poster that's blurred beyond reading, and getting perfectly crisp sharp text back.
It's not entirely about whether a digitally de-blurred image is more accurate than an inherently sharp image.
It's also about whether the sensory impression made on a human by a digitally de-blurred image is a more accurate a model of the reality than the sensory impressions made by a blurred image. Of course your occipital lobes do plenty of interpolating of their own, so surely the question becomes which system (digital or organic) produces the more reliable interpolations.
Maybe if a person studied the blurred picture long enough they could have reconstructed that telephone number, hard to say from the mediocre quality video of a small display window.
I certainly wouldn't discount software being able to outperform even a trained person in sufficiently narrow fields, and I doubt you would either.
... will never ever need this feature.
Here's a datapoint for you - I can take a barcode (which is 108 pixels wide) and image it on a webcam, out of focus, at 90 pixels wide, and my software can correctly output the 13 digits that the barcode represents.
This doesn't break any information theoretic laws, but it does give me the intuition to stop before claiming that all clever image processing is just "interpolation" and that data is "missing".
http://people.csail.mit.edu/billf/papers/deblur_fergus.pdf
Yea. It's the "I'm just going to load some parameters" that is the "interesting" part.
You totally missed the clue. Boomerang scratch? Going under the I-37 bridge as the crossing arm lowers. We know EXACTLY where they are!
There are two types of people in the world: Those who crave closure
At best it will make something ugly LOOK a little better. But that's just a computer-generated illusion, not a reflection of reality.
So, it's perfect for pr0n. Who cares if it's fiction, isn't that a good thing?
This will make things LOOK pretty. It won't make missing data suddenly appear. At best it will make something ugly LOOK a little better. But that's just a computer-generated illusion, not a reflection of reality.
It's not that simple. What's likely happening here is two things: First, the image is analysed and the probable motion trajectory of the camera while the shutter was open is calculated. Then a convolutive algorithm is used to reverse the motion. This is entirely doable. The information is there in the image, the trick is just how to extract it.
A gaussian blur, for example, can be applied "backwards" and the sharp original recovered, if you know the parameters used. So if Adobe's motion trajectory analysis is good enough the results could be quite close to the truth.
I don't know the math well enough myself to prove that I'm correct, but I'm sure someone better educated in the area could confirm or deny my claims?
.: Max Romantschuk
The data here isn't missing, just obfuscated. A common lab for EEs is to unblur a picture in this way. It's much harder in real life, where you need to find what they're calling the blur kernel, but there's no reason for it to be impossible.
Too bad he didn't have the even newer version of the software or he'd be zoomed in on the DNA in those skin cells and running it against the all-knowing database.
We've known about deconvolution forever, the trick figuring out the path of the camera to generate the kernel for the deconvolution. In the TFV, he says we use the custom parameter file (that they probably spent months tweaking for each image), lots of computing power and TADA! unblurred image.
Microsoft had something similar a few year ago, where you have a blurred image and a second underexposed image to do the same thing. see paper here and examples here
More like de-streak. This isn't CSI technology come to real life. If you take a picture while moving the camera it will basically retrace the camera's movement to make a better picture of it.
"When information is power, privacy is freedom" - Jah-Wren Ryel
I work in image processing, and think this may be useful.
At the risk of a buzzkill, this is nothing more than deconvolution. This is used in science and behind-the-scenes in all sorts of industries to correct for the point-spread function (PSF) of a given imperfect system in the general field of 2-D digital signal processing, most often applied in imaging.
The innovation is that they have come up with an algorithm to analyze a given photo (or region) in reasonable amounts of time to find or approximate the PSF from blurring. After this is done, applying it (deconvolving) is a fast operation. Note that he calls what gets extracted a "blur kernel" and then goes on to explicitly call it a PSF as an aside in the presentation.
What this is NOT is magic. The information is there, it's just not sharp due to blurring; what should be consolidated in one pixel is now spread over a region. It won't fix every photo, won't bring back lost information, and isn't making data appear out of thin air. With those caveats, however, it is a powerful technique.
Not really. It analyses the photo to determine the movement of the camera at time of shooting, and un-does that movement. Given enough colour resolution, i'm pretty confident that useful detail retrieval could be achieved - not just some artificial generated details to make the photo "look good".
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
No, and really no to everyone else. This is making _obfuscated_ data suddenly because visible.
It characterizes the the motion of the camera from the blur then reverses it: essentially an image stabilization algorithm. It's like making voices audible over loud music by figuring out what the song is and subtracting it from the mix.
It's cool, but not magic. They aren't even pretending to add in missing data like a CSI zoom. Nor does it even seem to take care of simple out of focus situations. So let's not get too excited, well, unless you've got a cheap/slow camera.
You're also exchanging some pixel depth (color/greyscale levels) and information about the point spread function for spacial resolution. It looks like magic but it's a very old technique. I did similar in the mid 1990s. Capturing camera motion (accelerometer) during the photograph would be useful here since it helps you calculate the point spread function. To poster who worried about the memory this would require, we're talking about three long ints or 6 bytes for the X, Y and Z vectors. Since we're below $1/gigabyte for flash, you're talking about ~0.0000006 cents per photo. So, no I wouldn't worry about it. It would be ideal for iPhones and Androids which already have an accelerometer and the small image sensor means low shutter speeds which means lots of motion blur in dim light.
In the case of fixed-base (like security) cameras, there is very little camera shake that would blur the image. So tracking the motion of the camera (via 3-axis accelerometer for example) wouldn't help.
Unless you can compute separate motion vectors for each element in the image (think people walking in different directions, each face to deblur would have a different motion vector) this would not seem to improve the performance.
And, of course, the choice of motion vectors would have a huge impact on the reconstructed image. It could easily turn an identifying scar from "across" the cheek to "down" the cheek. I would hope the criminal justice system will have trouble proving anything based on photoshopped evidence.
--Joe
... the "enhance!" command. Yay!
We have done this in my image processing class. It isn't CSI bullshit.
It won't make missing data suddenly appear.
The thing is that the data isn't missing, it is just distributed throughout the image. For example consider an unfocused camera. Instead of each point in the image mapping to a single point in the image, it results in a gaussian centered at that point, and these are all summed together. In signal processing terms, you can think of the blurred image as being the convolution of the desired image and a gaussian function (plus some noise):
xb = x # g + n
Take the Fourier transform, an this becomes a simple multiplication.
Xb = X * G + N
Divide it out, and invert the transform and you get the original image (plus noise):
xEst = X + n#(1/g)
The same can be done with motion blur, except now G is is a curve through space, not a gaussian. The hard part is knowing exactly how the image was blurred (what path it followed for motion blur, or the optic properties for unfocused images). I don't know what they are doing to discover the blur kernel, but it is impressive.
The other hard part is that this procedure can really amplify high-frequency noise, (consider dividing the 1/G; the tail of the gaussian is close to zero, which makes the reciprocal very large), and JPEG artifacts in particular are hell to deal with. There are ways to minimize this problem (for example, optimal Weiner filters), but you have to have a fairly high-quality image to begin with.
If you want to learn more the key terms to search for is image deblurring with deconvolution.
This will make things LOOK pretty. It won't make missing data suddenly appear. At best it will make something ugly LOOK a little better. But that's just a computer-generated illusion, not a reflection of reality.
Not really. If I take a picture of a sign and my hand is shaking as I take the picture, you already know it's a fixed, flat surface. If you can algorithmically find exactly how my hand was shaking and so clear up the image, that's not an illusion. Consider it more like that even in the short shutter time you have many images superimposed on each other, this aligns them so the picture becomes clearer. It won't work for the generic case but for a certain class of pictures this is close to magic.
Live today, because you never know what tomorrow brings
It makes unreadable text readable. That falls into the category of making missing data suddenly appear.
I don't know the details of that feature, but wouldn't be surprised that the "repair" mode comes into two flavors: automatic and manual
- the automatic mode would try to guess from embedded data like Exif and the picture itself what it is supposed to show.
- the manual mode would ask the user if the part to be repaired is text, landscape, face...
In the case of a text, the algorithm could either compare each letter with the ones it knows (from fonts), or words from a dictionary - the user would select a language first.
I don't believe in miracles.
Slashdot, fix the reply notifications... You won't get away with it...
Microsoft did a similar demonstration one year ago.
This could be usefull in low-light situations where you don't have a tripod, but yeah, it's not adding in any new information, it's just better organizing the data that's been captured. Also, I doubt it would be able to do anything for a specific subject that was moving during the shot, rather than the camera.
Those are digits in the 0-10 range right? So you have a key space of 10^13, which is 0x918 4e72 a000, or 44 bits. So the 90 pixels could easily be more than enough. My information theory is a bit wobbly, but I think that works. Unless your encoding is horribly inefficient you've got plenty of samples there. Also if you're being smart you could probably read the barcode diagonally to gain some extra information.
Enhance....enhance...enhance...enhance...enhance
That's not true; the data is there. This is not for OOF (out of focus) photos due to focus error (so if you have a "bad copy" of a lens, or you didn't micro-adjust your lens, or if your lens is decentered, or you simply blew it on the shot it won't help), but for motion blur. All of the data is there but exposed multiple times as the subject moves. Topaz Labs has had a plugin for this a while and it works pretty well.
In the case of focus errors, you're right; the data isn't really there in a way to help this algorithm work.
The Christian Right is Neither (Christian nor right). See: Matthew 23, Matthew 25, Ezekiel 16:48-50
The data in csi is not obfuscated it's missing. This algorithm reverses motion blur it does not bump up any resolution, these are two completely different things.
Zoom in on that raindrop. Enhance!
I hereby challenge them. Their software versus my fast moving kids who often show up in photos as blurs. I think kids have built in sensors to let them know precisely when a camera is going off, thus enabling them to move at the exact moment to blur and/or ruin the photo.
My sci-fi novel, Ghost Thief, is now available from Amazon.com.
Each pixel element consists of at least three eight or sixteen bit values, so that's at least sixteen million values per pixel. Assuming monchrome, thats 256 or 65536 values per pixel.
I did some experiments with those digital photograph postcard printing booths and a scanner. You could easily create an image, print it out, and scan it back in again to recover a few hundred kilobytes.
The trick is to create a couple of horizontal and vertical bands of coordinates encoded in binary. Then you can see what the recoverable resolution is.
Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
Great video that basically does what you're saying: http://www.youtube.com/watch?v=KUFkb0d1kbU
and of course just for fun: http://www.youtube.com/watch?v=Vxq9yj2pVWk
Of course. You can't get back more information than that is on the picture. But for a photograph it's enough that it looks good.
Wich reminds me another similar algorithm that worked on human faces. It could restore very lowres images to a sharp, almost perfect face. It's just that face was completely different from the one on the original picture.
It looks like the filter looks for images which have been taken by slow ccds or similar where someone moves the camera / phone while they're taking the pic. The image is in focus but the exposure time is so long it gets smeared. The analysis appears to figure out how the camera was moving during the image capture and reverse that. It's very clever but it would be nice to see some genuine before & after shot without someone's shakey audience cam, YouTube encode on top. Also, the issue of "parameters" would be nice to understand. After all, I could supply some "parameters" to transform 1 bit of data and into the complete works of Shakespeare. Some fine tuning parameters are okay but if it's dozens or hundreds then maybe not so much.
It's cool, but not magic.
Right. I did exactly this with at least one ring image from Voyager 1's encounter with Saturn, and that was in 1980 (although I think I didn't get around to writing the code and actually de-blurring the image for two or three years after it was taken). I believe we used a VAX 11/730 to perform the computations.
FYI, Voyager pictures were 800x800 pixels, taken in monochrome with a filter applied in front of the camera. I don't recall whether this particular picture was a single image or a colour image taken with three filters. If the latter then there would have been an interesting twist: the three images would have been taken 48 seconds apart, so the spacecraft would have moved detectably from one colour to the next, so some semi-clever stuff would have been necessary to deblur three individual images and then merge them. But I honestly don't remember after all this time whether we had to do that.
Dammit .. now the CSI "Can you clean that up?" question is yes, and people will continue to expect miracles from technology.
We emerge from our mother's womb an unformatted diskette; our culture formats us. - Douglas Coupland
For prior free offerings, see http://www.simg.de/index.e.html or http://sourceforge.net/projects/iterativedeconv
Oh, I'm sorry sir, I thought you were referring to me, Mr. Wensleydale.
Well, sort of. This class of algorithms is called "deconvolution", because the cause of the blur can be interpreted as convolution.
The information is already there, it's just not apparent. On the other hand, that's so trivially necessary that it isn't worth mentioning, and your interpretation remains: You couldn't see it before and now you can.
Deconvolution isn't easy and there are many common signals where it doesn't work. In digital photography, the most problematic signals are pictures with overexposed parts, for example bright sky and reflections of lights. Smeared highlights can not be deconvoluted back to sharp highlights. That information is lost. Instead you get dimmed streaks of the highlight over an otherwise nicely sharpened picture.
Eventually you zoom in beyond the Plank distance and know the mind of God. All crimes are simultaneously solved, and the dominion of quantum peace prevails.
I wrote a deconvoluter once. Really nasty simple one, using sheer processing power instead of effective math. Worked, but too slow to be of any practical use. Weeks per frame.
Zoom in on that raindrop. Enhance!
I thought I had seen most of Red Dwarf but I don't remember that scene... hilarious. I'll be impressed when Adobe comes up with the ability to "uncrop" :)
If you have a fine detail that only covers a small portion of a pixel and is thus not resolvable, but you have multiple "frames" and you know exactly how the subject and the camera moved between frames (easier for a static object obviously) then you should be able to solve the system for much higher effective resolution. It has always been my assumption that this sort of thing would be used in spy satellites and other systems where you want to do much better than the diffraction limit.
At one point some years ago I started to try to work out if I could resolve the footsteps left in the lunar surface by the Apollo astronauts, given a reasonably large earth-based telescope and enough multiple-frame high-resolution video passes over the target where the motion of the telescope and its optics system was known to a high enough degree of accuracy.
G.
until you get them up to speed you could not fix some photos for them.
I see all sorts of good uses for this, including hitting up some old old photos we have around the house too what other details pop.
photography never ends, it just changes and gets better. Eventually someone had to figure out how to compensate for the photographer's mistakes.
* Winners compare their achievements to their goals, losers compare theirs to that of others.
It turns out Kevin Costner really is the Soviet sleeper agent. Looks like the remake will be shorter.
This isn't new. There's a shareware plug-in, "DeblurMyImage", for it.
There are two main cases - focus blur and motion blur. Dealing with focus blur is well understood, because what defocusing does to an image is well understood. Motion blur is harder, because you have to extract a motion estimate first.
This demo was shown at an Adobe event called "sneek peeks". Demos show there are in development. There is a disclaimer said at the beginning that states the code they are running is in development and may or may not make it into an Adobe product.
That said, the demo was impressive. Especially making blurry text clear enough to read. Take it from someone who was there, yes, it was exprimental, but it was impressive.
(in this case, an incorrect focal plane)
This case involves motion blur, not incorrect focus.
deblurmyimage
> I think it would be better to say that [most of] the data are already present; the data just happen to be initially in an unwanted form.
Not necessarily.
Some techniques of reconstruction use information that is not present. There's a video about reconstructing 3D images of people (with Tom Hanks as an example) which produces a 3D model from data in a picture _AND_ a database of preprocessed perspective angles of known stereotype 3D face models (ggl "morphable", video). I guess this is "thinking outside the box". Literally.
the "deblur" Microsoft did required a hardware addition to the camera.
Anons need not reply. Questions end with a question mark.
There are Photoshop plugins that do this, e.g. Topaz InFocus: http://www.topazlabs.com/infocus/
But that's just a computer-generated illusion, not a reflection of reality.
Technically, any picture you take with a digital camera is just a "computer-generated illusion", even in raw format. It's not reality. It's a programmer's interpretation of the data from camera's sensors. The camera's sensors detect a different range of light than the human eye, filters are used to try to keep it within the human range. And when you view the picture, what are you going to use to view it with so that the picture is displayed in it's truest form?
Even the images you see with your own eyes are subject to the way your brain handles the data from your organic optical sensors. And then there's the issue with how your brain handles storage of that image. When you view a picture on your monitor, there's probably at least three levels of virtualization separating you from reality.
You'll go crazy trying to record reality, and eventually you'll find it's overrated.
A microscopist would call this "deconvolution."
Yeah, this is standard math. A completely out-of-focus picture actually contains nearly as much information as a sharp photo, it's just smeared with a reversible mathematical transform called a point spread function. Reverse it and you get the in-focus image back. There have been third party programs to do this for about a decade. The main problems have been processing speed (it could take a half hour or more a decade ago), determining the point spread function (you have both focus and camera shake, and the former can make figuring out the latter really hard), lens/sensor defects and image format compression (the PSF you calculate for a local region may not work well for the entire picture), and boundary conditions.
'Enhance' clatter-clatter
'Just print the damn photo.'
I suspect similar software's already in use at Ft. Meade and Langley.
I certainly hope not.
This will make things LOOK pretty. It won't make missing data suddenly appear. At best it will make something ugly LOOK a little better. But that's just a computer-generated illusion, not a reflection of reality.
So what you're saying is Apple isn't the only company with a reality distortion field?
Actually if you have the original PSD file you can uncrop an image.
At best it will make something ugly LOOK a little better.
Or in the case of cosmetics ads, make something that looks good look a little uglier.
Remember, Beauty is in the eye of the beerholder.
And yet it takes them half a minute to remember what kPa stands for.
IIRC, Focus Magic did not correct for motion blur - it was a pretty neat program otherwise. I've not seen a commercial program that actually does what this thing does - tracks the motion blur - then corrects for it.
And motion is the real killer on a number of real world shots. If you're not using a good tripod you have it. May not be visible, but it's there.
But it seemed to work well and I will go out on a limb to say it will be one of two useful additions to Photoshop 6 (along with six hundred minor fixes, 1800 new bugs and a dozen UI changes). Oh, and a $200 - $1000 upgrade fee.
Faster! Faster! Faster would be better!
You forgot the step where they would pull the convenient database of every crossing arm to match it up (I watched a few episodes, they have a database of everything dontchaknow)
What took Adobe so long? The military has probably been doing this for eons already.
Haven't read the article but of course you can't restore missing data 100% reliably but there are cases where you can unhide what is meant to be hidden or do a little better than "interpolation" as you describe it. It really depends on the picture, how its blurred and what it is meant to be hiding. You can for example sometimes deblur pictures of faces to make them more recognisable. You can also sometimes retrieve blurred characters from a number plate. I've been doing this for years though typically by hand. If you can incorporate data from a similar pattern/source you can restore missing data somewhat reliably.
This doesn't recreate lost data, It merely orders randomised data. The blurring of the picture is caused by the movement of the camera, and this photoshop plugin can only unblur motion blurring. It is impossible for it to work on unfocussed images, as the data is indeed completely lost there. It is entirely possible that if you can create an algorithm that can essentially backtrack the camera movement, you can get a pretty accurate representation of the image. It will never be a 100% perfect representation, but the margin of error is almost completely negated by the actual process of unblurring.
It happens in physics, so why not in digital imagery?
If you gave me a choice between a printer and a giraffe with explosive diarrhoea, i'll get my ladder and my raincoat
I thought I had seen most of Red Dwarf but I don't remember that scene...
That was what I thought when I first saw it. Turns out that it's from a three-episode special called Back to Earth made for a TV channel called Dave.
It's simple deconvolution FFS, there's no magic... In the fourier domain, it can be expressed as division, nothing more. The hard part is actually figuring out what the image was convolved with, e.g. the path the camera took as the picture was being taken.
weinersmith
Yes. As far as I know the process is called deconvolution.
What is I think new about the thing that Adobe shows here, is that it doesn't just compensate for out of focus or other instrumental effects, but for camera motion. (yes I watched the video). It determines the likely motion the camera made during the exposure. and then uses that as some kind of matrix for deconvolution.
What makes that tricky compared to a classical point spread function that only includes instrumental effects, is that it's probably not uniform across the image. There's not just x,y,z movement, but also yaw, pitch and roll. The video is very poor, but that seems to be what this algorithm is able to detect and then correct for. As such it goes beyond what we've been able to do since before the Hubble telescope needed glasses. Hogbom clean algorithm and such are from the 70's. This goes beyond that.
RogerWilco the Adventurous Janitor
I suspect similar software's already in use at Ft. Meade and Langley.
20 years ago. I did a talk in college (before I went to university) about 20 years ago about this. its been around for a long long time. You need kick-ass embarrassingly parallel hardware to make it time-practical, but its been around for a long time. It means that "sadly out of focus" films such as that taken by Abraham Zapruder (John F. Kennedy Assassination) are no longer "sadly out of focus". Go ahead and make your own conclusions about that reality as you like...
Unblurring without having any additional information has been done by academic software before using a process called blind deconvolution, look it up for some interesting pictures and videos.
It's a rather "expensive" (cpu-intensive) operation, and indeed having sensor data about how the camera has shaken during exposure would significantly help in restoring the image. Interestingly, even cheap smart phones with crappy cameras will often already have movement sensor on-board, so there are some possibilities to improve image quality right after taking a picture; all it takes is a bit of software. How long until someone here whips up an improved Android camera app?
I'm probably under-informed, but I haven't heard of any cameras with full-blown movement sensor, although I know some of them can work out portrait vs landscape by now. Sounds like camera manufacturers have some catching up to do in the hardware department.
Visit http://ringbreak.dnd.utwente.nl/~mrjb/growingbettersoftware to download your free copy of the book
This uses a single image as input, and tries to determine a local prior (L) and a motion kernel (f). It switches between optimization of each in turn, and produces results similar to the demo seen in the video. Given that Aseem works for Adobe, I suspect this work is now close to release.
Cheers,
Toby Haynes
Anything I post is strictly my own thoughts and doesn't necessarily have anything to do with the opinions of IBM.
It uses the gyroscope to feed data to the camera's image stabilization program.
From my watching of the video the only NEW FEATURE demonstrated was compensation for motion blur.
The rest of the focus blur problem was handled by the standard (and not very clever) photoshop sharpening logic.
We would have learned much more if red tennis shoe guy could get over himself and STFU instead of
grandstanding the entire video.
Just wait 30 years, and you'll be able to do it in seconds with the same algorithm.
The Tao of math: The numbers you can count are not the real numbers.
No, if you zoom beyond Planck distance, you observe and thus collapse quantum states which were never intended to be observed, and therefore you'll destroy God's mind instead of reading it.
The Tao of math: The numbers you can count are not the real numbers.
Unreadable just means our visual system cannot figure it out. It does not mean the information is not there.
The Tao of math: The numbers you can count are not the real numbers.
I saw a demo of correction for motion blur on Tomorrow's World about, um, 20 years ago. Compensation for a poorly focussed single lens, as I understand it, is not possible as the information has been lost (though the physics lessons where this was explained to me was from, erm, about 20 years ago.)
No, this can bring out information...because the information, with all of its frequency components, is there.
For some shutter length, you end up with an image that's the sum of all the viewpoints that the camera saw while the shutter was open. Since most shaky handheld shots will be translational across the image sensor, you assume that the scene didn't change much, estimate this motion by looking at easy-to-see features, then sweep your estimated, non motion blurred image, over the estimated path a few times until you reduce the error to some satisfactory amount.
Unlike a Gaussian blur, the information is still there assuming you don't overexpose.
They've been doing this for a while in astronomy. Motion deblurring will bring up many many interesting papers.
Unblurring without additional extrinsic information is different than unblurring intrinsic information that doesn't exist, which is what he was saying.
Blind deconvolution works, but, from what I understand, can only restore information up to the frequency components of the point spread function of the blur; you can only restore information that was preserved. If you do something like a Gaussian function, you've destroyed information and it'll never be sharp again.
Indeed. There is an interesting SIGGRAPH paper by Rob Fergus that shows how one can use natural image statistics to recover a kernel from a single image. Historically this has been pretty easy in the astronomy domain because they can observe the kernel formed by a point light source (stars). It's much more difficult for arbitrary photographs.
>Also, I doubt it would be able to do anything for a specific subject that was moving during the shot
Agreed. It probably goes and looks for many features whose bulk motion correlates...the background. I but you could tell it to unblur just the person though...that could make some interesting shots.
I'm super crazy excited about this. Ever indoor picture I've ever taken that's been ruined was ruined because of motion blur. I wonder how sensor noise works into the final image.
Anyone know what projection surface is used during this presentation? that's the most impressive thing in the video.
Am I the only one who suspects that these pictures might have been sharp and clear when they were shot, and only artificially blurred by software? Is there a famous blurred picture (Kennedy assassination perhaps) that could convince even the skeptic?
I'm guessing all the constant luminance point sources all over in the frame would make motion "estimation" much more accurate. :) Especially considering the path along with relative velocity (v = k*brightness) is, literally, presented. Try doing it when your only motion reference is a blurry mailbox :P
Still, very cool though.
Cool beans that covers this problem. As the paper mentions, the common and fast math for deconvolution (division in the s-domain) only works for uniform blur with a PSF. It seems like a more iterative approach is required since the easy math can't be used.
Ah, but the mind of God is as indestructible as it is inscrutable, grasshopper.
No, this is motion deblurring. The information is there since the final image is just the integral of images from every viewpoint the camera saw while the shutter was open...so it's just a sum of an image that was translated and rotated. Figuring out how the image was translated and rotated and rotated through time to lead to this final image sum is what they're doing. There are many interesting papers on google.
I actually came up with my own deblurring algorithm based on the same type of idea just a few weeks ago. Glad to see it actually works!
I said no... but I missed and it came out yes.
Well it's first to file so you can't do it anymore.
I've seen this kind of thing 5 years ago in research laboratories in the image recognition field, and it was running in real time.
I know some start-ups specialize in that kind of things to make photos taken by smartphone much better. The smartphone makers put a crappy camera for the phone to be cheaper, and then fix it all in software.
It works even better when you know the kind of faults the camera in certain smartphones have.
This is why I have a job -- nobody wants to wait weeks for the algorithm to run today, nor 30 years for the hardware to negate the weeks. They want better (implementations of) better algorithms. Except that they don't know that's what they want...
IIRC out of focus is harder to do automatically. However, I've been waiting for motion deblurring in PS and GIMP for some time. It's something I would have done eventually given enough time and lack of anyone else implementing it - fortunately I don't have to now.
On a related note, when are cameras going to use motion estimation (already there for video compression) to prevent motion blur when using longer exposures in low light? It seems obvious enough - you've got the motion vectors, why not sum multiple short exposures into a final long exposure with motion compensation?
Does it solve the:
1. picture is blurry because camera's autofocus did not lock on subject (or operator locked it onto a subject other than what was intended) - and intended subject was outside of the camera's depth-of-field for that distance+lens+aperture)?
or
2. picture is blurry because it's a crappy digicam with an unsensitive sensor or crappy flash recharge cycle rate, and any photo taken out of full-sunlight either drives the auto ISO adjustment to "super-mega-grainy" or shutter-speed to "1-full-second, next-time-bring-a-tripod-sucker".
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
Did not, but now perhaps does:
http://www.focusmagic.com/examplemotionblur.htm
I say perhaps, as it seems like it might limited to unidirectional motion smears.
Anyway, the clever thing in this demo was not the deconvolution (a trivial convolution operation in itself), but the calculation of an accurate estimate to the original jitter pattern.
Also FatPhil on SoylentNews, id 863
As another poster pointed out, this was in their "Sneaks" show, where various engineers showed off pet projects. No promise this will be in a future project. It really was a serious of scripts the engineer had written, and it might require more horsepower/wait time than your typical Photoshop machine/customer has.
For those who can't see the video, it was at least as amazing as you think it would be. I personally thought the Sneaks were more impressive than the keynotes.
The best thing about a boolean is even if you are wrong, you are only off by a bit.
DVD is crappy Mpeg2 and does *not* use the bitrate efficiently.
Is there a general ratio between MPEG-2 efficiency and AVC efficiency?
You cannot compare bitrate values between codecs.
What do you mean by this? In the statement, "An encode of typical SD video with MPEG-2 at 6 Mbps and with AVC at x Mbps are roughly equivalent in quality", is it possible to solve for X?
The bitrate figure is completely useless and means nothing by its self.
A bitrate specifies a point on an encoder's rate-distortion curve. If I say bitrate x with codec a is like bitrate y with codec b, I'm saying they have comparable distortion at those respective bitrates.
http://refocus-it.sourceforge.net/ How does it compare to this plugin for gimp from 2004 which seems to do exactly the same thing? Except maybe it guesses at the input parameters first?
This statement makes no sense:
A completely out-of-focus picture actually contains nearly as much information as a sharp photo, it's just smeared with a reversible mathematical transform called a point spread function.
Either the transformation is reversible, and the two pictures contain the same information; or it is not, and they don't. If you don't believe me, I've posted a completely out-of-focus picture of a white object in the margin directly to the left of this post. I dare you to tell me what it is.
I attended an interesting presentation in 1990 on transmission electron microscopy being used to determine the structure down to an atomic level of a growing area of a tooth. Calcium and other atoms of interest are far too small to image so you get something blurry. The structure was determined computationally by working out what a series of different structures would look like after being blurred by the limited resolution of the microscope and then comparing that to bitmaps of the captured images.
I know that is a different approach but people have been working on getting information from defocused images for a long time.
It is not always true that the transform is reversible, and in fact it is seldom true. There are many problems, and three of them are:
And there are other problems.
Contribute to civilization: ari.aynrand.org/donate
It would be nice if Adobe put this into their software, but it's been technically possible for decades, what's taken them so long?
True image stabilization has been available for a few years in cameras for $130 or less. The need for motion blur fixing in software is considerably reduced if an IS camera is used.
Contribute to civilization: ari.aynrand.org/donate
Is that the same as when putting a fence up the post holes are already there and you just have to take the dirt out?
Everybody taking pictures with cheap cameras, in low light conditions or in shaky environments has at some stage researched for deblur solutions. Personally, about once every year I survey the status of deblurring software to see if there is any progress. There are about a dozen (mostly shareware) alternatives out there, some of them pretty expensive too, but I don't find them very spectacular. Most rely on trial-and-error tweaking some cryptic parameters (point spread function radius etc) and eyeballing the results.
My favorite so far is a Danish freeware by Michael Vinther called "Image Analyzer". It is a standalone image processing toolbox with several features not meaningful to the average photographer (FFT etc), but allows several adjustments to be made, even adaptive noise reduction. Deblurring is implemented as a "Restoration by deconvolution" plugin and its distinguishing feature is that it automatically guesses the amount of motion blur (x and y axis), circular blur or Gaussian blur present in the image and allows previewing the result. One can iteratively apply deblurring steps and see the algorithm's convergence on a small graph, or scan a range of parameters and visually compare the individual corrected images. I have found that most problematic pictures suffer from both motion blur and poor focus, so correcting both often provides spectacular results. If you are really desperate about getting that unique image fixed, Image Analyzer is a very good and free start point.
However, after exploring the possibilities of Image Analyzer and other apps, I found that their weak point is that any significant image noise will distract the algorithms resulting in images worse than the original - noise itself gets focused and amplified. The Adobe demo worked well in the well-lit and therefore noise-free sample, but I doubt it would work with typical indoor photography using a weak flash or no flash at all (i.e. pics taken with mobile phones and of course tablets). In such cases, a workaround is to denoise the picture first (Absoft's Neat Image still is the king for the purpose) and then applying the deconvolution. Another trick, in case you are particularly interested in deblurring a part of an image, e.g. a face, is to crop the image to the part of interest, play with deblurring parameters and then apply them to the whole image.
Image blur is and will remain a big problem of digital photography, as long as the cameras are handheld, lightweight and used on the move or in low light conditions. Millions of unique shots worldwide are ruined by it and it's no wonder that a lot of effort and resources is spent by big companies and researchers (not to mention three letter agencies) to deal with it. If this new Adobe solution is a better one, I expect several lawsuits in the future.
Actually zooming in beyond the Planck distance is the same as zooming out from the Planck distance - see T-duality.
And see how ~40 year old tech hits the crowds. These moments are usually quite fascinating, but more from a tech-sociological point of view. Same happened with HDR, anisotropic diffusion, face detection, and so on and so forth.
I am putting myself to the fullest possible use, which is all I can think that any conscious entity can ever hope to do.
I can do the same thing with any version of Photoshop. Add a secret gaussian blur layer, start the demo, delete the gaussian blur layer, and there's your clear picture.
Video was too long, didn't read. Too blurry, couldn't see.
A bitrate, codec and resolution combination gives you a bit more information though.
That's why one might specify 4 to 6 Mbps as the bitrate, MPEG-2 as the codec, and "SD video" (shorthand for 704x480 at 24 fps) as the resolution, and then ask what the bitrate might be if I were to hold the resolution and distortion constant and change the codec to H.264.
Each codec compresses different things more effectively
Which is why one might put together a more-or-less representative corpus and ask questions about that. Say I had about four hours of standard-definition video: two 22-minute episodes of My Little Pony: Friendship Is Magic, a 44-minute episode of NCIS, an episode of The Rachel Maddow Show, the last 45 minutes of the film The Matrix, and the fourth quarter of an NFL game. What bitrates can I expect to "get away with" for the two codecs on this material?
"""
This will make things LOOK pretty. It won't make missing data suddenly appear. At best it will make something ugly LOOK a little better. But that's just a computer-generated illusion, not a reflection of reality.
"""
If you believe that, then get your hands on any of the software that's been able to perform deconvolution before this Photoshop plugin (it's at least 10 year old technology, the only thing that's new about this is the speed), and run the process over a blurred image and over the same blurred image once you've run a non-linear noise reduction filter over it (such as a 3x3 median filter). The former will deconvolve much more successfully, which should prove to you that the noise in the original wasn't actually noise, but contained real information that had been smeared about. I.e. the process will make smeared data appear closer to where should be. So you do see more data - it's not an illusion.
Also FatPhil on SoylentNews, id 863
For each depth, out-of-focus can be seen as equivalent to being a Gaussian 'motion' kernel. The issues with that are that each layer of the image needs its own kernel (and it's even worse if the image doesn't come in clear layers, but for a typical portrait shot, it works), and that the inverse of the Gaussian can cause noise to blow up dramatically, as some of the multiplication terms can be huge.
So it's hard, but it's not quite fair to say the data isn't there, it's just very well smudged.
Also FatPhil on SoylentNews, id 863
Now don't you confuse the issue with your fancy mathematics!
Nope. Prior invention is still a defense, and prior publication will still invalidate a patent. The only change here is that if you aren't the first to file, you can't get a patent yourself -- but you can certainly still invalidate the patent of the person who was first to file if you published your technique earlier (or keep using the technique you discovered in-house but didn't publish, as long as you can prove you were using it before their supposed date of invention); you just can't then turn around and get a patent yourself.
Seriously, I wish the people who got so bent out of shape over this thing learned WTF it was. It doesn't fix our patent system's problems, but it's really not the scary awful thing you're making it out to be -- to the contrary, it's an improvement: First-to-file only means fewer patents total.