Slashdot Mirror
Microsoft Tech Can Deblur Images Automatically
An anonymous reader writes "At the annual SIGGRAPH show, Microsoft Research showed new technology that can remove the blur from images on your camera or phone using on-board sensors — the same sensors currently added to the iPhone 4. No more blurry low light photos!"
Enhance!
Probably only half-working coming from microsoft, plus if you use black light in the room you can get brick you're phone/camera.
I bet it can remove the blur from the titlebar for screenshots of a Windows 7 app. Now we can all see what those developers are viewing behind that window!
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
This is like one of those "Why didn't I think of that?" ideas that you wonder why your camera doesn't already have. The nice part is that it can be done very cheaply (relative to the cost of a camera) and would improve images in many cases. My only tiny little concern is that you might introduce artifacts into your photos - which makes me wonder if it wouldn't be better to store a raw image and the data from these sensors independently? I wonder if there is a scenario where you might be moving but the object you're taking a picture of is stationary relative to your movement. Like for example you're standing on a boat rocking in the waves, you take a photo of the deck, and this technology compensates for the rock which results in a ton of blur.
Maybe I failed to catch it from the article:
At the annual SIGGRAPH show, Microsoft Research showed new technology that can remove the blur from images on your camera or phone using on-board sensors -- the same sensors currently added to the iPhone 4. But WTF? Steve please!!! It's not always about you.
There is a lot of poor porn out there from people that can't hold a camera still. Microsoft should redeem itself and sort that out asap.
Finally, we'll have to quit making fun of the redhead cop every time he asks to zoom into a blurry license plate.
Step back! This is a job for Frankencamera. Run it on your Nokia N900 today.
OTOH having that Arduino board and a mess of wires attached to your camera does score you a lot more geek cred than photographing using an plain old mobile phone.
/greger
Social networking sites are about to get a whole lot more ugly
I recall that some other cameras, like a Casio I've seen a friend using, also do deblurring, but rather by stacking of rapid subframes (I guess using bright reference points). If I understand correctly, this new method is operated on a single frame. I wonder if anyone has a useful comparison of the hardware requirement/image quality/useability differences between the two methods?
Great, you can improve your motion blur removing algorithm by recording the motion which created the blur.
Although technically, the blur in the image itself already recorded the motion, with better precision and without calibration issues. So this is more of supplementary data. The before and after images leave out the whole "you can already do this without the extra sensor data" aspect.
And really, you'll get far better results if you just use an adequately short exposure time and some mechanical stabilization. Brace your shooting arm. If you want to get fancy, use something like Canon IS lenses.
Yeah, this is nifty, especially for smartphone based cameras which may already have built-in sensors to do this. But neither is it exactly revolutionary. You'll get better photos out of learning some basic photography than you will out of fancy sensors and analysis software.
That's the first thing that came to mind. Luckily, it's a hardware attachment so we can still tell people to fuck off when they come to us with blurry photos.
Unless they have the attachment.
Sounds like a great way to land a spot on a terrorist watch list, to me...
Microsoft, you can black out the brand logos, but you forgot to black out the red stripe on that L-Series lens. Dorks.
Also, if you want working de-blurring, try turning the lens's image stabilization on. This is something better suited to the optics than the sensor, and Canon and Nikon both do a very good job with image stabilization. All doing it in-camera will do is suck processing power.
This is hardware that takes into account movement of the camera as you take a picture. Meh... I thought this was a software solution that could deblur a picture after it has been taken. This technology exists but I guess it's more used to get at redacted information rather than make a picture clearer. Seems like it could work though.
While it's a nice idea, isn't this just a poor man's image stabilization? Even cheap compacts come with some form of IS these days, and high end SLR lenses certainly do.
..and shame on you, camera manufacturers, for not thinking about this already!
For the past 8 years or so, MicroSoft has been co-author on more papers than any other organization at SIGGRAPH. This is impressive because SIGGRAPH has a the highest paper rejection rate of any conference I know of - they reject (or downgrade to non-published session) 85% of the paper submissions. And you have to submit publication-ready papers nearly a year in advance, with a video summary.
This reminds me of Xerox PARC - great R & D output, poor commercialization of these results. People wonder if their lab was a toy-of-Bill or a tax write-off.
My only tiny little concern is that you might introduce artifacts into your photos - which makes me wonder if it wouldn't be better to store a raw image and the data from these sensors independently? I wonder if there is a scenario where you might be moving but the object you're taking a picture of is stationary relative to your movement.
I suspect in the majority of cases, this would improve photos. As to your query, my first thought of a problematic environment would be trying to take a photo of a friend sitting next to you--in a moving roller coaster as it hurls around a bend. You and your friend are [mostly] stationary WRT each other, but you (and the camera) are all undergoing acceleration, which the camera dutifully attepts to remove from the photo. Certainly a comparatively rare event compared to the majority of photo-ops.
This is probably not "new technology".. just Microsoft's version of image stabilisation that "real" camera companies have been using for years. I sounds just like the in-camera image stabilisation used by many point & shoot cameras, and some dSLR's... and it would be VERY like Microsoft to copy someone else's technology and pass it off as a new thing. (just look at the Mach Kernel which underlies NT, 2000, XP, 2003 etc.. versions of Windows.)
Not unless you know the sizes of all the objects that made it into the frame, and the distance of each to the camera. The camera, without this sort of motion sensor, at most knows the focus distance and view angle, so *maybe* it can guess the height and width of objects at the plane of focus--but then the problem becomes knowing which pixels are recording the object at the plane of focus.
But anyway, if you know the focus distance, view angle and camera motion, you can apply some corrections that are likely to improve acutance of objects at the plane of focus, and maybe regain some resolution. It wouldn't turn a photo with motion blur into the equivalent of one without, because motion blur causes information loss, which will show up as loss of resolution. Or in other words, the correction will probably make large high-contrast edges look sharper, but there will be some loss of fine detail due to the motion.
Are you adequate?
Well, many cameras have optical vibration reduction, either on the lenses or using a sensor-shift mechanism. This mechanism, to the extent that it works, should work better than the software solution being described in the article.
It's important to understand that random camera motion blur in almost all cases leads to information loss. The rays of light that would have hit only one pixel if the camera had been steady, because of the motion, will end up hitting more than one pixel--whereas moving the lens elements or sensor tends to keep the same pixel aligned with the same point of the photographic subject.
My guess is that recording the motion of the camera and doing the post-processing described in the article will reintroduce some acutance to the image (high-contrast edges will be sharper), but that there will still be a significant loss of resolution (the finest detail that can be recorded). So, for example, the edge of a person's face will be reasonably sharp, but there won't be a lot of detail on the hair or skin.
Are you adequate?
There are real limits to the human body. Anyone who says "I can hold a camera perfectly steady," is lying. We are not perfect platforms. So image stabilization can help a lot. Long range photography, in particular of fast moving objects like in sports, got a big boost when optical image stabilization came out. The length that you could zoom and still get a good shot increased. Wasn't that the photographers were bad, it was that they were at the human limits. The optical stabilizers enhanced that are upped the limits. After the fact deblurring could up the limits more.
It won't help at all if the object is moving. In fact, this feature should be switched off if you're trying to photograph a moving object with the camera (common enough, and not just in sports). It would not be able to compensate for a mismatch between the object speed and your tracking movement, and would do entirely the wrong thing even if you tracked the moving object perfectly for the shot. In this case, there is no substitute for adequate light and/or a fast lens and/or a smooth accurate tracking movement.
As another comment, deconvolution requires a very accurate approximation of the true convolution kernel, which may be provided by the motion sensors. However, to reconstruct the image without artifacts, the true kernel must not approach zero in the Fourier domain below the Nyquist frequency of the intended reconstruction (which is limited by the antialias filter in front of the Bayer mask). In fact, if the kernel's Fourier transform has too small a magnitude at some frequency, the reconstruction at that frequency will be essentially noise, or will be zero if adequate regularization is used. If the motion blur is more than a few pixels, this will generally mean that the reconstructed image will have an abridged spectrum in the direction of blur, compared to directions in which no blur occurred. Of course, if your hand is so shaky and the exposure so long that blur occurs in all directions, then the spectrum of the reconstructed image will be more uniform. It is likely to be truncated compared to the spectrum of an image taken without motion blur.
The quality of the reconstructed image would also be limited by the effects of other convolutions in the optical pathway. For instance, if you're using a cheap superzoom lens, don't expect to get anywhere near the antialias filter's Nyquist frequency in the final image, as the lens will have buggered up the details nonlinearly across the image even before the motion blur is added. If you're using nice lenses (Canon "L" series or Pentax "*" series and suchlike), then this will not be an issue.
The method would seem to be useful in low-ish light photography of stationary objects. A sober photographer would beat a drunk photographer at this, but the technique would help both to some extent. A photographer using a tripod would do best, of course.
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
...a flying chair.
Table-ized A.I.
This is going to revolutionize the hentai industry.
The whole premise seems kinda ridiculous. You might have some idea how the camera swung, but that only helps you if you're pointing at some 2D surface that's perpendicular to the camera.
If there is any depth to the scene, points closer will move more than points farther away. You might have an estimate of the distance from the auto-focus feature, but that's only going to help you fix up points near the focus sweet-spot. Points closer and farther away are going to be made worse, not better.
Clearly (pun intended) the results have a ways to go yet. Look at the coca-cola image, at the 'a' on the end of the cola... that thing is hosed by the blur, and they're unable to recover it because there's no intermediate contrasting color. Same thing for the spokes on the car rims.
This problem can't be completely solved post-picture. Only large-scale elements with nothing else around them will yield pixel-sharp solutions.
The optimum way to correct blur is to apply active or passive (e.g. tripod) stabilization to the lens prior to the shot; active technology is already pretty decent (photographers tend to measure things in stops; it's intuitive to them... when they say an active stabilizer "gives you" four stops, for instance with Canon, what they mean is that you can shoot four stops slower with the shutter and you won't get blur from camera movement.) Doesn't solve subject movement at all, but then, nothing really does other than cranking down the exposure time.
So... considering lens stabilization has been in-camera for years, and this requires more hardware, but gives you less... I'm going to go out on a limb and say it isn't of interest to camera folks. Maybe in some esoteric role... a spacecraft or something else with a tight power budget where stabilization can't be done for some reason (certainly measurement takes less power than actual stabilization)... but DSLRs and point-and-shoots... no.
I've fallen off your lawn, and I can't get up.
Now they just need to attach this to Ballmer's head to deblur the company vision a little.
You know, you -- and 99% of the others bitching about the Gimp -- you're utterly full of shit. I write commercial image processing / editing / animation / generation software for a living, I'm expert - you can read that as "terrifyingly exert" - with Photoshop, Gimp and a whole raft of others... and Gimp is an easy to use powerhouse.
Now I will grant you exactly ONE thing, and that is, you need to sit down and learn to use it. That should take a few hours if you're familiar with something (anything) else; maybe a week hunting down tutorials, or a day hanging with a qualified mentor, if editing bitmaps is all new to you.
If it takes you longer than that, you're either stupid or lazy.
There's *nothing* significantly wrong with the Gimp. It has its limits, like everything does (Photoshop has some really annoying limits too), but for the vast majority of image processing and touch-up needs, it's very nice.
Oh, mommie, my crop function is in a different menu... Some people just need a good smack in the head.
If you really knew what you were doing, you'd have, and use, a whole suite of these programs, because for the big ones, there are areas where they excel, and that's the time to put them into play. If you can't learn to use them because the keystrokes are different, or there is a different paradigm... it isn't the program that sucks. It's you.
Also, if you actually knew how to use them, you wouldn't be bitching about them.
I've fallen off your lawn, and I can't get up.
Information théory tell us that once some info has been lost, it can't be recovered. If the picture has been somehow "damaged" by some motion blur, the original picture can't be reconstructed.
On the image, we'll have much more than the motion blur from the camera's movement : ...
- noise added from sensor electronic noise
- blur from target movement
- distortion coming from lens defect (mostly for low end cameras)
- distortion/blur from bad focus (autofocus in not perfect)
The operation that will reduce the camera's motion blur will probably increase the effect from all other defects. You reduce one kind of image destruction and increase the impact of the other one.
There are some full-size samples of the results of the technique, where you can compare the original image with the result of their technique, and the results of two older techniques. Their technique show some very obvious problems:
There are several examples in the link that seem to be like that. The technique doesn't seem to figure this out in all cases, and renders the two ghost lines as separate, sharp lines. Most obvious example: the edge of the front rim of the red car in the second photo. Though compare with the result they got in the photo of the Coca-Cola cans, where it did figure it out for the rack, but not for the text on the cans, and where it introduced some artifact lines perpendicular to the rack.
The more instructive comparison is the results of these guys' techniques with the older techniques. Clearly, they're doing a lot better than the older techniques. Still, this is very far away from primetime, IMO.
Are you adequate?
What about combining the accelerometer data with a setting that records low-light images is a series of high-speed, underexposed images, then just using to accelerometer data to merge them?
Something to deblur all that japanese porn I have laying around...
Here's a dumb question...
If you just need some shaking data to unblur very nicely, why can't one just (offline, with a hour or two to crank on it) just figure out what the motion was by unblurring as hypothesis testing, perhaps on a small section of the picture. Then you unblur the whole thing on the most likely candidates?
I think the problem with any method that doesn't change the optical path or move the sensor is that it just can't deal with parallax.
So, your accelerometer records that between the first and the second microexposure, the camera shifted by x amount to the left. What relative shift do you apply to the frames? Well, the problem is that the correct shift is different for objects at different distances--so as soon as you have an image with large depth of field, there is no solution that corrects the blur for all objects in the frame. It might still be useful, though, because you'd be able to reduce camera blur at one distance--e.g., the camera could assume that the correct distance is the focus distance, or if you used RAW processing, you might be able to choose the correction distance at processing time.
Note that optical stabilization systems don't have this problem to the same degree, because they're designed to keep the same ray of light hitting the same pixel during the whole exposure.
There are other complications, though, because each of the microexposures will have more noise and reduced dynamic range compared to the full conventional exposure. I.e., by spending less time recording the value of a pixel, a microexposure is correspondingly less able to finely discriminate its level, and more so when the pixel is dark. Combining the microexposures has the potential to average out the noise, thus gaining you more shadow detail and dynamic range; theoretically you can get the same dynamic range and noise floor as the conventional exposure, but in practice it might well be different. There's a problem, however, that if the sensor noise is not random, the accelerometric shifts you apply to the microexposures as you combine them runs the risk of producing noise artifacts, as the pattern of the noise might produce interference patterns when superimposed on shifted copies of itself (see moiré, or more generally, interference). That's because, to put it briefly, camera motion moves the apparent position of the objects in the frame, but doesn't move the noise patterns.
Yeah, this stuff is complicated.
Are you adequate?
At the annual SIGGRAPH show, Microsoft Research showed new technology that can remove the blur from images on your camera or phone using on-board sensors -- the same sensors currently added to the iPhone 4. No more blurry low light photos!
Uh, what? No more blurry low light photos... if you can get your Apple phone to work with Microsoft technology!
This is a hacked account, for which the owner can not be held responsible.
No deblurred porn for you
...for my bukake library!
The summary doesn't make clear that this is a way to correct for motion blur, when the camera moved during the exposure. It won't deal with poorly focused images. For that something like the GIMP's Refocus plugin can help. It's a pity that the motion of the camera needs to be recorded - it can't be inferred from the blur of the photograph. So this software would only help once manufacturers include motion sensors in their camera and a way to record the information in the image file. (Though, as noted, many mobile phones do include such sensors.)
-- Ed Avis ed@membled.com
Blind deconvolution and computational photography have been around for a long time. They are being used, for example, to enhance astronomical images.
Microsoft is making an incremental improvement to this field. That's nice, but why is it worth reporting any more than any of the other papers on this field?
What they have actually used is the fact that motion blur is not normal blur since normal blur would most definitely result in information loss. Apparently motion blur can be counteracted with the extra "motion" information. Now, all I wonder, how hard would it be to brute force that information that the extra camera sensors record. In many cases, I bet that in many cases, the direction wont have time to change during the short exposure, which could limit the number of directions to one. Now all you need is to go trough all of those directions and acceleration combinations and use an algorithm that approximates the resulting pictures sharpness, right?
They've been doing absurd levels of deblurring on CSI for years.
I'm pretty sure they took a clear photo, and blurred it with their custom software to look like an unsteady hand, and then, knowing how it was blurred, their un-blurring algorithm worked great... show me some real-world examples of photos you didn't take.
To avoid additive errors due to summing up wrong pixels in the movement good cameras instead sum up a load of very short shutter-time photos where movement is smaller than the pixels. This gives even better results than post-correcting for the movement as it allows the imagedata to be shifted before summing to remove the summing-error alltogether.
Anyway, the real innovation here is suggesting that one should use already present onboard sensors to try and post-process away movement induced blur. It's a good idea but not a eureka moment. Wouldn't be surprised if they patent the heck out of it anyway.
This tech is still unproven: they need to try it on new phone models--cause for some reason, every new phone spied that comes out is always burry.
TFA says they tested with an SLR; cameraphones could *really* use this, as the quality is low enough already (at least mine is.)
I listen to both RIAA and non-RIAA stuff if I like the music, tangential business/politics nonwithstanding.
The manual for my `Cannon digital IXUS 860IS' has this to say regarding blurred images. Does this mean Cannon owes MS revenue for using its patented anti-blurring technology?
"The image stabilizer function allows you to minimize the camera shake effect (blurred images) when you shoot distant subjects that have been magnified or when you shoot in dark conditions without a flash"
Microsoft looks to have come up with a Rube Goldberg version of Pentax' elegant "Shake Reduction" SR system that's been in their DSLR bodies for years.
http://photography.suite101.com/article.cfm/pentax_dslr_shake_reduction
"I might have made a tactical error in not going to a physician for 20 years." -- Warren Zevon
Galaxy S can do it too cause it has 6 axis motion sensing , please add it to your article...
http://www.handheldusers.com/forum/t7232.html
or
http://android.hdblog.it/2010/07/20/galaxy-s-sensore-di-movimento-a-6-assi-facciamo-chiarezza/
Just so everyone knows: this isn't a new technique. It is called Optical image stabilization. It is common in digital cameras. The novelty is doing it on a camera that wasn't intended for this, like the iPhone.
boudoir photography ?