Slashdot Mirror
Scaling Algorithm Bug In Gimp, Photoshop, Others
Wescotte writes "There is an important error in most photography scaling algorithms. All software tested has the problem: The Gimp, Adobe Photoshop, CinePaint, Nip2, ImageMagick, GQview, Eye of Gnome, Paint, and Krita. The problem exists across three different operating systems: Linux, Mac OS X, and Windows. (These exceptions have subsequently been reported — this software does not suffer from the problem: the Netpbm toolkit for graphic manipulations, the developing GEGL toolkit, 32-bit encoded images in Photoshop CS3, the latest version of Image Analyzer, the image exporters in Aperture 1.5.6, the latest version of Rendera, Adobe Lightroom 1.4.1, Pixelmator for Mac OS X, Paint Shop Pro X2, and the Preview app in Mac OS X starting from version 10.6.) Photographs scaled with the affected software are degraded, because of incorrect algorithmic accounting for monitor gamma. The degradation is often faint, but probably most pictures contain at least an array where the degradation is clearly visible. I believe this has happened since the first versions of these programs, maybe 20 years ago."
Photographs scaled with the affected software are degraded, because of incorrect algorithmic accounting for monitor gamma.
Seriously!
I have a theory on why this has gone unnoticed for so long, but I'll keep it to myself...
To display the pictures, it makes sense to use the monitor gamma. But to actually modify the data using that information which is probably flawed in 99.9999999% of cases? That's just wrong.
Most scaling algorithms treat brightness as a linear space, so e.g. if you're doing downscaling to 1/2 the size in each dimension, collapse 4 pixels into 1 by setting the 1 pixel to the numerical average of the original 4 pixels. But, most images are displayed with an assumption that brightness is a nonlinear space, i.e. gamma > 1. Therefore, scaling changes the perceived brightness, an unexpected result.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
This is only a bug depending on what you are doing with your final images. One of the things that annoys me is that many image manipulation programs do not actually explain the primitives they are using. The result can be a complete mess depending on what you are trying to accomplish. This article is an example of this effect.
If you want photo-realistic results, then you need to take Gamma into account. However, very few file formats specify the Gamma, the grey level, the white level, the black level or the colour space of the original image. The result is that the many imaging operations must be wrong, as they can never be accomplished the way intended. For the most part, no one cares. This person found an application where people care.
Well, I am SURE glad I'm using Linux^H^H^H^H^HWindows^H^H^H^H^H^H^HMac^H^H^Hshit.
It is pitch black. You are likely to be eaten by a grue.
I've been telling people for years that I look better in person.
I told them that there's something wrong with pictures of me.
HA!
Now I know.
It's the Scaling Algorithm BUG!
Come on, this isn't news...
Helmut Dersch (of Panorama Tools fame) certainly posted about this before;
http://www.all-in-one.ee/~dersch/gamma/gamma.html - Interpolation and Gamma Correction
There's no factual error in the scaling algorithm, as the /. headline would like you to believe - it's a color space (linearity) issue; you have to do your calculations in linear space which means a typical photo off of a camera/scanner gets the inverse of an sRGB curve applied (a gamma of 0.454545 is 'close enough' if you can't do the proper color bits). Then scale. Then re-apply the curve.
And no - for real life imagery, nobody really cares - the JPEGs out of the cameras and subsequent re-compression to JPEG after scaling will have 'destroyed' far more data than the linearity issue.
They're nice example images in the story, but they should be called 'academic'.
Gray rectangles in Picasa 3 and in Irfanview 4.25 :-(
I am sure the Chinese government prefers the current implementation.
Gamma is often poorly understood even by people doing scientific and engineering work using images.
Does your algorithm depend (explicitly or implicitly) on the light intensity -> pixel data mapping?
If NO: You're probably wrong. Go read about gamma. Just because the picture looks right to you, doesn't mean it looks right to your code.
If YES:
Do you have the luxury of taking the pictures yourself?
If NO: You're stuffed. Pretty much all images on the internet and most public research databases have unknown/unreliable gamma curves.
If YES:
1. Spend a lot of time calibrating your camera yourself. This is only cheap if your time is worthless
or
2. Buy a machine vision camera. A $600 machine vision camera will have the specs of a $50 consumer camera, but at least you will know the gamma curve.
or
3. Ignore the gamma issue, cross your fingers, hope it's not hurting your performance, and publish your results knowing that you're in good company and nobody will call you out.
The example images that make it really clear are academic examples. But the scaled photos are all enough of a change to be worth noticing and caring about if you're a serious amateur photographer (never mind professional). And they don't look particularly unusual to me (I haven't looked for odd trickery, but I assume he's being honest here).
The basic issue here has to do with gamma curves and the way they're being handled (they're not).
Most image files on your computer (BMP, JPG, PNG, etc.) are stored in the sRGB color space. sRGB defines the use of a gamma curve, which is a nonlinear transformation applied to each of the components (R, G, and B). The issue here is that most scalers make the assumption that the components are linear, rather than try to process the gamma curve. While this does save processing time (undoing the gamma curve then redoing it), it does add some error, especially when the values being scaled are not near each other.
So does this matter? Well, in some pathological cases where there are repeated sharp boundaries (such as alternating black-white lines or fine checkerboard patterns), this would make a difference. This is because the linear average of the pixels (what most image scalers use) yields a different result than if the gamma value was taken into account. For most images (both photographic and computer generated), this shouldn't be a big problem. Most samples are close in value to other nearby samples, so the error resulting from the gamma curve is very small. Sparse light-dark transitions also wouldn't be noticeable as there would only be an error right on the boundary. Only when you exercise this case over a large area does it become obvious.
One final point: this gamma scaling effect would occur regardless of the actual scaling algorithm. Bilinear, bicubic, and sinc would all have the same issue. Nearest neighbor interpolation would be unaffected, but in these cases, the output would look far worse.
That might be true, but it's no reason to turn this into an undocumented and unavoidable feature.
Ok, so he made a very informative page about it, but this is still a well known effect. It affects practically everything you can do in image editing. Blurs, etc. Most people neither notice nor care. It's rooted in the fact that most images come with undefined black and white points and a gamma chosen for artistic effect rather than physical accuracy. Thus correctly converting to linear gamma is hardly ever possible. You can still correct for monitor gamma to avoid some rarely seen inconsistencies and artifacts, but most people don't even notice, so why bother? However, Photoshop does have everything you need to avoid the effect completely, even in the ancient Photoshop 6.0.
Here's how to properly resize in Photoshop:
1. Convert mode to 16 bit (to avoid tone aliasing in the next step, no other influence on the calculations)
2. Convert to profile, select "Custom RGB", set Gamma to 1.0 (this converts the internal image data to linear gamma, no visible change because the image is color managed and corrected back to monitor gamma on the fly)
3. Image Size
4. Convert to profile, select "Custom RGB", set Gamma to 2.2 (default)
5. Convert mode to 8 bit
Done. You can substitute your favorite image filter for the image resize. Unsharp mask works much better at gamma 1.0, for example. Of course you can use several filters before converting back to monitor gamma and 8 bit.
If you're looking for lcd test images, http://www.lagom.nl/lcd-test/ is probably better. It's got a whole bunch of images dedicated to various monitor problems, along with explanations.
You're absolutely correct, AC. The reported issue isn't about a linear/nonlinear gamma bug at all - it's an averaging side effect.
The sample Dalai Lama image on TFA's page is intentionally constructed of interlaced lines of red and green data to thwart the averaging of source data used in common scaling algorithms. If you use the Gimp with the "None" scaling method, which will just pick-up every other row and column when scaling by 50%, (instead of trying to average 2x2 grids) you get a mostly-green image instead of the grey image advertised.
If you're a *serious* amateur photographer, then you should already know about this and not be using those apps / using them in the color modes (to use Photoshop parlance, as I guess most serious amateur photographers will have a copy (legit or otherwise of that)).
I guess the argument would hinge on who is a serious amateur photographer and who is just a regular amateur photographer.
As for the actual examples - sure, you can see the difference.. especially since they're in before/after -style swappable pages. If I presented you a random image off of a random image gallery online, though, would you be able to tell the difference?
If I showed you an online photo album and pointed at an image's thumbnail, had you click the thumbnail, and open up the full size image.. would you notice that it was scaled to thumbnail incorrectly?
"Nobody really cares" may have been too broad a statement - but those who really care, already know.. or reasonably should know.
imho.
Note that I'm not excusing the software programs from handling this better - certainly not Photoshop - but it's 1. not a new revelation and 2. certainly not a "scaling algorithm bug".
It's basically an implementation issue. The algorithms may be fine as intended ... in linear space. The programmers that implemented them didn't understand linear vs. gamma, or didn't care, or had a fire breathing PHB on their back. Hence we get junk software.
At least all MY image processing code always works in linear space. Bu merely converting 8-bit gamma to 8-bit linear is no good because that now introduces some serious quantizing artifacts (major banding effects happen). So I convert the 8-bit gammas to at least 30 or 31 bit integer if I need processing speed, or all the way to double precision floating point if I need as close to correct as possible. After processing, then I convert back to 8-bit gammas. Even then, you can't totally eliminate some banding effects that result from being in 8-bit. If you can get more bits from the raw images from your camera, that's the best to use. Apparently many JPEG compressors are also doing their DCT calculations in the non-unit gamma space instead of the linear space, too (which reduces the effectiveness of the compression somewhat, and may add more compression artifacts).
now we need to go OSS in diesel cars
My software has been calculating in linear space for over a decade now (this is the Nuke Compositor currenlty produced by The Foundry but at the time it was used by Digital Domain for Titanic). You can see some pages I wrote on the effect here: http://mysite.verizon.net/~spitzak/conversion/composite.html. See here for the overall paper: http://mysite.verizon.net/~spitzak/conversion/index.html and a Siggraph paper on the conversion of such images here: http://mysite.verizon.net/~spitzak/conversion/sketches_0265.pdf, in fact a lot more work went into figuring out how to get such linear images to show on the screen on hardware of that era than on the obvious need to do the math in linear. Initial work on this was done for Apollo 13 as the problems with gamma were quite obvious when scaling images of small bright objects against the black of space.
For typical photographs the effect is not very visible in scaling, as the gamma curve is very close to a straight line for two close points and thus the result is not very much different. Only widely separated points (ie very high contrast images with sharp edges) will show a visible difference. This probably means you are trying to scale line art, there are screenshots in the html pages showing the results of this. Far worse errors can be found in lighting calculations and in filtering operations such as blur. At the time even the most expensive professional 3D renderers were doing lighting completely wrong, but things have gotten better now that they can use floating point intermediate images.
One big annoyance is that you better do the math in floating point. Even 16 bits is insufficient for linear light levels as the black points will be too far apart and visible (the space is wasted on many many more white levels than you ever would need). A logarithmic system is needed, and on modern hardware you might as well use IEEE floating point, or the ILM "half" standard for 16-bit floating point.
But the point of the article is that it is possible to do it correctly. As many posts have pointed out, the standard gamma for sRGB images where the gamma is not specified is 2.2. Not taking that into account is, in fact, an error. If the image data are not linear to begin with, then why are we applying algorithms to the image as if they were?
Changing the image's gamma value during scaling is an error, plain and simple, especially when we know what the gamma of the image is to begin with!
Programs like Picasa and other photography programs certainly should be taking this into account.
Most professional photographers capture all their images to raw format. I think raw images are linear and are mapped to a gamma in post-production. So they probably will be less affected by this error. This may be why the pain/gain ratio for Adobe, for example, would be too large.
I think the author specifically isn't stating whether the scaling is correct or not - it is; the whole story doesn't relate to scaling at all, but rather color space and how -it- affects, among other, scaling. Yes, with filtering - scaling without filtering can hardly be called scaling at all as you're just discarding data - and for anything but multiples of 2 (4x, 2x, 0.5x, 0.25x, etc.) that'd have a whole 'nother set of problems.
The author, I think, is suggesting, quite rightly so, that while...
The desired result for scaling down likely being that of the same visual image as when you simply stand further back.
( although at some point the resolution limit of a display and the image itself being presented on that display prevents that concept from being applied to "moving your eyeballs closer to the screen" for scaling up. )
Well, as much as desired goes, this also affects how a lot of filters and effects work. For example, it causes most Gaussian blur implementations to 'flare' brights into darks more than they should. And that's been happening for so long, that that's now the expected/wanted behavior out of 'Gaussian Blurs'. If you changed that, you would have some confused/annoyed users.
"There is an important error in most photography scaling algorithms."
No, there isn't. If millions of professional users haven't been bothered by it over the course of two decades, it is CLEARLY not important.
Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
Several people have spoken about "linear" RGB. That's nice and gets rid of some small level of distortion introduced by the non-linearity. However, it only starts there. For example, the eye sees R, G, and B differently. It is more sensitive to green than red, and to red more than blue, but it's not even that simple as the equations in your eye's processor are much more complicated. Many algorithms that treat the three "equally" are going to change the perceptual mixture. One can use other color spaces, such as HSV, Yuv, xyY, etc. with different advantages and disadvantages
Sound makes a good analogy. When you play music through any given combination of source, amp and speakers, it sounds different. Sometimes we actually like a particular type of sonic "distortion". It's never exactly like the "original" live music, though.
Likewise, any graphics manipulation is "distorting" the original. In fact, when I take a digital image and run it through Lightroom, do a range expansion/equalization, and do a bunch of tweaks to make the image look good, I'm making much larger changes than those little scaling problems listed in the article. The point is, do you think the result looks good?
There's other important variables, such as what colors are next to other colors in the image, how long you look at the image, what else is around you, how tired you are, etc. There's no such thing as color fidelity, there's only approximations to it. Color is hard, and I mean, really hard. See Hunt, "The Reproduction of Colour", or any number of other fine texts to learn more.
Join the window installer's union, where prosperity is a brick throw away!
It's not at all complicated. Many applications do properly handle it. Nuke, Shake and other compositing apps have no problem.
Pixel^(GAMMA) -> Scale -> Pixel^(1/GAMMA) I wouldn't call that a terribly complicated process.
Or even better. On open convert it to linear. Then on save convert it back. Maybe then Photoshop and company would actually handle alpha channels correctly *grumble* *grumble*...
Note that I'm not excusing the software programs from handling this better - certainly not Photoshop - but it's 1. not a new revelation and 2. certainly not a "scaling algorithm bug".
In what sense is it not a scaling algorithm bug? The images look different after scaling than before, when interpreted in accordance with the appropriate specs. It seems to me that the specification for the scale function is something like "returns an image that is as visually similar as possible to the original, but reduced in size by the specified amount." It might be known, and it might be better described as using the wrong algorithm than an algorithm bug, but it's definitely a bug in the program.
The rest of the post I basically agree with: the differences are minor except in weird test images. However, if I want to adjust the brightness, I'll do that. If I edit the photo at full res and then save at lower res for use on the web, I don't want the result to look different. I wouldn't be able to tell the difference if not in swappable comparisons, but one might still look better.
Actually a good scaling algorithm should perform a lowpass filter when downscaling. This is similar to downsampling of digital audio where you do need to filter out frequencies above half the sampling rate. Leafing these higher frequencies in would cause noise because they can not be faithfully represented in a lower resolution file.
The 270-year-old professional portrait artist tells me there is just a difference and photography hasn't made it up yet. He isn't giving up his canvas and I can't really blame him. I guess this isn't the film's fault and it shouldn't affect anything if the subject can be made to hold motionless for several minutes in a brightly-lit setting, but it makes you question if human-driven devices and systems should be abandoned at the current rates.
The "magic" tools are done right. Scaling (for stamps) needs fixing.
It's GPL. Grab the code if you want it: rgblinear.c and rgblinear.h have what you need.
(and yes, the difference is very noticable for special-effect paint tools)
Check the Gimp bug database. It has complaints going back many many years. Trouble is, a bug report is easy to ignore and/or misunderstand.
Getting this fixed requires sitting down with all the core developers and using small words until it sinks into their thick skulls.
Eric Brasseur, in his otherwise excellent web page on the subject writes:
Technically speaking, the problem is that "the computations are performed as if the scale
of brightnesses was linear while in fact it is an exponential scale." In mathematical terms:
"a gamma of 1.0 is assumed while it is 2.2." Lots of filters, plug-ins and scripts probably
make the same error."
That is sort of right in a sense, but the assumption that it is a problem with the scaling algorithm and worse, that it should be fixed in the algorithm are both wrong and a dangerous way to start thinking about this situation. In fact, this isn't really a bug anymore than the misuse of any tool is a bug. What is really needed is more knowledge and expertise on how to create and utilize a color managed imaging workflow.
Almost all image processing algorithms contain simple addition somewhere and for simple addition to wok as you might expect, the image encoding function must be linear. (Adding logarithms for instance actually results in a multiplication operation. That is how slide rules work. What's a slide rule? Hmmm...) Scaling involves filtering and nearly all filtering algorithms involve addition, but so does a blur, over, blend, key, mask, matte, color correction, dithering and quite a few others. This problem isn't limited to scaling operations and it isn't a actually a bug at all, but rather an education and understanding issue.
The right thing to do is to linearize your images prior to doing any kind of manipulation and keep them linear throughout the whole image manipulation process, only transforming them to something else for final delivery (or approval if that is an issue). By linear, I mean that the relationship between the luminance encoding and light is linear. You see, it isn't the image or the file that is linear or non-linear, it is that transfer function between code values and light. When I say linear, I mean that for a function f, f(a) + f(b) = f(a+b) and a*f(b) = f(a*b). Anything else is non-linear.
The problem with working with linear images is that unless you are careful and know what you are doing, your monitor being a non-linear viewing device, won't display the images correctly, so if you linearize your images, they won't look "right" on the monitor, although they will be correct mathematically. The solution to that is to employ LUT in the viewer such that linear data is displayed correctly. Photoshop can do this and has been able to do it for years. The fact that people don't understand how to use that feature and why doesn't mean that there is a bug in the scaling algorithm.
By the way, high end 2D image manipulation tools like Shake and Nuke to name two are used in the most demanding imaging application possible, motion picture visual effects. With regard to internal image processing, they presume that data is linear. They rely on the expert knowledge of the user to insure that the proper image encoding, that being linear to light is used.
Lastly, I think thinking of this as a bug in scaling algorithms and trying to fix it by revising scaling algorithms rather than recognizing that it is really a problem with image data being encoded non-linearly is a completely wrong headed approach. It only addresses one use case, that being scaling and ignores all of the other problems with non-linearity. A better approach would be to educate people as to what is actually happening, why it is happening and then just teach them to use the tools they have.
Just presuming that the input image has a 2.2 gamma and correcting for it in just the case of scaling but not in other 2D image manipulations only just serves to muddle the issue.
Ultimately, you want is a display subsystem (display card and monitor) that has been profiled and then corrected to match as closely as possible some idealized target like sRGB or Rec. 709 just to name two. The easiest way to do that i
KDE's KolourPaint (MS Paint clone) gets it right! Yay KDE!
It is dangerous to be right when the government is wrong.
a low pass filter wouldn't work per se, as the color encoding used is non linear. most filters instead takes each channel of the picture producing an area average akin to a lowpass filter, but this is exactly where the problem lies as a: checkered black and white image (that is, every odd pixel on one row is white and the others black, reversed on the next row) is averaged to a contiguous half tone gray image, which is not the expected result because the gray scale is not linear in terms of luminance, so you'll need to have it averaged to a rgb(180,180,180) instead that to a rgb(128,128,128)
*180 is not the actual value
They summary already names a fix for Gimp (GEGL), but the posters only seem interested in whining instead of RTFS. Sigh.
Well, I RTFA and I completely understand why this is happening...the so called "sample photo" is covered with annoying little gray lines. No wonder the picture looks bad when you scale it. The dude needs a new camera.
I noticed this bug the other day but I thought perhaps I made a mistake somewhere. I am creating a Drupal site for photos and it has a dark background. I was just testing out the image upload and I used an unscaled image. Later I scaled the same image down to save space and re-uploaded the image. The brightness was noticeably different. It's actually very hard to tell in a lot of cases, especially with a brighter background. A dark background really makes the bug apparent.
Time makes more converts than reason
"the assumption one makes is that these integer values are not photons ^( 1/gamma) but simple photon counts (scaled to the 0-255 range)."
That is a very reasonable assumption to make, and one that most people who don't know anything about gamma make. Unfortunately, it's flat out false. Both MATLAB and PIL return the gamma compressed data, which, unless you used a linear machine vision camera (and you should if you're serious about this stuff), which of course had no gamma compression to begin with. If you need proof, load and save an image. The image data will be bit for bit identical to the original, indicating that no conversions were performed. (note that the header might have slightly different metadata, and JPEG re-compression is usually always lossy)
Gamma is so rarely handled properly, even by scientists and engineers, that OpenCV (the most popular library for computer vision) does not even contain a function for doing gamma (de)compression.