Slashdot Mirror
Scaling Algorithm Bug In Gimp, Photoshop, Others
Wescotte writes "There is an important error in most photography scaling algorithms. All software tested has the problem: The Gimp, Adobe Photoshop, CinePaint, Nip2, ImageMagick, GQview, Eye of Gnome, Paint, and Krita. The problem exists across three different operating systems: Linux, Mac OS X, and Windows. (These exceptions have subsequently been reported — this software does not suffer from the problem: the Netpbm toolkit for graphic manipulations, the developing GEGL toolkit, 32-bit encoded images in Photoshop CS3, the latest version of Image Analyzer, the image exporters in Aperture 1.5.6, the latest version of Rendera, Adobe Lightroom 1.4.1, Pixelmator for Mac OS X, Paint Shop Pro X2, and the Preview app in Mac OS X starting from version 10.6.) Photographs scaled with the affected software are degraded, because of incorrect algorithmic accounting for monitor gamma. The degradation is often faint, but probably most pictures contain at least an array where the degradation is clearly visible. I believe this has happened since the first versions of these programs, maybe 20 years ago."
Photographs scaled with the affected software are degraded, because of incorrect algorithmic accounting for monitor gamma.
Seriously!
I have a theory on why this has gone unnoticed for so long, but I'll keep it to myself...
Most scaling algorithms treat brightness as a linear space, so e.g. if you're doing downscaling to 1/2 the size in each dimension, collapse 4 pixels into 1 by setting the 1 pixel to the numerical average of the original 4 pixels. But, most images are displayed with an assumption that brightness is a nonlinear space, i.e. gamma > 1. Therefore, scaling changes the perceived brightness, an unexpected result.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
This is only a bug depending on what you are doing with your final images. One of the things that annoys me is that many image manipulation programs do not actually explain the primitives they are using. The result can be a complete mess depending on what you are trying to accomplish. This article is an example of this effect.
If you want photo-realistic results, then you need to take Gamma into account. However, very few file formats specify the Gamma, the grey level, the white level, the black level or the colour space of the original image. The result is that the many imaging operations must be wrong, as they can never be accomplished the way intended. For the most part, no one cares. This person found an application where people care.
Excellent point. Just to be safe though, I'm going to take another look through my porn crypt to see if that's true.
BRB.
Well, I am SURE glad I'm using Linux^H^H^H^H^HWindows^H^H^H^H^H^H^HMac^H^H^Hshit.
It is pitch black. You are likely to be eaten by a grue.
The responses that they post are also inaccurate it seems.
From
meanwhile, I see a grey rectangle in firefox, and I still don't get what that signifies.
I've been telling people for years that I look better in person.
I told them that there's something wrong with pictures of me.
HA!
Now I know.
It's the Scaling Algorithm BUG!
The data in the pictures is not linear data. It assumes that it will be displayed on a system that introduces a gamma of 2.2. (If your display system does not do that physically, it should correct for this.) That is, a gray 127 should not display as halfway between a white 255 and a black zero, in terms of light output. (It should *appear* halfway between them visually, because your eyes aren't linear — that's (part of) why gamma is in use in the first place.) So, a checkerboard pattern of white / black squares will have half the luminosity of the white squares. When scaling down, software will turn it into a bunch of gray pixels. But they should be gray pixels of value 186, not 127.
The page is not well written, but his example images make the issue very clear. It's not about your monitor gamma; it's about the "standard gamma" that all image files assume your monitor has.
Come on, this isn't news...
Helmut Dersch (of Panorama Tools fame) certainly posted about this before;
http://www.all-in-one.ee/~dersch/gamma/gamma.html - Interpolation and Gamma Correction
There's no factual error in the scaling algorithm, as the /. headline would like you to believe - it's a color space (linearity) issue; you have to do your calculations in linear space which means a typical photo off of a camera/scanner gets the inverse of an sRGB curve applied (a gamma of 0.454545 is 'close enough' if you can't do the proper color bits). Then scale. Then re-apply the curve.
And no - for real life imagery, nobody really cares - the JPEGs out of the cameras and subsequent re-compression to JPEG after scaling will have 'destroyed' far more data than the linearity issue.
They're nice example images in the story, but they should be called 'academic'.
Gray rectangles in Picasa 3 and in Irfanview 4.25 :-(
I am sure the Chinese government prefers the current implementation.
Gamma is often poorly understood even by people doing scientific and engineering work using images.
Does your algorithm depend (explicitly or implicitly) on the light intensity -> pixel data mapping?
If NO: You're probably wrong. Go read about gamma. Just because the picture looks right to you, doesn't mean it looks right to your code.
If YES:
Do you have the luxury of taking the pictures yourself?
If NO: You're stuffed. Pretty much all images on the internet and most public research databases have unknown/unreliable gamma curves.
If YES:
1. Spend a lot of time calibrating your camera yourself. This is only cheap if your time is worthless
or
2. Buy a machine vision camera. A $600 machine vision camera will have the specs of a $50 consumer camera, but at least you will know the gamma curve.
or
3. Ignore the gamma issue, cross your fingers, hope it's not hurting your performance, and publish your results knowing that you're in good company and nobody will call you out.
Actually, any well-specified file format will specify the gamma. Not all allow you to set it per-file, but they do specify it. Normally this is a line in the spec that reads something like "color values use the sRGB color space" or similar — which specifies a gamma of 2.2 (roughly). And sRGB with it's nearly 2.2 gamma has become so standard that assuming anything else (in the absence of a clear spec) would be idiotic.
The example images that make it really clear are academic examples. But the scaled photos are all enough of a change to be worth noticing and caring about if you're a serious amateur photographer (never mind professional). And they don't look particularly unusual to me (I haven't looked for odd trickery, but I assume he's being honest here).
The basic issue here has to do with gamma curves and the way they're being handled (they're not).
Most image files on your computer (BMP, JPG, PNG, etc.) are stored in the sRGB color space. sRGB defines the use of a gamma curve, which is a nonlinear transformation applied to each of the components (R, G, and B). The issue here is that most scalers make the assumption that the components are linear, rather than try to process the gamma curve. While this does save processing time (undoing the gamma curve then redoing it), it does add some error, especially when the values being scaled are not near each other.
So does this matter? Well, in some pathological cases where there are repeated sharp boundaries (such as alternating black-white lines or fine checkerboard patterns), this would make a difference. This is because the linear average of the pixels (what most image scalers use) yields a different result than if the gamma value was taken into account. For most images (both photographic and computer generated), this shouldn't be a big problem. Most samples are close in value to other nearby samples, so the error resulting from the gamma curve is very small. Sparse light-dark transitions also wouldn't be noticeable as there would only be an error right on the boundary. Only when you exercise this case over a large area does it become obvious.
One final point: this gamma scaling effect would occur regardless of the actual scaling algorithm. Bilinear, bicubic, and sinc would all have the same issue. Nearest neighbor interpolation would be unaffected, but in these cases, the output would look far worse.
Ok, so he made a very informative page about it, but this is still a well known effect. It affects practically everything you can do in image editing. Blurs, etc. Most people neither notice nor care. It's rooted in the fact that most images come with undefined black and white points and a gamma chosen for artistic effect rather than physical accuracy. Thus correctly converting to linear gamma is hardly ever possible. You can still correct for monitor gamma to avoid some rarely seen inconsistencies and artifacts, but most people don't even notice, so why bother? However, Photoshop does have everything you need to avoid the effect completely, even in the ancient Photoshop 6.0.
Here's how to properly resize in Photoshop:
1. Convert mode to 16 bit (to avoid tone aliasing in the next step, no other influence on the calculations)
2. Convert to profile, select "Custom RGB", set Gamma to 1.0 (this converts the internal image data to linear gamma, no visible change because the image is color managed and corrected back to monitor gamma on the fly)
3. Image Size
4. Convert to profile, select "Custom RGB", set Gamma to 2.2 (default)
5. Convert mode to 8 bit
Done. You can substitute your favorite image filter for the image resize. Unsharp mask works much better at gamma 1.0, for example. Of course you can use several filters before converting back to monitor gamma and 8 bit.
If you're looking for lcd test images, http://www.lagom.nl/lcd-test/ is probably better. It's got a whole bunch of images dedicated to various monitor problems, along with explanations.
If you're a *serious* amateur photographer, then you should already know about this and not be using those apps / using them in the color modes (to use Photoshop parlance, as I guess most serious amateur photographers will have a copy (legit or otherwise of that)).
I guess the argument would hinge on who is a serious amateur photographer and who is just a regular amateur photographer.
As for the actual examples - sure, you can see the difference.. especially since they're in before/after -style swappable pages. If I presented you a random image off of a random image gallery online, though, would you be able to tell the difference?
If I showed you an online photo album and pointed at an image's thumbnail, had you click the thumbnail, and open up the full size image.. would you notice that it was scaled to thumbnail incorrectly?
"Nobody really cares" may have been too broad a statement - but those who really care, already know.. or reasonably should know.
imho.
Note that I'm not excusing the software programs from handling this better - certainly not Photoshop - but it's 1. not a new revelation and 2. certainly not a "scaling algorithm bug".
It's basically an implementation issue. The algorithms may be fine as intended ... in linear space. The programmers that implemented them didn't understand linear vs. gamma, or didn't care, or had a fire breathing PHB on their back. Hence we get junk software.
At least all MY image processing code always works in linear space. Bu merely converting 8-bit gamma to 8-bit linear is no good because that now introduces some serious quantizing artifacts (major banding effects happen). So I convert the 8-bit gammas to at least 30 or 31 bit integer if I need processing speed, or all the way to double precision floating point if I need as close to correct as possible. After processing, then I convert back to 8-bit gammas. Even then, you can't totally eliminate some banding effects that result from being in 8-bit. If you can get more bits from the raw images from your camera, that's the best to use. Apparently many JPEG compressors are also doing their DCT calculations in the non-unit gamma space instead of the linear space, too (which reduces the effectiveness of the compression somewhat, and may add more compression artifacts).
now we need to go OSS in diesel cars
My software has been calculating in linear space for over a decade now (this is the Nuke Compositor currenlty produced by The Foundry but at the time it was used by Digital Domain for Titanic). You can see some pages I wrote on the effect here: http://mysite.verizon.net/~spitzak/conversion/composite.html. See here for the overall paper: http://mysite.verizon.net/~spitzak/conversion/index.html and a Siggraph paper on the conversion of such images here: http://mysite.verizon.net/~spitzak/conversion/sketches_0265.pdf, in fact a lot more work went into figuring out how to get such linear images to show on the screen on hardware of that era than on the obvious need to do the math in linear. Initial work on this was done for Apollo 13 as the problems with gamma were quite obvious when scaling images of small bright objects against the black of space.
For typical photographs the effect is not very visible in scaling, as the gamma curve is very close to a straight line for two close points and thus the result is not very much different. Only widely separated points (ie very high contrast images with sharp edges) will show a visible difference. This probably means you are trying to scale line art, there are screenshots in the html pages showing the results of this. Far worse errors can be found in lighting calculations and in filtering operations such as blur. At the time even the most expensive professional 3D renderers were doing lighting completely wrong, but things have gotten better now that they can use floating point intermediate images.
One big annoyance is that you better do the math in floating point. Even 16 bits is insufficient for linear light levels as the black points will be too far apart and visible (the space is wasted on many many more white levels than you ever would need). A logarithmic system is needed, and on modern hardware you might as well use IEEE floating point, or the ILM "half" standard for 16-bit floating point.
Well?
I think the author specifically isn't stating whether the scaling is correct or not - it is; the whole story doesn't relate to scaling at all, but rather color space and how -it- affects, among other, scaling. Yes, with filtering - scaling without filtering can hardly be called scaling at all as you're just discarding data - and for anything but multiples of 2 (4x, 2x, 0.5x, 0.25x, etc.) that'd have a whole 'nother set of problems.
The author, I think, is suggesting, quite rightly so, that while...
The desired result for scaling down likely being that of the same visual image as when you simply stand further back.
( although at some point the resolution limit of a display and the image itself being presented on that display prevents that concept from being applied to "moving your eyeballs closer to the screen" for scaling up. )
It seems crazy to me to embed a particular Gamma value into an image. ...In fact it seems so crazy I must be missing something. Am I?
The article actually touches on this point. The sensitivity of the human eye isn't linear. If you use a linear scale to store luminosity information for an image, you waste a lot of bit depth at high luminosities - the eye has difficulty distinguishing between very bright and very bright plus a little tiny bit. On the other hand, the eye is very good at telling the difference between very dark and black. You need a lot of finely-graduated steps at low luminosity or else your shadows get jaggy.
If you uniformly (linearly) space out luminosities on an 8-bit (256-shade) scale, you store a lot of uninteresting information at the high end, and lose out on visible detail at the low end. A scale with gamma of 2.2 (typical these days) fits a full twenty-eight grey values between 0 and 1 on our hypothetical linear scale. To maintain that kind of luminosity resolution (down where it matters), you'd have to store an extra five bits on your linear scale. An extra sixty percent costs.
~Idarubicin
Well?
Somehow I don't think you've given him enough time.
Several people have spoken about "linear" RGB. That's nice and gets rid of some small level of distortion introduced by the non-linearity. However, it only starts there. For example, the eye sees R, G, and B differently. It is more sensitive to green than red, and to red more than blue, but it's not even that simple as the equations in your eye's processor are much more complicated. Many algorithms that treat the three "equally" are going to change the perceptual mixture. One can use other color spaces, such as HSV, Yuv, xyY, etc. with different advantages and disadvantages
Sound makes a good analogy. When you play music through any given combination of source, amp and speakers, it sounds different. Sometimes we actually like a particular type of sonic "distortion". It's never exactly like the "original" live music, though.
Likewise, any graphics manipulation is "distorting" the original. In fact, when I take a digital image and run it through Lightroom, do a range expansion/equalization, and do a bunch of tweaks to make the image look good, I'm making much larger changes than those little scaling problems listed in the article. The point is, do you think the result looks good?
There's other important variables, such as what colors are next to other colors in the image, how long you look at the image, what else is around you, how tired you are, etc. There's no such thing as color fidelity, there's only approximations to it. Color is hard, and I mean, really hard. See Hunt, "The Reproduction of Colour", or any number of other fine texts to learn more.
Join the window installer's union, where prosperity is a brick throw away!
Actually a good scaling algorithm should perform a lowpass filter when downscaling. This is similar to downsampling of digital audio where you do need to filter out frequencies above half the sampling rate. Leafing these higher frequencies in would cause noise because they can not be faithfully represented in a lower resolution file.
The 270-year-old professional portrait artist tells me there is just a difference and photography hasn't made it up yet. He isn't giving up his canvas and I can't really blame him. I guess this isn't the film's fault and it shouldn't affect anything if the subject can be made to hold motionless for several minutes in a brightly-lit setting, but it makes you question if human-driven devices and systems should be abandoned at the current rates.
meanwhile, I see a grey rectangle in firefox, and I still don't get what that signifies.
Right-click the image, then click view image. You'll see the image full-scale, like the first image. Scaling it down 50% shouldn't make it gray.
They summary already names a fix for Gimp (GEGL), but the posters only seem interested in whining instead of RTFS. Sigh.
IMO, what it actually means is that the so-called image is deliberately designed to be as catastrophically horrible as possible when scaled down.
Yes, the image is designed to exploit the bug in way that makes it very obvious that the scaling is wrong, but no, the gray rectangle is not the correct solution. The problem is that the browser or the image application assumes that the brightness is stored on a linear scale, while in reality its stored exponential one. Thus when you scale things or apply other filters the brightness will be messed up. In real photos it will be less noticeable then here, but it will still happen.
The correct way would be to transform the image to a linear scale before applying the filter and then restoring it the exponential one for display. A simple example of how to do that in Gimp (using gausian blur instead of scale, but it is the same bug) would be this. The left image is the original from the article, the middle one is blur applied and the right one is applying a gamma of 0.5 to get it to a linear scale, then the blur and then a gamma 2 to restore it to its original scale. As you can easily see the right one looks like the left one, but blurred, while the middle one has no resemblance to the original, all the information got lost due to the bug. In practice you would need a higher bit depth to make this trick practical, as else you would end with banding artifacts.