Content-Aware Image Resizing

An anonymous reader writes "At the SIGGRAPH 2007 conference in San Diego, two Israeli professors, Shai Avidan and Ariel Shamir, have demonstrated a new method to shrink images. The method is called 'Seam Carving for Content-Aware Image Resizing' (PDF paper here) and it figures out which parts of an image are less significant. This makes it possible to change the aspect ratio of an image without making the content look skewed or stretched out. There is a video demonstration up on YouTube."

Comments

Re:I For One

[cnettel]

It's not compression as we know it, Jim. It's more like scaling on totally overcool steroids. The basic idea seems rather simple. I would even imagine you could get a bit of enhanced picture quality by coding simplified vector info on seams, and then doing a normal JPEG of a downscaled picture. That would be a quite contrived way to get a kind of VBR-like behavior in normal JPEG. One issue with JPEG is, after all, that redundancy is detected and handled on the block level, while this algorithm works along arbitrary paths.

I'm really impressed. Again, maybe not too hard to implement at first, but probably damn hard to get working perfectly, and I might just be ignorant (and I'm entitled too, it's far from my field of work), but I've not seen anyone doing it before.

A picture speaks a thousand words...

[Aphrika]

So does this mean you're taking some of those words away?

There are probably a few situations where the 'unimportant' bits of an image are still as relevant as the rest. Sports photos for instance - especially those played on grass - would not give you a true picture (literally) of what's going on in the scene.

This'd be good for reference photos - like the animals at the start of the YouTube video, but applications where precision and distance are required wouldn't benefit. Nice bit of work though and I reckon with some smart scaling embedded too (rather than its 'folding effect'), it'd cater for most image retargetting requirements.

Re:A picture speaks a thousand words...

[zippthorne]

I don't know whether I'm "used to it" or not, but after watching the video, I'm totally ready for more intelligent image resizing that isn't quite scaling. Most of the applications I see this being used in don't really require that the exact photographic position (which really isn't the same as what you'd see if you were there) relationships be maintained anyway.

Hopefully someone will write a GIMP plugin and we can all experiment with it. Also a firefox plugin. Obviously some metadata will eventually need to be included in the the images to delineate faces and whatnot, but web designers can easily handle sloppy painting-over in photoshop type tasks.

Re:A picture speaks a thousand words...

[pclminion]

There are probably a few situations where the 'unimportant' bits of an image are still as relevant as the rest. Sports photos for instance - especially those played on grass - would not give you a true picture (literally) of what's going on in the scene.

Sorry -- "true picture?" That assumes such a thing can exist in the first place. Take a color-blind viewer for instance. Can he (and I say he because statistically, most color-blind people are male) look at ANY image and say that he is seeing the "true image?" How is his experience any more or less true than the experience YOU have when you look at the image?

Any scaling of an image, by definition, must remove (or insert, if up-scaling) information in an image. Usual scaling techniques insert or remove a constant information density across the image. This means that areas with low information lose just as much fidelity as regions with high information. A better method would have removed more information from the area that is already low in information to begin with, leaving more information in the area where it matters. This is exactly what this new algorithm does.

So it is fairly obvious that this method is superior, from a purely information-theoretic standpoint, to typical scaling algorithms. Are there images where its application might be inappropriate? Yes. Compressing an image of an abstract piece of art might do unforgivable damage to it. There is a simple solution -- do not use this algorithm on such images.

I Think You'll Find

[JamesRose]

10 Seconds of work there, most probably a good deal longer finding a picture that is easy to do it to...

Re:Slightly Strange

[Nutty_Irishman]

I think you're missing the point of their method, which is to provide realistic images during rescaling that aren't corrupted by blind interpolation (equal averaging). In downscaling the images, it preserves parts of the images that would lose their information through downscaling (e.g. complex textures, people), while at the same time removing textures that would not lose information through downscaling (sky, water, sand). The sky, water and sand will still look like sky water and sand whether it's at 1/4 or 10x resolution, people however look much different if you try and downscale them or upscale them(they would appear blurry and hard to distinguish). The same works in reverse. The sky is still going to look like the sky whether you scale it to 10x or 5x-- it would still look natural. Tree's on the other hand, would not. Once you start to scale up the trees you would expect to be seeing different characteristics-- leaves, branches, etc. Any type of scaling up of a tree would make it seem very blurry and unnatural (lacking leaves, branches, etc.)-- you cannot create an additional information that isn't present in the original image. Therefore, the most natural looking image would be to increase the sky.

It's not perfect of course. I'm guessing that if you had a picture of two people next to each other, one with a solid colored shirt, and the other with a striped colored shirt, that the solid colored shirt guy would get skinner than the striped when shrinking, and the reverse when enlarging. However, it's a neat idea, and I look forward to reading the paper.

Paranoia! It's not just for Gimps

[Anonymous Coward]

"More likely Adobe will write them a check and license it to make sure that never happens."

Is that check going to cover the removal of their paper from above and the ACM archives, let alone OUR archives?

Re:The paper via ACM

[Anonymous Coward]

I think you've got it except for a small detail in the "Remove objects", which the narrator alludes to around timestamp 4:01 of the video. You might want to add:

Step 6: Extend image to match original size using the previous extend image algorithm

(Of course, I leave the obligatory Profit step as an exercise for the reader).

removing the intended layout

[__aapbzv4610]

What about artistic photographs? Most photos in that sense are planned to have a certain layout, composition, empty spaces, etc. Say I make a nice panorama shot with a 6:1 aspect ratio. Now my photo that took careful planning is reduced to a 4:3 with all the 'unimportant' spaces removed? Maybe it's just me, but there seem to be lots more instances where this would hurt than help. Journalistic images? Sports photos? Oh, the image can't fit, let's get rid of everything between the 50 and 20 yard lines. There aren't any players standing there. I really only see this being beneficial for web ads. Instead of creating square, vertical, and horizontal versions of the same ad, just make one and let the image be 'resized' accordingly.

Re:DP Approach

[The New Andy]

I certainly hope it isn't patented, since by just watching the video once (without sound) I was able to to make my own implementation in C in under two hours. I completely agree that it is a cool idea, but I think the reason it is so cool is that the parts they used to build it are all so simple/well known - it is just a really novel combination of ideas that people have already come up with. The idea of a patent (I believe) is so that an inventor won't keep their invention to themselves, so that people can see how it all works and it benefits the public. There aren't any hidden tricks here - the (image-processing) public can easily work out how it is working just by looking at it.

Just in case I haven't been clear - I think that the idea is awesome, novel and brilliant. And I believe that it is possible for something to be awesome, novel and brilliant but also 'obvious'. Just like in maths when they showed you complex numbers, and how they bring some sanity into the system. Once they give you the hint that the square root of a negative number can be defined, then you can go away and easily derive all the cool things like Euler's form and whatnot. Now replace 'the square root of a negative number can be defined' with 'you can crop a jagged column from an image' and you have a pretty good parallel.

Re:The Commissar Vanishes

[Anonymous Coward]

PC.. The 'correctness' part of the phrase really irks me. It's censorship, propaganda and thought police all at the same time. At least with a bigot you know where you stand - with a PC freak nothing is sacred, noone is safe. Our society would be far better off with blatant racism over this politically correct crap any day - at least racism is in the realm of debatable idiocy, the kind of PC stupidity you presented just has no rational, logical or sane evaluation possible.

Re:Not ready for Prime Time

[pclminion]

It has nothing to do with edge detection. The algorithm simply detects paths of minimal gradient which lead from one side of the image to the opposite side. This can be used to produce a "pretty picture" which shows the edges -- but this is merely fallout.

They showed what I thought were several realistic photos with complex backgrounds, and the algorithm did well overall, except on structures where people are closely attuned to exact detail -- such as human faces. If we weren't innately wired to process faces in incredible detail, we wouldn't even notice the distortion.

So it's not perfect. Can you show me something in this world that is? And I don't think there has been any mention of "prime time" application, whatever that means.

Re:The Commissar Vanishes

[compro01]

I think it's obvious that this technique is just plain cool and has great potential for beneficial use, even if it might be used for ill. what do you expect? it's technology. technology works to the highest bidder, not the people with the highest morals.

Re:Great - We can do this, but should we?

[vasanth]

Your comment seems to be similar to the headline on tabloids.. Just because a technology could be used for negative purposes does not mean that it should not be developed.. If your reasoning was used, we should have all been living in caves by now..

By your reasoning
Cars can be used by criminals to travel faster.
A knife can be used to kill
Electricity can be used to kill
Computers can be used by the govt to collect more information abt us effectively

Is that really what we want?

see the flaw in the logic?