Slashdot Mirror
Image Processing By Example
Aaron Hertzmann writes: "My collaborators and I will present a paper called
Image Analogies at
SIGGRAPH 2001 this summer, where we describe a machine learning method for 'learning' image filters for example.
For example, given a Van Gogh painting, the algorithm can process other images to look somewhat as if they were painted by Van Gogh."
"It can also 'texturize' images based on a sample textured image, e.g. to create landscape photos. It can do many other types of filters, as long as you give appropriate 'before' and 'after' examples to learn from." I especially like the idea of inferring a high-resolution image from a low-res one. The software is available for Windows and Unix, and "the source code is freely distributed for educational, research and non-profit purposes."
Why are so many people talking about Van Gogh rather than talking about taking pictures of industrial wastelands and applying the industrial wasteland filter to woodland scenes, or their own face, or computer renderings? This is potentially a _fascinating_ tool, and I want one. But the last thing I'd use it for is imitating paint brushstrokes. I can make brushstrokes with real paint, or fake brushstrokes with Painter or something. I can't make a digital image of a guitar made out of water ripples with anything but this- at least, not so effectively.
This _is_ a great thing, but it's not an artist replacement. It's a tool. It's convergence. Your ability to use it is very much dependent on what you can imagine, such as taking a portrait and doing it all in the textures of woodgrain while keeping the colors the same, or giving everything the textures of concrete, or fur, or cloth. The fact that it's not strictly texturemapping is what makes it potentially huge in significance...
Could this one day be used to turn my dull crud into something Fitzgerald or Hemingway or even Asimov or Heinlein might have written?
Then you could take news from say CNN and process them to be Slashdot-ready. Imagine a Jon Katz or Taco filter.
__
__
Men with no respect for life must never be allowed to control the ultimate instruments of death.
GW Bu
Perhaps this software could be trained to learn from images of masonry, cliffs or pebbles. It could then apply the knowledge to create 'petrified' images of Star Wars actresses.
-- Ed Avis ed@membled.com
At first, I was going to say, "Well, I guess this is further proof that using computers doesn't count as fine art." (Only joking, I'm a BFA student and use computers plenty.)
Then I looked at the images this program produces. I guess I can't expect anything BETTER than what is stuck into it - Van Gogh would never have painted scenes like those. Hell, any beginning art student could "paint like Van Gogh" - just mix your paints a bit thick! Perhaps the program needs a content selection algorythm. (I'd like to see it do a self portrait.)
At any rate, the "holy grail" of this technology is to emulate air brushed velvet. Not a fan of elvis but love the look of painted velvet? Just use a pic of your favorite celeb! Finally, Natalie Portman recognised as she should be!
Know what I like about atheists? I've yet to meet one that believes God is on their side.
So, you're going to find all the pr0n and remove the ears? Whatever floats your boat.
This was an (admittedly sad attempt at) humor, my friends, humor...
The problem of finding "tanks" or whatever is actually an old one that goes back to the late sixties and seventies when the CIA was planning on sending up real reconaissance satellites with multispectral capabilities and NASA was sending up their interplanetary probes. The math used for this problem is older than that, but some new methods are being implemented by both government agencies and private corporations. It used to be much easier when we were just taking visible light photos. For instance, the wife of a friend of mine used to specialize in examining runway lengths. All she would do all day is look at photos of runways and compare their lengths to previous images to determine if they were being upgraded for bigger/more powerful planes. Now you can imagine that given the same image of a runway, but now you have twenty to one hundred and twenty separate images at different spectral wavelengths why there is a need for automation. Unfortunately all of the methods to analyze this type of data require fairly significant amounts of computational time and analyst time to extract the information you are looking for. There are many ways of going about looking for these "tanks" or whatever you want to look at in images (SCUD missiles, concrete desguised to look like granite, gold bearing strata, oil bearing strata, alpha ganglion cells in the retina, etc etc etc....) Using traditional multispectral classification techniques, one does not neccessarily have to implement neural networks. (There are times they can be problematic, but this is probably due in many ways to immature code) You can simply use supervised classification techniques where you specify the spectral fingerprint (you have to know what you are looking for) and have the algorhithm extract or highlight the pixels it finds. Alternatively, if you do not know exactly the spectral fingerprint of what you are looking for, you can perform an unsupervised classification such as a k-means or ISODATA classification. These techniques can be fairly time intensive from both a computational and an analysts perspective thus the push for more automated methods. It is the automation that has proven to be difficult. Many things like co-registration of hyperdimensional images is easily implemented using embedded fiducial points, but the actual analysis and taking apart of hyperspectral data often is more of an art than science (I hate to say that as thats what people say when they don't understand all of the aspects of the problem but....)and as such it currently requires lots of "biological" as opposed to silicon supervision.
Visit Jonesblog and say hello.
It looks like the training data needs to be similar to the target data if the results are to be visually similar. In the watercolor example the training data of the apples produces the best results on the photo of the tulips. The filtered landscape photos don't look anywhere near as good. I wonder if you could use several sets of training data (still life, landscape, portrait, etc.) to create a more general purpose filter.
I would think that approximating the unfiltered source part of the training data from a painting (like the Van Gogh example) would produce kind of twisted results from photographic data. I wonder if they've tried getting a "real" painter to mimic Van Gogh's style from a photograph and using both the photo and the painting for training data. I expect the results would be better. (Although that wasn't really the case with the pastel examples so maybe not.)
Maybe I'll get a chance ask at the show.
It's not just using the filter; it is creating the filter.
--- witty signature
Take a look at some of the other images. And they didn't just "go into xv and sharpen the image" they got the computer to figure out how to do it on its own, given only two inputs!
I doubt you could program something similar if your life depended on it, looser.
ReadThe ReflectionEngine, a cyberpunk style n
That is what they did, although in the case of the paintings, they had the 'after' shot to start with and simply blured it to get the 'before' shot. In other words, there was no 'orgional' filter.
ReadThe ReflectionEngine, a cyberpunk style n
. In the case of Van Gogh, you just need to fuzz up the image a little. But there's a lot more to Van Gogh than fuzziness!
Wow, THEY DID Did you even read the site? or just look at the example? Here's one of some guy named Lucian Freud
And keep in mind they didn't 'just tell the computer to 'fuzz up' the image' they just gave the computer a copy of Starry night, and a blurry copy of Starry night and said 'figure out how to go from the blurry one to the original'. After that, the computer did all the work
ReadThe ReflectionEngine, a cyberpunk style n
You can use PPM compression for this.
Say you have samples of the works of N authors and a text T that has to be identified. Compress the text N times, each time the system is initialized with the samples from another author. T will usually compress best when the system was initialized with the samples from its own author.
See Bill Teahan's PhD thesis.
Well, while filters have existed in PhotoShop or the GIMP for ages, they tend to do a not altogether excellent job. Did you look at the one where they took an image and applied a watercolour filter? The result looked nothing like a watercolour, but had encoded in it what some folks might think a watercolour produced from an image might be. They then fed other photographs into the engine and produced what look to my untrained eye like real honest-to-goodness watercolours. I think that's pretty neat: they created good stuff by mimicing bad filters.
The point is, this is an ecellent first step on the way to making software which can paint. It may not be here today, but we are progressing twoards that end. What I find amusing is that this software has no concept of aesthetics or beauty; it is by no means a thinking machine. And yet it creates what we find to be aesthetically attractive pieces. Indicates to me that perh. artists are not more deeply connected to the soul or heart than the rest of us after all.
What I've started to wonder is where else it's underlying principles could be used, or where this sort of technology could lead in the future.
Could it be used to analyze text from certain authors (hey, text and art are no different to a computer - treat words as "pixels" and sentences and structures flow like colors) and mimic their style? Could this one day be used to turn my dull crud into something Fitzgerald or Hemingway or even Asimov or Heinlein might have written?
I also have the following few questions:
I think that sums up my feelings. This stuff is really impressive guys, I hope the conference goes well.
---
---
"Of course, that's just my opinion. I could be wrong." --Dennis Miller
Since when did Slashdot become... ... a vehicle for people to toot their own horns and advertise their OWN papers, etc?
Yeah! A story is so much more interesting when it languishes unknown in the vastness of the web for six months, only to be discovered by the time it has officially become Old News!
Bring back the good old days of virus reports from even mistier pasts.
Become a FSF associate member before the low #s are used
Actually, text and art _are_ substantially different to a computer. The former has rules, the latter does not.
If a single pixel is off by a bit, you won't notice - your brain subconsciously blends the whole thing together anyway. If, on the other hand, the wrong word is chosen, it will stick out like a sore thumb - even if it's only a preposition.
Finally, I would posit that even if a filter could make your prose sound like Asimov or Hienlien, the real advantage both masters have is not in their actual prose but in the ideas they express - and no filter is going to be able to duplicate that.
There are a few companies (iterated.com, lizardtech.com, some others) that have been doing fractal and wavelet scaling for a while now. Pretty impressive stuff. I don't know all the theroretical details, just the practical uses. Scaling up to about 1600% is possible with no noticable artifacts (to the human eye). We've been using some of this stuff in the prepress/graphic arts industry for a while.
Jason
Wow! Does that mean that a certain staple of movies with spy-cameras crucial to the plot will finally have a basis in reality?
Every time I hear, "I can't see his face because the video's not clear enough - here, let me enhance the image," I just wanna scream...
--
So when's it going to show up in the Gimp? This is some cool stuff. People could make custom filters without programming anything, just train the filter to do what you want. Let the computer do the thinking in terms of imitating the effect you want. Very very impressive.
"I may not have morals, but I have standards."
"I may not have morals, but I have standards."
Sometimes, when I look at a Van Gogh, it breaks my heart,it is so beautiful. When I look at these pictures, I get the same nausea-induced feeling that any cheap knock-off imitation gives to me. It's not even interesting technology.
As for my interpretation of "the true meaning of the artist" - whether or not I agree with the concept of all art needing some intangible, metaphysical, or even spiritual explanation that is derived from the ego of the artist (which I don't necessarily) - Van Gogh has become a cause celebre for the artist whose life was as important as their work. It's part of the reason for the ridiculous prices associated with his work. His letters to his brother established him as a figure after his death, and a cursory read would reveal that there were specific personal and deterministic notions in his endeavours. It may be trite to yourself, but it drives the ridiculous art market today, and many would say that it began with Van Gogh. Van Gogh's work is part and parcel with his life, interminably, for better or for worse - it is impossible to seperate him from his time and its technology and image processing capabilities.
And I do stand corrected on what's different about this work and what it's for (imagine using texture by numbers to generate landscapes for computer games rather than storing the textures). I will stand by the fact that when it comes to mimicking an artist's strokes the material will only ever be as good as a) what's fed into it and b) a pale imitation, good for only some FX gag I expect to see in some music video and then vanish.
** http://www.nkhumanrights.or.kr/ ** Human rights in North Korea. 1 million estimated dead from starvation.
This is interesting and all well and good, but ultimately where it fails is that the produced image is entirely dependent on the original photograph's perception of the world. A reproduction of an image through halide crystal activation, which is enough for human memory and recognizance, but it lacks the true meaning of the artist. Van Gogh never used contrast or flat lighting as exhibited in the source pictures, and he often burst highlights with striking colors that may not have been actually present to his eye. It's what seperates him from a Turner - not just his brush stroke or how thick he worked in paint but how he saw the world. It's pretty churlish to adopt the first real expressionist painter (who deliberately attempted to paint their perception of the world rather than reproduce it) as an example of this algorithim, as the resultant images show that without an interpretation or perception this is pretty useless stuff. All I see here is a souped up photoshop filter.
** http://www.nkhumanrights.or.kr/ ** Human rights in North Korea. 1 million estimated dead from starvation.
Van Gogh filters are cool and all, but in this geeky day & age, I would be more impressed with a code filter that could be trained to make my C programs look like they were written by Linus Torvalds... you know, change the indention styles, white space usage styles, identifier naming styles, and how bout going as far to use the same algorithmic patterns too?
The question is, what would happen if you fed your average Visual Basic program written by your average VB coder, through a Linus Torvalds filter? Wouldn't that be like "crossing the beams" in Ghost Busters?
...how long will it be before somebody creates a VanGogh version of goatse?
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
I kind of wish they would have added one more picture to compare which was an actual high res shot.
So you could compare low res, high res, and filtered high res.
They do look impressive though!
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin
Hey this is pretty cool. You take a good photograph, and make it into a bad painting. Paining is more than just mixing oil paints to be the average color for part of an image... Van Gogh paintings typically emphasize the edges, the cheekbones, the jaw, the eyebrows. I'm bitter that someone can take an image, use xv to sharpen the image and then shift the colors and get a PhD out of it. I wonder if he was funded by the department of defense?
As I was going through the page I was fairly unimpressed, largely. Much of what was there has been in Photoshop (and probably the Gimp) for years now, i.e. filters to make photo's look like paintings or apply textures or whatever. ... the texture by numbers section was exceedingly cool and I could see a myriad of uses were someone to port it to Photoshop's plugin format. Being able to take an image, mask it with various different colors, and then have an entirely new image generated based on the textures you extracted ... yeh, thats nifty =) /. in a while.
However
By far the coolest thing I've seen on
I'd be more impressed if they had trained the software to imitate multiple artists, such as O'Keefe, Rembrandt, Klee, Gauguin, Monet, Picasso, etc., shown as a single image, as processed by all of the above, side by side.
__
They were working on this kind of image recognition in a JPL lab near my own in the early '90s. (I was an intern working on the flight computer for CRAF/Cassini at the time.) They were using some funktastic optical computer jive and I heard the thing was good... damn good. There was even a toy parking lot full of Hotwheels cars, and they would have the computer try to pick out a specific car, which might be in a group of other similar cars, or partly covered by a building, etc.
If this was going on in an unclassified lab almost 10 years ago, I imagine that the best gov't computers today can easily Spot the Loony, or tank, or asian woman or nearly anything else desired.
Actually, what they haven't shown (that I saw) that would be most cool for this is to have a photo of a scene and a painting of that scene and have the filter learn that! Otherwise, what they have done that's still pretty neat is built a system that figures out what mathematically needs to be done to one set of numbers (i.e. the RGB bytes for each pixel) to get it to a second state. Not bad.
I do not have a signature
to see Dali through cubist eyes....
Don't pick up the pho*(@)$*@&@!@ NO CARRIER
That is, why a "learning filter?"
It would be more useful if it could discover a technique by looking at one image.
Looks like these guys have much to improve. Yet it is an effort in the right direction and we can congratulate them, for at least their program can copy lesser styles, if not the masters.
And above all no matter how a good a copy it is, the copy can never measure up to the original.
There's always sufficient, but not always at the right place nor for the right folks.
If someone can use this to encode DeCSS, so it cans till be decoded, maybe THAT will convince people that its an artistic piece.
Seriously, though, this is an amazing project. I'm impressed.
____
Blood, guts, guns, cuts;
Knives, lives, wives, nuns, sluts.
___
The way to see by faith is to shut the eye of reason. --Ben Franklin
you know it's only a matter of time till someone Van Gogh's the goatsecx man.
ugh.
This learning algorithm looks similar to a neural net. My CS professor told us a story about the US government making a neural net that would be used to recognize tanks in satellite photographs.
They spent many hours loading photos for training data. Some photos had tanks and others did not. Once they thought it was working, they tried it on some non-training data --it didn't work.
Instead of recognizing tanks, it learned to distinguish cloudy days from ones with sunshine. All of the pictures with tanks were taken on a cloudy day & all other pictures were taken on days that had sunshine.
He didn't know if they ever actually got it working or not...
I think the important point of this article is not that they can make the painting look sort of like a Van Gogh painting.
I think the new important discovery is that they can take an image and process it so it looks sort of like a Van Gogh, or a painting I did, or a painting your pet chimp did. Or maybe not even necesarily a painting at all.
Instead of saying "here's a photo, make it look like a painting" they are saying "here's one image, now make this other image look sort of like the first one."
-J5K
The libertarian solution to the failures of capitalism is to apply more capitalism til the failures are fixed.
Terrific...a computer that behaves like Van Gogh.
Next thing you know I'll come home from work to find that my PC has severed its own mouse cord in a fit of psychosis.
How useful this particular application is remains to be seen; most people probably have a harder time giving an example of a van Gogh filter for the system to learn from than to use a canned filter.
You're correct that it's impossible to increase *actual* resolution, but in the cases of art and music it's often quite desirable to *simulate* increased resolution. Much is done with lo-fi music to enhance the perceived high frequencies to make it more pleasing to the ear, though this doesn't restore actual high frequency information lost from the original performance. I would assume similar principles are applied to visual art in this case.
-- -Keith