Slashdot Mirror
Algorithm Seamlessly Patches Holes In Images
Beetle B. writes in with research from Carnegie Mellon demonstrating a new way to replace arbitrarily shaped blank areas in an image with portions of images from a huge catalog in a totally seamless manner. From the abstract: "In this paper we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database that are not only seamless but also semantically valid. Our chief insight is that while the space of images is effectively infinite, the space of semantically differentiable scenes is actually not that large. For many image completion tasks we are able to find similar scenes which contain image fragments that will convincingly complete the image. Our algorithm is entirely data-driven, requiring no annotations or labelling by the user."
Uncensored Japanese pornography!
This message printed on 100% post-consumer recycled electrons.
Broken or flaky video files. Nothing is more irritating than an mpeg, etc error that causes an entire block to go black and smear itself all over the place until the next keyframe. I don't expect realtime correction, but it would be nice if I could patch the file rather than do another six hour encode.
May the Maths Be with you!
It was as if a million fake celebrity pr0n websites cried and were suddenly silenced...
My blog
They use pictures gathered on the web to fill their database. Just 'repair' your favourite shots from your holidays at the beach if you want porn.
Slashdotted already.
BBC News coverage of the story is here: http://news.bbc.co.uk/1/hi/technology/6936444.stm
you can deduce what was *actually* in the missing part of the image - rather than guessing it - by tracing back photons from teh camera lense back to their source. I seem to recall *that* technique being used in an episode of ST:TNG, probably dreamt up by Wesly Crusher - I nearly fell of my chair laughing.
There's also that questionable bit of image analysis in Blade Runner where Deckard zooms in and shifts left/right on an image of a mirror and the image responds like it is an actual real mirror. oh dear.
...call me when they make this into a plugin for Photoshop.
And if you dont have any pictures database, there's always GREYCstoration:t ion/index.htmlt ion/demonstration.html
http://www.greyc.ensicaen.fr/~dtschump/greycstora
It's pretty impressive:
http://www.greyc.ensicaen.fr/~dtschump/greycstora
and works with the gimp.
I want a picture of me with no face, and see what face it gives me! (c'mon clint eastwood)
Ze Atomic Device! It iz Ztolen!
My first thought on this was how easy it will be change just enough of a picture before releasing it to make it incredibly tough to find out the way you're representing the picture isn't the way it really was.
"It is a miracle that curiosity survives formal education." -Albert Einstein
I take a picture of a hole?
I prefer Flambe as apposed flamebait.
You never know what that "kinda-like" picture used to patch contains. You might get the opposite of what you want.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
If any hole in the image can be filled with a part of another pic, can't you compress an image by replacing one piece at a time with a reference to a patch? Also, how about replacing with patches of higher resolution than the original? I realize it would all be technically lossy as hell, but the compression artifacts should not be very noticable to the human eye, right? Additionally, how about using this for movie compression? Filling in based on info from previous and next frame.
I may have to actually RTFA this time.
CSI Miami and NY have had infinite zoom capability with photos for years, and you excited about this? Bah.
If brevity is the soul of wit, then how does one explain Twitter?
Our chief insight is that while the space of images is effectively infinite, the space of semantically differentiable scenes is actually not that large. For many image completion tasks we are able to find similar scenes which contain image fragments that will convincingly complete the image. Our algorithm is entirely data-driven, requiring no annotations or labelling by the user.
What are the "semantics" here?
Is this like google images, where the nearby HTML text determines the classification of the image [i.e ASCII-text as meta-data for images]?
Or is this a great big neural net of wavelet data which classifies the images mathematically?
PS: I have the same question about that infamous Photosynth/Sea Dragon demonstration:
http://www.ted.com/index.php/talks/view/id/129
It takes an existing image and finds a very similar image in a huge catalog, then adds in a similarly-shaped piece to the existing image where applicable. So it's more like a puzzle solver than an image completion engine. If you don't have a huge, huge catalog of images, it won't really work for any given image as well as their samples.
stuff |
Due to recent advances at Carengie Mellon, you have all been made redundant by a computer algorithm. Sorry, progress is a biatch.
Yours,
some code and a database
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
This is really cool.
Much of our perception is affected by out experience
As someone who has been slowly losing their hearing, I can attest that the mind REALLY DOES "fill in the blanks".
This concept can probably be extended to other sensory perception (other than hearing/sight).
Looks like the start of an intelligent sensing device. -- A baby step toward Asimov's dream.
Actually, laser printers and supermarket scanners often use holograms of optics rather than actual glass optics (mirrors and lenses). The holograms are usually CGI - making equivalent real glass optics would be fabulously expensive and heavy, or even physically impossible.
.. a picture of yourself, with your face blanked out... whose face would you get?
simon
Instead of "You appear to be writing a letter. Should I format it for you?" I guess we'll get "You appear to be viewing Japanese pornography. Should I de-pixelate it for you?"
SJW: Someone who has run out of real oppression, and has to fake it.
"this section has been intentionally left blank"
If at first you don't succeed, call it version 1.0.
Although the summary says the method will fill arbitrary holes, at the link that claim is not made, and in their examples they delete specific picture elements.
Reminds me of the recently announced Photosynth from Microsoft that seems to do something similar, but focuses on stitching images together rather than replacing parts of existing ones.
...I may have to actually RTFA this time. Don't you do it! If you do, you can hand in yourNext thing you know, you'll be reading the flippin' article and posting insightful things that the rest of us, who spend our 9-5 journey together every day, will have no way to counter unless we start to read the freakin' articles! This will have an impact beyond what you realize. For the good of the greater, don't RTFA!
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
Would this be semantically valid patch for your holiday shots? http://www1.sch.im/wlp/large%20images/beaver%20nat %20geo.jpg
I wonder if this is part of the beginning of a new, computationally-driven problem-solving paradigm. As more and more data is stored, and if search algorithms become more and more clever, the cost of "looking up" (computationally speaking) the answer to a problem might be lower than the cost of "remembering" (using local storage) or "figuring out" (using local CPU power) the answer.
This is already happening informally in the personal sphere, because of things like Google, recently amplified by the iPhone and its inevitable successors in the ubiquitous rapid-access web-tool field. As they say, these days, if you have a web browser, you hardly have to wonder about anything anymore.
Of course, problem solving by search isn't exactly a new paradigm, but it could be a newly-cheap paradigm.
2*3*3*3*3*11*251
This sounds wonderful. Now if I can just output those images in my head from the "dreams that I want to come true" subdirectory, and turn this nifty program loose to fill in the holes. I should be rich in no time.
Oh wait, that particular technique is probably 1) patented, 2)copyrighted, and 3) a DMCA protected, non-reverse engineerable trade secret to boot. (yes I know that #1 and #2 are incompatible with #3. Get a life, people.
*sigh*. Guess I'll have to do those dreams myself the long way.
...Open Source isn't the only answer -- but it's almost always a better value than the alternatives...
We need a new way to replace slashdotted servers with portions of articles from a huge catalog in a totally seamless manner.
-- Boycott Shell
Cut! That's a wrap, people. Great scene. I loved the way you handed him the hush money, like it was dirty.
Now, remove the actors' faces, and use that new algorithm to seamlessly insert every known political figure for the stills.
Welcome to spam-blackmail baby.
So easy to do, so why doesn't /. provide these links as a matter of course?e -completion/
http://graphics.cs.cmu.edu.nyud.net/projects/scen
-- Boycott Shell
In the paper (yes, I did read the paper...... well most of it.) They point out that their algo doesn't work well with complex semantics, if you have a mostly incomplete object in the picture, it won't be able to make a convincing fix for the image. It also has problems with the subject of the scene, one example was a brick building where the algo had filled the hole with rocks of a similar color. The algo does do a very good job otherwise, many of the pictures were very convincing and could probably replace alot of the photoshopers in the tubes. They do note that their algo isn't ment to figure out whats missing, just fill it with something that fits with the rest of the scene.
It's an automatic photoshoping algo.
Group gives paper at conference, then endlessly spams media with paper.
Now you've done it...
Cut out a car, and the algorithm will replace the blank spot with a car. Not necessarily the same car, but a car of about the same size, etc.
So for location shooting, you no longer need to go through extensive "scene sanitation" ahead of the shoot. Take select frames of the footage, point out what you want to snip, and let software propagate the snips through the rest of the footage. Then let this new stuff fill those holes with "approved product placement" items that fit the scene. Yank the Chevys, put in Fords. Yank the Pepsi, put in Coke, etc.
The living have better things to do than to continue hating the dead.
you get if you unwisely follow the instructions on the other current front-page article about building your own high powered laser? That's uncanny.
What sound do people on rollercoasters make? Hint: it's not Xbox 360.
Applications like this make me in favor of DRM if it can be used to protect the rights of artists to retain the creative content of their work. I'm getting more than sick of people that seem to think its ok to just go and copy an image off the web like it is nothing. If these images were so easy to make, then, why not make them yourselves! It takes work to make an image, whether it is a photograph, painting, or some other likeness thereof. This stuff on the internet, is not just yours to take.
This is my sig.
I'm not an anthropologist, biologist, neurologist, or anything-ologist, for that matter, but isn't this very similar to how the human brain works? Why do we recognize images as being "Japanese porn" or not? Surely that's because we've seen that sort of thing before (well, most of us have), and we've stored these images inside of our memory. We can recognize them "semantically" because these images exist in our database, and our brain provides some sort of algorithm for detecting similarities in pictures.
If I am not mistaken, that is what this project is doing. Could they have stumbled upon a possible solution to a difficult AI proble m without even realizing it?
This is very cool, and I wonder how similar it is to what the brain does with respect to blind spots?
For those who don't know: each eye has a surprisingly large blind spot at the place where the optic nerve enters the eye. At reading distance, in the right eye, it's about four or five inches to the right of the spot at which you are gaving, and many textbooks and "fun with optical illusions"-type books will have a diagram... like the one on this web page... and directions for finding it. The blind spot is much larger than the dot on that web page, incidentally. If you explore, you'll find that... at the distance at which the dot disappears... the blind spot is nearly an inch wide and an inch-and-a-half high.
Even allowing for the fact that each eye has the blind spot in a different place so they fill in for each other, once you discover how big the blind spot is... and how relatively close to your position of gaze it is... you'll be astonished that almost nobody notices it until it is pointed out.
The brain does something more or less like filling in the blind spot. I say "more or less like" because it is very hard to answer the question "what do you see in the blind spot." For example, if you hold a computer keyboard at the right distance so that you're looking at the "G" key and the "K" key is in your blind spot, what do you see? Certainly not a black spot, certainly not a white spot, certainly not a "hole" or emptiness. Probably you have an impression of computer keys. Do you see a letter K? Certainly not, yet somehow you don't see a blank key, either.
Incidentally, I used to suffer from migraine headaches, and one of the symptoms for some people is the formation of blind spots which can be even larger than the "normal" blind spot, and can appear in central vision. One one memorable occasion, I was looking at the cover of a hardbound book, and I can tell you that when I looked at the title, my perception was the stamped, printed title disappeared, yet I would have sworn in a court of law that I still saw the cloth texture extending across the blind spot.
Although he does not specifically refer to it as a migraine illusion, I believe Lewis Carroll was known to be a migraineur, and in Chapter V of Through the Looking-Glass, "Wool and Water," Alice notices that "The shop seemed to be full of all manner of curious things -- but the oddest part of it all was that, whenever she looked hard at any shelf, to make out exactly what it had on it, that particular shelf was always quite, empty, though the others round it were crowded as full as they could hold." Any migraineur who experiences central blind spots will recognize this description.
Hays and Efros' system--relatively-simple algorithm operating on a large database of previously-seen images--seems to me to be the sorta-kinda way in which one could imagine the brain working.
I wonder if there's any way to test this?
"How to Do Nothing," kids activities, back in print!
and posted it to comp.theory: thread link
However, where did they get the database of images from? If they pulled the content from the web, certainly there are copyright issues (at least if they commercialize such an engine including that database). It might be hard to tell where the source came from, but if they're profiting off the use of my imagery, I'd expect a cut.
It would not seem to me to fall under fair use...
MadCow.
I used to have a sig, but I set it free and it never came back.
Browse around the site. Not only does it fill in uniform background color (easy), but also more complicated pattern (lines going through the blind spot), and even autocompletes repeating pattern (field of red circles).
"You appear to be viewing Japanese pornography. Should I de-pixelate it for you?"
And whoever thought we'd be glad to have clippy back?
I stole this sig from someone cleverer than me.
I'm sure that I can creatively construct a few holes -- and fill them in with appropriately matching chunks from out there on the web. And tell a much different story about her than reality at it's worst would not even begin to approximate in terms of horrible implications. [not that the reasons for which I divorced her weren't bad enough mind you, but I could construct a photo built scenario that is much worse.]
Interesting bit of science.... dangerous applications.
...Open Source isn't the only answer -- but it's almost always a better value than the alternatives...
You'd have to test the images on your little brother to see if they were safe for you to look at:
"Hey Billy, when my back is turned, turn on the monitor and tell me what you see." ....wow, that's a relief!"
If you're female, use your little sister instead...
"Ok" <click> "AEIEIIEEIEIEIIE! I just went blind!
Photoshop is now only to be used for high profile history; lesser history is to be automated.
IGNORANCE IS STRENGTH
Rarely do I make a "that was funny!" post, but... hats off, sir.
it would be really awesome if these guys started working with the gigapxl guys.
Those of us who think they know everything annoy those of us who do.
OK, now we know that there was one photographer who doctored a photo to make it look like there was smoke from shelling when there either wasn't any smoke or less than he had made it look. He did so rather badly and got caught rather quickly.
But as their example shows - they filled in the lagoon with images of boats and water that were not even from the same beach.
Now say someone (oh, I don't know - a newspaper) wants to portray someone in bad light. How hard would it be to use this software to place that individual in a different picture in some compromising situation?
Blackmail anyone? How could you prove that this wasn't really you or that you weren't involved in the situation shown?
It could also make all pictures/video/etc become invalid for use as evidence because of the simplicity and strength of the forgery.
Cool technology - but it could (and will) open a whole can of worms.
Sorry to disappoint you, but IIRC from the SIGGRAPH lecture, I think they removed people from the database.
It did work quite well, however, offering a choice of different image completions.
why not simply patch its semantics by rendering a goatse *around* it?
Wow... the first thing I'm waiting for the algorithm to do is, make a pic. of Jessica Alba's face, with Angelina Jolie lips, Salma Hayek boobs and JLo's butt!!
Hurray!
"just look for previous articles that cover the same technology."
In many cases, it's a dupe of the current article.
-- Boycott Shell
Then they'd create an algo. to remove satt. dishes from images. For now it's just another college experiment " We're so smart, neener neener neener" . Removing satt. dishes though, they'd make a million and would be admired world wide for their intelligence.
...Imagination.
Isn't that what's going on here? When a my brain imagines something that isn't really there, it draws on my memory of things I have seen throughout my life.
If you give a computer a huge memory of images and the ability to apply them towards creating a scene it has never "seen" before, you have given it an imagination.
Does that mean that if I erase part of the grassy knoll it will draw in the second gunman? Or more importantly if I erase the goatse hole will it draw in a hole?
I couldn't fail to disagree with you any less.
Anyone who really has merged two images together knows that seamless fitting is only a part of the problem. How is the perspective matched? The algorithm described here does not understand it, it simply throws a dice and picks randomly. This means we still need human to say: "No, that's crap. Try again."
So it's automated poor photoshopping!
The Abyss stares into YOU!
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
Reuters implements this technology immediately, and some right-wing bloggers wondered where all the fake photographs went.
Twitter supports and protects racists - by smearing their critics with the "Hate Speech" label.
God, RSJ should have patented the word "Seamless"
---
The origin of UFO pictures!
... on the GOPs public image appear promising.
Have gnu, will travel.
I'm not sure what their current process is (I had assumed it was costly and manual), but this might be great for removing dust and and excessive grain from old movies and TV shows. Perhaps that'll actually make it worth it to buy the old stuff on HD/Blueray.
... so I'll be able to see in the blind spot that I created using my new high-power DVD-burner laser pointer flashlight?
http://outcampaign.org/
Does anyone have a program that looks for blurs? Quite a few people are interested in knowing if certain photos have been faked. Is there a way to highlight changes in a faked photograph?
Alright, I do compositing and graphics professionally, and I just don't buy this. Unless this algorithm can automatically generate accurate transparencies, reflections, shadows, and all with the correct color blending, then- at the very least-, many of the "test cases" are clearly fake.
...it's not like this isn't happening already. Any professionally produced advertising will have pictures that have been tidied by photoshop at the ad agency already. Sure, it'd be false advertising to doctor these to remove building sites near holiday complexes, etc, but it's not like it can't and isn't being easily done right now.
Um, bad form to reply to yourself and all that, but wtf? Is it backwards day? How could this possibly be offtopic?
Sony ha
There goes my porn collection dream into pieces - thanks alot for making me sterile!
--- I am known for the ones who want to find me on the net. Is that a privacy risk or a privilege? One might wonder..
I know I'm a little late in the thread, but I just thought about it, couldn't the technology be adapted for, instead of patching holes, patching graphical elements in an image into elements found in the database, thus allowing you to turn drawings and such into photograph-like pictures?
You just got troll'd!