Slashdot Mirror
Searching by Image Instead of Keywords
Content based image retrieval (CBIR), the technique to search for images not by keywords, but by comparing features of the images themselves has been the focus of much research ever since the web emerged. Consider for instance adding CBIR to Google Images, where you would be able to search for images similar to a query image instead of using keywords. A research project at Penn State University has recently been applied to the biggest aviation photo database in the world with close to 800,000 images. You can search for images similar to a photo already in their database (click "View similar photos") or submit your own query image. Some queries generate better results than others but CBIR is certainly here to stay and will be standard in many image applications of the future.
I can't wait to put a nipple into it!
(\_/)
(O.o) This is Bunny. (> <)
and set for goatse!
--
"Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
What an awful beach.
After all, I am strangely colored.
I was just thinking about this the other day. I think content-based image search is one of the Next Big Things. Cameras are so ubquitous now (for better or worse), but having to rely on metadata to give meaning to images requires lots of effort up front.
It will be interesting if we ever get to a stage where we can just search for a random object (or person) in a database of photos. Then you could take pictures of everything with an always-on camera and if you need to find where you put your car keys, just do a search.
This is just asking for trouble. As most of you would probably imagine, some self-proclaim "comdeian" would post either porn pictures, or pictures that resembles porn body position.
;)
.
They would need a team of outsource Indian workers to go through each picture one by one!
I am not Indian but...can I apply for the image filtering job?
I said this first, I should get the job
Because it still has problems - you'll note that the pictures seem to be compared simply based on color similarity. That's the same thing imgSeek does (a great program for sorting and searching your photos) on photo searches. It works wonderfully if you're searching a very limited picture subset (say, airplanes), but if you search a wide variety of pictures, the results can be quite amusing.
It's a Cyrillic alphabet. It's like all those keys you never push on a calculator.
Something with two circles and dots in the middle of each circle.
(\_/)
(O.o) This is Bunny. Add Bunny to your signature
(> <) to help him achieve world domination.
Hmmm...
... Profit! (Oops, or something, grabbing for a Kleenex)
+"34b"
-"puffy"
Maybe trainspotting has died down because all you get on Google now are results for that wretched movie.
This space intentionally left blank.
Some Applications of Our Research
..:)
... now you should be able get similar pr0n (^H^H^H^H I mean art) with these algorithms
1. Airliners.net
A site with almost 1,000,000 aviation images.
Wow !!! I tested their Sample search and all the results were aeroplane photos !!! Ok, ok the site only has airplanes but still
On a more serious note the alogorithms seem to look for similatity in the colors and lighting rather than the subjects (for example it shows the interior of a cabin in photos similar to a whole plane in the sky. To really see its effectiveness we need to test in in the real world (google images) . The 'artisticly revealing' photo you always liked
Those must be old photos. There is no way that beach would be open to the public in the post 9/11 world.
Isn't this kind of how the human brain works to identify objects your eye hasn't seen before?
IANABP, I am not a bio-physicist but it seems very much like artificial intelligence to me.
... the search engine will support ASCII art image searches.
Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
Now I've never seen trainspotting, but I find that this happenns with a lot of movies/topics. Trying to filter through reviews, dvd sales, video games, etc. when trying to find info on the subject. (ie: I noticed this first when "The Mothman Prophecies" first came out and I wanted to know more about these supposed mothmen.
If I can't smoke and swear I'm fucked.
There's a bunch of interesting papers out there on content-based image analysis and retrieval. Below is a sampling from my bibtex file. Does anyone else have others they'd like to share?
* Finding Naked People (Fleck et al, 1996)
* Video google: A text retrieval approach to object matching in videos (Sivic & Zisserman, 2003): web page demo here
* Names and Faces in the News (Berg et al, 2004)
* FACERET: An Interactive Face Retrieval System Based on Self-Organizing Maps (Ruiz-del-Solar et al, 2002)
* Costume: A New Feature for Automatic Video Content Indexing (Jaffre 2005)
I forgot one more, where specific faces were automatically retrieved from feature-length movies and Fawlty Towers:
Automatic Face Recognition for Film Character Retrieval in Feature-Length Films (Arandjelovic & Zisserman, 2005)
The objective of this work is to recognize all the frontal faces of a character in the closed world of a movie or situation comedy, given a small number of query faces. This is challenging because faces in a feature-length film are relatively uncontrolled with a wide variability of scale, pose, illumination, and expressions, and also may be partially occluded. We develop a recognition method based on a cascade of processing steps that normalize for the effects of the changing imaging environment. In particular there are three areas of novelty: (i) we suppress the background surrounding the face, enabling the maximum area of the face to be retained for recognition rather than a subset; (ii) we include a pose refinement step to optimize the registration between the test image and face exemplar; and (iii) we use robust distance to a sub-space to allow for partial occlusion and expression change. The method is applied and evaluated on several feature length films. It is demonstrated that high recall rates (over 92%) can be achieved whilst maintaining good precision (over 93%).
I used to do research in CBIR, and in my image library I had 5 photos of a christmas tree. By any efficient metric, one of them was always way far away from the others.
Xcott
Talking about greatness to society and a little bit of skin. At university one of my projects was a system that used CBIR to try and diagnose skin cancer. The doctor would take an image of the suspect area it then would be compared against a database of cancers. It would then return a suggested likelyhood of being cancer. It also allowed the doctor to build a history of images allowing easy comparision over time.
I always felt good about working on projects like this, gives a warm fuzzy feeling.
One has to guess the search word which generated a given set of 20 images in google's image search
When things are moving forward, we have soomthing to talk about "those good ole days" but frankly the game is interesting initially but later gets boring due to the frequent repetitions..
What I got was an awful lot of red planes - some of which were actually Qantas planes, but I think more by coincidence (i.e., they're red) than design. Many images had nothing to do with Qantas, or even a red plane - they simply had a lot of red in the image.
This is impressive in some ways, but in others it seems like it's simply looking for similar patches of colour. I haven't done enough testing to see what happens if,say, I gave it a half red half green image.
Interesting, but not ready for public consumption just yet. A bit like A.L.I.C.E. the artifial intelligence system actually - neat, but not practical. Yet!
Physicist, consultant, science communicator
Now I can find all the other naked pictures of Bea Arthur on the web!
I guess eVision were just too early to market with their visual search engine. Here's a demo or three of eVe in action.
It sure was cool, just too far ahead of its time...
Programs like GQview (unix/linux) offer functions to search for similar images, mainly used to find duplicates.
It's not quite "put in an image and find me all the similar ones" but the underlying technology is the same, usually creating some kind of "signature" of each image and then comparing the signatures to find others visually similar.
Great for de-duplicating your por^M^M^MPhoto Collection.
I'm a perfectionist but I'm trying to cut back.
The big problem to me is specifying input. I know the "look" of the Mona Lisa's smile, but even with the best pen input methods I'd never be able to mimic DaVinci's subtle emotion of the smile; my hands just aren't capable of doing so. Using photos of the painting could simplify this, but this almost assumes that I'm only looking for the parody's and commercial exploiters of the image rather than the image itself (after all, I have the image to start with). And it raises the further issue that many photographic reproductions of the Mona Lisa that I can get my hands on are still under copyright and I'd be doing something legally questionable with an image long in the public domain.
Add to this the "infinite number of monkeys" issue where legally litigious companies will use technologies like this to scan the internet for litigation targets. Imagine Disney using a cell of Rafiki from the Lion King to find legally similar images that were created after the Lion King was released even if they were only superficially similar. Now do this for all movies back to Snow White or Steamboat WIlly and you could get to be a real visual mob boss with ownership (or at least threat of litigation) over huge libraries of works that weren't even created to intentionally violate Disney "Intellectual Property".
My need for this technology is small considering the input problems I'd have with my artistic abilities, while the litigation nightmare from large databases of "similar" visual data would seem to be more bothersome than helpful. I rather hope these visual search and categorizing methods don't catch on.
However, he put forth the concept of replacing the bit as the common unit of data with actual images - best described as holographic images of light manipulated by light. A picture really _would_ be worth a thousand words in such a system!
don't mod me as redundant you niggers
Yeah, in my early undergrad days I took a 200 level course for a gen ed requirement of discrete mathmatics. Wang was the professor, and too this day I haven't had a course that was as difficult or completely freaking insane as the one he gave. Glad he is doing more research and less teaching.
Pattern Rcognition is a novel by William Gibson, basically set in the present day or very near future. Image based search plays a central role in the plot. It's a very good read.
Did you mount a military-grade, variable-focus MASER on an unlicensed artificial intelligence?
I was looking at a picture of a plane on that web site and there was a link that said "Click for similar images". And what do you know? It brought up more pictures of planes. This is amazing stuff. How did it understand that I was looking at a picture of a plane?
Doesn't it make you feel good to know that our freedoms are protected by politicans, lawyers and journalists.
Whatever algorithm they're using, it seems to be sensitve to the horizon line, colour, shading, orientation of the aircraft, etc. It seems to be operating at the level of a pigeon (who have been shown to discriminate photos depicting trees, water, and particular people - as well as art by Picasso and Monet. See http://www.pigeon.psy.tufts.edu/avc/huber for other examples. It will be some time before algorithms can match on the basis of model numbers and such. It took humans quite a while to evolve a cortex to enable such fine discriminations.
We always tell people not to mod down improper observations, so I'm going to try to practice what I preach so to speak.
On what grounds could the TSA squelch photographers and their right to share their creative works (which is their livelihood)?
Oh My ${deity}, the noise must be horrific! My ears are hurting just from *looking* at that picture.
Actually, engines aren't at full throttle during landing, so it's actually not that loud at all
The GIFT (the GNU Image-Finding Tool) is a Content Based Image Retrieval System (CBIRS). It enables you to do Query By Example on images, giving you the opportunity to improve query results by relevance feedback. For processing your queries the program relies entirely on the content of the images, freeing you from the need to annotate all images before querying the collection.
GIFT It worked pretty well for me in the demos they linked too. I have been waiting for this type of application to gain momentum.
Wax on, wax off baby!
Got this error when testing out the system, they certainly need to fix this, especially if they want it to become popular.
'Coz I'm looking for more information on this image.
It says "multi lock on" and a date, but all Google reports is other forum posts looking for the creator of the image. Apparently, there's a high-res version of it too.
Actually, it will be hillarious what will happen when grandma puts in a picture of her grandson taking a drink from the hose in the backyard.
Its almost like telling someone to go to whitehouse.com
iView Media pro already does this. You just tell it to find dupes and set the tolerence to loose. http://www.iview-multimedia.com/
OMG Ponies!!! with Glitter!!!! I miss Pink
why bother making an algorithm that can recognise which images are porn and which are not when you can just set up a web site where people will do it for free? It reminds me of those "enter the characters in this image" tests that places like Yahoo do to ensure you can't sign up for a million email accounts a day. They're so easy to get around cause all you have to do is present the image to a man who wants porn and he'll happily provide his character recognition skills without charge.
How we know is more important than what we know.
I guess anyone can get a research grant these days.
Google actually did take this technology and try it. The first version of their image search had a "find similar" link next to every image. These tended to work okay at first (they weren't great, but you usually got enough photos back that you could visually scan them and find something of interest that was related to the original image). After a few months, for some reason, the "find similar" links started returning increasingly nonsensical results. After it degenerated to the point of near uselessness, they took the "find similar" link away from the image search results. I expected it to turn up again once they got the kinks worked out, but apparently they just decided to stop working on it.
Now, do this for something like Google Images or PBase or collections spanning infinite numbers of subjects and image sizes, then I'll get excited.
No, I've never had a rejection from A.net, I've never submitted there. Two minutes in their forum will tell you how anal their 'screeners' are, for whatever reason. It's just freakin' pictures of airplanes, for chrissakes.
They are doing it based upon the shades of color in the image. So if you query for a image of an aircraft in flight with a lot of white clouds behind it, you get more of the same, but you also get aircraft parked on snow-covered ground.
So when can we search for music? I'm trying to find this song that goes like "dah daahh, dumpiti, dum dum"? Any answers?
Imagine if this was incorporated into Spotlight, not having to rename your entire photo library.
I've tried two different images of airplanes; one of a bright red flying car on bright green grass and one of SpaceShip One against a deep blue sky. Both times, the results looked surprisingly like my query images in color composition only. Red planes on grass and white planes against a blue sky. Inauspicious start.
Next experiment: I took a picture of a highly distinctive plane, a harrier, climbing at a steep angle and viewed in profile. I got, in return, a list of passenger jets, and even a helicopter. Hardly surprisingly, all of the result pictures had the same bluish white sky as my original image. That was literally the only similarity.
According to the introduction on the search page the heuristics used compares colors, contrast and shapes in the images themselves. I saw no correlation whatsoever between shapes, and any correlation in contrast seems to be to be the result of the search engine simply looking for images that contain the same colors in a similar ratio to the original. In short, nothing to see here, move along.
On the other hand, one of the projects listed under the Penn State University link looks fairly fascinating. The Riemann a-LIP project (automatic linguistic indexing of pictures) doesn't allow user input of images, unfortunately, but it does show some fairly fascinating attempts at verbally qualifying image data. For example, it describes a blue and orange mandelbrot as pattern agate shimer abstract scene, and a sunset over a lake as Berlin Devon Namibia landscape lake scene. Okay, it may still need some work, but it sure beats the hell out of the "find the same color airplane engine".
I want the fire back.
http://www.spilth.org/pictures/girls/ceren/jkh-and -babe.jpg
Impossible!
Oh, you mean like imgseek?
Trainspotting seems to still be around as well, see http://www.railpictures.net/.
End of Line.
IBM was pushing this in 1996/1997 with DB2 v...uh...4? Didn't work all that well then...and it was just basic shape outlines.
Pie. Sky. In.
Blar.
some self-proclaim "comdeian" would post either porn pictures
Porn DOES however, make you spell better!
If image search is going to be the next big thing bots are going to need to crawl deeper and more often... Most engines probably don't have the resources currently to support the extra space/load.
All the torrents you could want.
...why does it show me pictures of donkeys when I use my school portrait for the search?
Non, je ne veux pas coucher avec toi ce soir.
see http://www.oracle.com/technology/products/intermed ia/index.html (oracle)
This document was written in 2002 (and that version is as old too)
I can remember some sales guy saying "You can look for a couch that's like this one. But blue."
So that means it should do a bit more than just color patches..
and then there is Google images..
nice thong and an A-340 -- Same beach.
747 landing
This famous pic is the GIS for 747 and beach.
Poster is very right about the small subset. We might be able to see very similar plane pictures, because the subset is so small, and even in this small subset, we cannot find, say similar pictures of helicopters. The final searching algorithm should be a combination of CBIRS and metadata (to subset the images).
This Photo ID has not been indexed in the similarity database yet.
It just keeps coming up with photos of airplanes.
^^vv<><>BA
Purdue also has a 3D shape search. More can be found at Here .
I'm pretty sure altavista had this feature several years ago, but removed it. I remember that it worked fairly well. Does anyone know what happened to it?
I'm sorry. The number you have reached is imaginary. Please rotate your phone 90 degrees and try again.
Or maybe trainspotting died when they realized it was more fun to shoot up than to NOTE THE FUCKING ARRIVAL TIMES OF TRAINS!
The road to hell is paved with good intentions.
I took Professor Wang's networking class this Semester and have him for databases in the fall. He is a very interesting professor and was in the same class as the people who invented google.
quite right. Color-based retrievals are much easier and cheaper to implement, you know.
I did my BS thesis in CS/CE on CBIR back in '99 and decided to use interest points instead of color statistics in order to explore the options.
While color-based CBIR gives very "good" results as far as color match is concerned and human context is therefore many times hit on the spot and sometimes missed altogether, a simple interest points calculation (filtering the image with a double Gaussian etc.) yields "interesting" results, sometimes returning nice hits and sometimes grouping images with shapes that no human soul could count for similar.
Note that I used and tweaked a method that successfully used when checking for identical images (against forgeries, for example) as a foundation, deliberately, to explore its usefulness in searching for similar images.
Searching for similar images or checking if identical are actually two quite different things.
I imagine you would be just as disappointed using my thesis C program, as the field of computer vision, as we called it, was relatively fresh on the subject at the time. I admit I haven't done any further research since, having to make a living, so I'm not sure what's up and cookin' these days...
Naturally, there are entirely different methods that could serve CBIR purposes (machine learning, neural networks, metadata, text-tags-assisted stuff), but that be a different focus, not the computer vision approach.
Of course, given the usual course of things, it will instead be deployed at JFK's formerly-TWA terminal, assigned facial recognition tasks, and immediately declare everyone to be among the 10-most-wanted terrorists. I can't wait.
Village idiot in some extremely smart villages.
This is great and all, but I would love for a way to upload a graphic file to a search engine and have it report back any identical (or nearly identical.. perhaps a "threshold" setting) copies on the web.
This would be useful to me as a photographer to see if anyone out there is using my photos.
-David
> This is just asking for trouble. As most of you would probably imagine, some self-proclaim "comdeian" would post either porn pictures, or pictures that resembles porn body position.
Everyone around here is a "comdeian". I wish everyone on this site would grow up and take things seriously for once.
python -c 'print "butter me".join(["\0"*n for n in range(100,1,-1)*2+range(1,100)])' >
An open-source (GPL v 2) Content Based Image Retrieval program already exists: imgSeek .
To search a photo, you don't need a similar photo, simply draw a rough sketch. See this screenshot.
Because what you can do is draw a horrible representation of the photo you want, say for arguments sake in Paint, which is very vauge. You want a photo that is like this.
I tried it with a horribly drawn plane flying into the sunset (A vauge shilouette of a plane, and orange background with yellow circle on it). Admittadly not all of the images were right, but a lot of them were exactly what I wanted to see.
Give it a shot, I think you'll be surprised.
I could be wrong, but I got the impression that Imgseek uses the position as well. I have tried running it on a collection of photos from a concert, and it is very good at returning those with the stage in the centre. Then again, the object in the centre can effect the colour of everything on most cameras.
# cat
Damn, my RAM is full of llamas.
About a year or so ago, I and three other Masters students worked on a similar project at the University of Southampton.
I've not RTFA (not had the time), but our approach was to split the images into segments (based on colour and texture) which were assumed to be objects. The segments would then be analyzed for various feature vectors, such as shape, texture, colour etc. These vectors would then be added into a database of numbers, and finally the segments grouped, giving a collection of classified sections which (hopefully) represent similar objects.
From related metadata such as keywords, you could then hope to build up an idea of what keyword matches which section. You could also come up with a relevance between two images, and thus search for similar images.
We didn't have enough time to make it bulletproof by any means, but our limited results were very promising.
Sorry I can't find the paper, but we've got some screenshots of the application here and here (you can see false colouring applied to the original image to display the segments)
We Build Beautiful Websites
The second photo returned from this Imgseek search uses significantly different colours, but similar layout.
# cat
Damn, my RAM is full of llamas.
Querying for similar images on a collection with 5215 unrelated images.
Search for yellow flowers gets pink flowers. The layout must be being analysed, surely?
# cat
Damn, my RAM is full of llamas.
I've just found some more here and here
We Build Beautiful Websites
Image search will kinda work for airplanes in this database,as there are a very limited set of airplane model numbers, which are going to be attached to each photo.
But if the database didnt have these text clues the image search is going to be unlikely to see the similarity between an 747 in the air, as seen from the ground, with a head-on view of a 747, or one at the gate, or one in a hangar, or one in twilight, or one of a different color.
Maybe it could be done by outsourcing the task to India?
I was trying this out a bit, and have to admit that it's cool that something like this exists at all.
However, I think it would be better if it were able to realize what the 'background' was and filter it out. (Though I couldn't begin to guess how you'd do this.)
For example, I searched for this image. Many of the results are of something completely different, such as a white jet. Which is nothing like a camo helicopter. But the sky and the ground are pretty similar, and I think that's how it's matching.
It's incredible that we got this far, but I think there's still a long ways to go before it's at the stage where you put in an image, and are awed at how quickly it works.
Also note that I'd tend to think this would exponentially more difficult than searching HTML files, so it might be much more expensive to implement large-scale.
________________________________________________
suwain_2
The first thing I though of when seeing this was the next wave of law-suits.
Search for images you created, and you will find all kinds of similar/duplicate images.
Lots of web-sites uses images gotten from other sites, may modify them slightly, however much of the images is based upon images found on the net.
A program like this is likely to be popular among lawyers and design power houses.
Actually, this story (the veracity of I do not know) predates our modern concepts of "neural networks" - that is, multi-layer networks of nodes (typically three - input, output, and intermediary layers), in which the nodes simulate neurons via weighted thresholds and other mechanisms for "firing" an output based on inputs aggregated over time and/or frequency - coupled with back-propagation "learning"...
Instead, the story seems to have popped up soon after the introduction of, in the late 1950's to early 1960's - of the ideas behind the perceptron, the direct precursor to modern neural networks. Historical perceptrons could be conceptually visualized as simple, single layer neural networks.
So, your original statement applies, but I wanted to clarify where and how this story seems to have originated (computer science, on the whole, seems to be one area of research where almost nobody knows, remembers, nor seems to care - about its history, thus we seem to be forever reinventing the wheel in many areas). You find it brought up as an apocryphal story everywhere in liturature about neural networks, seemingly no matter how far back you go, until you get into the Minsky era (late 1950's - early 1960's) of such machines.
Reason is the Path to God - Anon
In general image understanding is equivalent to general AI. We won't get a CBIR system that works well before we get an AI that works and vice versa, because people expect to be able to match the *content* of the image they submit as template and not the general appearance of the image. The problem is then too unspecified.
Even in the restricted context of aeroplanes this is not a trivial problem. Someone in the list of replies submitted an image of a warthog (A-10) and got nonsensical results. Somehow the CBIR system would need to be able to infer a model of the A-10 from a given random 2D projection, and match it to the other 2D projections of the same A-10 model that they do have in the DB. This doesn't sound impossible but it is hard, and I suspect the Penn State people didn't do that. Instead they are probably matching on colour, texture, general appearance, etc.
This is not to say that CBIR is not a nice problem to apply new image processing/image analysis algorithms to, which are developed all the time.
When this becomes a little more refined, it will be nice to apply to frames of video to assist in creating video loops. For example, you mount a camera near the coast, and get 3 hours of footage of the waves crashing against the rocks. You want to make a nice loop of that, which can play indefinitely, but you want to avoid that 'jump' that always seems to occur. Find small sections of the movie where say...10 frames in a row are similar to 10 other frames... It'll take out a lot of the usual pain.
Surviving America