Slashdot Mirror
Searching by Image Instead of Keywords
Content based image retrieval (CBIR), the technique to search for images not by keywords, but by comparing features of the images themselves has been the focus of much research ever since the web emerged. Consider for instance adding CBIR to Google Images, where you would be able to search for images similar to a query image instead of using keywords. A research project at Penn State University has recently been applied to the biggest aviation photo database in the world with close to 800,000 images. You can search for images similar to a photo already in their database (click "View similar photos") or submit your own query image. Some queries generate better results than others but CBIR is certainly here to stay and will be standard in many image applications of the future.
I can't wait to put a nipple into it!
(\_/)
(O.o) This is Bunny. (> <)
I don't understand how this is relevant to slashdot...
If this technology has been in development for a long time, why hasn't google taken the ball and run with it?
and set for goatse!
--
"Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
What an awful beach.
After all, I am strangely colored.
I was just thinking about this the other day. I think content-based image search is one of the Next Big Things. Cameras are so ubquitous now (for better or worse), but having to rely on metadata to give meaning to images requires lots of effort up front.
It will be interesting if we ever get to a stage where we can just search for a random object (or person) in a database of photos. Then you could take pictures of everything with an always-on camera and if you need to find where you put your car keys, just do a search.
How long until this is employed by the FBI to attempt to discover illegal images on Usenet etc?
More importantly, how many agents will jump at the begruding task of sorting through the thousands of false-positives?
Next up : Mission remote control.
This that I have an image to search with...which is why I am searching for images to begin with - to find an image.
"Oh, you hate your job? There's a support group for that, it's called everyone, they meet at the bar."
Green money says if this technology were rolled out in force, pr0n sites would find a way to hijack it.
I took a photo of me giving my webcam The Finger and uploaded it. The first thing that came up was a picture of George W. Bush! I guess it's a match because we are both good at giving the entire world The Finger.
omg a dupe from last year
This is just asking for trouble. As most of you would probably imagine, some self-proclaim "comdeian" would post either porn pictures, or pictures that resembles porn body position.
;)
.
They would need a team of outsource Indian workers to go through each picture one by one!
I am not Indian but...can I apply for the image filtering job?
I said this first, I should get the job
Something with two circles and dots in the middle of each circle.
(\_/)
(O.o) This is Bunny. Add Bunny to your signature
(> <) to help him achieve world domination.
Forget slashdotting Airliners.net, how long before the TSA shuts down that website? The trainspotting hobby has already died off following terrorism fears, I can't help but think that other enthusiast sites like Airliners.net will be next.
before some troll adds goatse or some spammer adding random advertisements.
for the love of god think of the children
perpetually dwelling in the -1 pits
Hmmm...
... Profit! (Oops, or something, grabbing for a Kleenex)
+"34b"
-"puffy"
But I found I just couldn't type images as easily as I could words so I went back to using google.
Some Applications of Our Research
..:)
... now you should be able get similar pr0n (^H^H^H^H I mean art) with these algorithms
1. Airliners.net
A site with almost 1,000,000 aviation images.
Wow !!! I tested their Sample search and all the results were aeroplane photos !!! Ok, ok the site only has airplanes but still
On a more serious note the alogorithms seem to look for similatity in the colors and lighting rather than the subjects (for example it shows the interior of a cabin in photos similar to a whole plane in the sky. To really see its effectiveness we need to test in in the real world (google images) . The 'artisticly revealing' photo you always liked
Those must be old photos. There is no way that beach would be open to the public in the post 9/11 world.
Isn't this kind of how the human brain works to identify objects your eye hasn't seen before?
IANABP, I am not a bio-physicist but it seems very much like artificial intelligence to me.
... the search engine will support ASCII art image searches.
Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
Would it be possible to simply draw a picture and have a search be based upon that?
If you have any skills at drawing, it would be quite a useful tool.
There's a bunch of interesting papers out there on content-based image analysis and retrieval. Below is a sampling from my bibtex file. Does anyone else have others they'd like to share?
* Finding Naked People (Fleck et al, 1996)
* Video google: A text retrieval approach to object matching in videos (Sivic & Zisserman, 2003): web page demo here
* Names and Faces in the News (Berg et al, 2004)
* FACERET: An Interactive Face Retrieval System Based on Self-Organizing Maps (Ruiz-del-Solar et al, 2002)
* Costume: A New Feature for Automatic Video Content Indexing (Jaffre 2005)
http://science.slashdot.org/article.pl?sid=05/01/1 3/184226&tid=152&tid=14
tried a search for hot midget monkey sex... no hits? WTF?
I forgot one more, where specific faces were automatically retrieved from feature-length movies and Fawlty Towers:
Automatic Face Recognition for Film Character Retrieval in Feature-Length Films (Arandjelovic & Zisserman, 2005)
The objective of this work is to recognize all the frontal faces of a character in the closed world of a movie or situation comedy, given a small number of query faces. This is challenging because faces in a feature-length film are relatively uncontrolled with a wide variability of scale, pose, illumination, and expressions, and also may be partially occluded. We develop a recognition method based on a cascade of processing steps that normalize for the effects of the changing imaging environment. In particular there are three areas of novelty: (i) we suppress the background surrounding the face, enabling the maximum area of the face to be retained for recognition rather than a subset; (ii) we include a pose refinement step to optimize the registration between the test image and face exemplar; and (iii) we use robust distance to a sub-space to allow for partial occlusion and expression change. The method is applied and evaluated on several feature length films. It is demonstrated that high recall rates (over 92%) can be achieved whilst maintaining good precision (over 93%).
Talking about greatness to society and a little bit of skin. At university one of my projects was a system that used CBIR to try and diagnose skin cancer. The doctor would take an image of the suspect area it then would be compared against a database of cancers. It would then return a suggested likelyhood of being cancer. It also allowed the doctor to build a history of images allowing easy comparision over time.
I always felt good about working on projects like this, gives a warm fuzzy feeling.
One has to guess the search word which generated a given set of 20 images in google's image search
When things are moving forward, we have soomthing to talk about "those good ole days" but frankly the game is interesting initially but later gets boring due to the frequent repetitions..
They might have to filter out currency searches- one lousy scan of a dollar bill might give results of high quality color images. There have been enough problems with amatuer counterfeiting already without making it any easier. Maybe I shouldn't even be posting this idea on /. Oh well...
What I got was an awful lot of red planes - some of which were actually Qantas planes, but I think more by coincidence (i.e., they're red) than design. Many images had nothing to do with Qantas, or even a red plane - they simply had a lot of red in the image.
This is impressive in some ways, but in others it seems like it's simply looking for similar patches of colour. I haven't done enough testing to see what happens if,say, I gave it a half red half green image.
Interesting, but not ready for public consumption just yet. A bit like A.L.I.C.E. the artifial intelligence system actually - neat, but not practical. Yet!
Physicist, consultant, science communicator
It had to be said
Damn filters. "(.)(.)" doesn't work
Now I can find all the other naked pictures of Bea Arthur on the web!
I guess eVision were just too early to market with their visual search engine. Here's a demo or three of eVe in action.
It sure was cool, just too far ahead of its time...
Programs like GQview (unix/linux) offer functions to search for similar images, mainly used to find duplicates.
It's not quite "put in an image and find me all the similar ones" but the underlying technology is the same, usually creating some kind of "signature" of each image and then comparing the signatures to find others visually similar.
Great for de-duplicating your por^M^M^MPhoto Collection.
I'm a perfectionist but I'm trying to cut back.
The big problem to me is specifying input. I know the "look" of the Mona Lisa's smile, but even with the best pen input methods I'd never be able to mimic DaVinci's subtle emotion of the smile; my hands just aren't capable of doing so. Using photos of the painting could simplify this, but this almost assumes that I'm only looking for the parody's and commercial exploiters of the image rather than the image itself (after all, I have the image to start with). And it raises the further issue that many photographic reproductions of the Mona Lisa that I can get my hands on are still under copyright and I'd be doing something legally questionable with an image long in the public domain.
Add to this the "infinite number of monkeys" issue where legally litigious companies will use technologies like this to scan the internet for litigation targets. Imagine Disney using a cell of Rafiki from the Lion King to find legally similar images that were created after the Lion King was released even if they were only superficially similar. Now do this for all movies back to Snow White or Steamboat WIlly and you could get to be a real visual mob boss with ownership (or at least threat of litigation) over huge libraries of works that weren't even created to intentionally violate Disney "Intellectual Property".
My need for this technology is small considering the input problems I'd have with my artistic abilities, while the litigation nightmare from large databases of "similar" visual data would seem to be more bothersome than helpful. I rather hope these visual search and categorizing methods don't catch on.
However, he put forth the concept of replacing the bit as the common unit of data with actual images - best described as holographic images of light manipulated by light. A picture really _would_ be worth a thousand words in such a system!
Download the demo for Taos if you've got a good 3d card. http://www.taos3d.com/
Yeah, in my early undergrad days I took a 200 level course for a gen ed requirement of discrete mathmatics. Wang was the professor, and too this day I haven't had a course that was as difficult or completely freaking insane as the one he gave. Glad he is doing more research and less teaching.
Pattern Rcognition is a novel by William Gibson, basically set in the present day or very near future. Image based search plays a central role in the plot. It's a very good read.
Did you mount a military-grade, variable-focus MASER on an unlicensed artificial intelligence?
I was looking at a picture of a plane on that web site and there was a link that said "Click for similar images". And what do you know? It brought up more pictures of planes. This is amazing stuff. How did it understand that I was looking at a picture of a plane?
Doesn't it make you feel good to know that our freedoms are protected by politicans, lawyers and journalists.
Whatever algorithm they're using, it seems to be sensitve to the horizon line, colour, shading, orientation of the aircraft, etc. It seems to be operating at the level of a pigeon (who have been shown to discriminate photos depicting trees, water, and particular people - as well as art by Picasso and Monet. See http://www.pigeon.psy.tufts.edu/avc/huber for other examples. It will be some time before algorithms can match on the basis of model numbers and such. It took humans quite a while to evolve a cortex to enable such fine discriminations.
of a black and white, really crude shape of a plane from above...
It actually found pictures that were very similar!!!!
I'm imagining and interface in google image search that lets me quickly paint a picture then search on that... of course I'm using a wacom tablet, so it's much easier, but with the tablet pc's coming...
Check out imgSeek: http://imgseek.sourceforge.net/
I wonder what an image search for GW Bush would produce, considering that he's the greatest leader the world has seen since Gidget.
Oh My ${deity}, the noise must be horrific! My ears are hurting just from *looking* at that picture.
Actually, engines aren't at full throttle during landing, so it's actually not that loud at all
The GIFT (the GNU Image-Finding Tool) is a Content Based Image Retrieval System (CBIRS). It enables you to do Query By Example on images, giving you the opportunity to improve query results by relevance feedback. For processing your queries the program relies entirely on the content of the images, freeing you from the need to annotate all images before querying the collection.
GIFT It worked pretty well for me in the demos they linked too. I have been waiting for this type of application to gain momentum.
Wax on, wax off baby!
Wow it works I sent a picture of a cockpit and I got cockpits back! It took a while but it worked :)
Taco?
Got this error when testing out the system, they certainly need to fix this, especially if they want it to become popular.
'Coz I'm looking for more information on this image.
It says "multi lock on" and a date, but all Google reports is other forum posts looking for the creator of the image. Apparently, there's a high-res version of it too.
http://www.cs.umd.edu/hcil/photomesa/
"PhotoMesa is a zoomable image browser. It allows the user to view multiple directories of images in a zoomable environment, and uses a set of simple navigation mechanisms to move through the space of images. It also supports grouping of images by metadata available from the file system. It requires only a set of images on disk, and does not require the user to add any metadata, or manipulate the images at all before browsing, thus making it easy to get started with existing images."
Actually, it will be hillarious what will happen when grandma puts in a picture of her grandson taking a drink from the hose in the backyard.
Its almost like telling someone to go to whitehouse.com
iView Media pro already does this. You just tell it to find dupes and set the tolerence to loose. http://www.iview-multimedia.com/
OMG Ponies!!! with Glitter!!!! I miss Pink
why bother making an algorithm that can recognise which images are porn and which are not when you can just set up a web site where people will do it for free? It reminds me of those "enter the characters in this image" tests that places like Yahoo do to ensure you can't sign up for a million email accounts a day. They're so easy to get around cause all you have to do is present the image to a man who wants porn and he'll happily provide his character recognition skills without charge.
How we know is more important than what we know.
I guess anyone can get a research grant these days.
Now, do this for something like Google Images or PBase or collections spanning infinite numbers of subjects and image sizes, then I'll get excited.
No, I've never had a rejection from A.net, I've never submitted there. Two minutes in their forum will tell you how anal their 'screeners' are, for whatever reason. It's just freakin' pictures of airplanes, for chrissakes.
As "sickening" as you may find it, children are actually one of the easiest items to produce, and as such one of the easiest items to replace. For instance, the 3 billion USD design of a marginally improved turbine is worth more than the lives of 100 children of the age of 5, considering even a high of 300 000 in productive capacity from unique attributes with perhaps 20 000 worth of productive time spent in emotional raising annually. Compared to 3 000 000 000 + 100 000 000, the 300 000 000 000 entering a burning building for instance it is better to save the plan than the children by practical measures. It is for the same sort of practical measures, over which all war is actually waged and not the smear/glorification campaigns dreamed up afterwards or falsified by hedging comments before or afterwards.
They are doing it based upon the shades of color in the image. So if you query for a image of an aircraft in flight with a lot of white clouds behind it, you get more of the same, but you also get aircraft parked on snow-covered ground.
So when can we search for music? I'm trying to find this song that goes like "dah daahh, dumpiti, dum dum"? Any answers?
Imagine if this was incorporated into Spotlight, not having to rename your entire photo library.
I've tried two different images of airplanes; one of a bright red flying car on bright green grass and one of SpaceShip One against a deep blue sky. Both times, the results looked surprisingly like my query images in color composition only. Red planes on grass and white planes against a blue sky. Inauspicious start.
Next experiment: I took a picture of a highly distinctive plane, a harrier, climbing at a steep angle and viewed in profile. I got, in return, a list of passenger jets, and even a helicopter. Hardly surprisingly, all of the result pictures had the same bluish white sky as my original image. That was literally the only similarity.
According to the introduction on the search page the heuristics used compares colors, contrast and shapes in the images themselves. I saw no correlation whatsoever between shapes, and any correlation in contrast seems to be to be the result of the search engine simply looking for images that contain the same colors in a similar ratio to the original. In short, nothing to see here, move along.
On the other hand, one of the projects listed under the Penn State University link looks fairly fascinating. The Riemann a-LIP project (automatic linguistic indexing of pictures) doesn't allow user input of images, unfortunately, but it does show some fairly fascinating attempts at verbally qualifying image data. For example, it describes a blue and orange mandelbrot as pattern agate shimer abstract scene, and a sunset over a lake as Berlin Devon Namibia landscape lake scene. Okay, it may still need some work, but it sure beats the hell out of the "find the same color airplane engine".
I want the fire back.
Now people will finally be able to know the names of those pornstars that they always see in the movies.
:-p
You know, like that guy who's in so many movies.
I forget his name.......well, now I guess I have a way to find out....
http://www.spilth.org/pictures/girls/ceren/jkh-and -babe.jpg
Impossible!
Oh, you mean like imgseek?
IBM was pushing this in 1996/1997 with DB2 v...uh...4? Didn't work all that well then...and it was just basic shape outlines.
Pie. Sky. In.
Blar.
some self-proclaim "comdeian" would post either porn pictures
Porn DOES however, make you spell better!
If image search is going to be the next big thing bots are going to need to crawl deeper and more often... Most engines probably don't have the resources currently to support the extra space/load.
All the torrents you could want.
...why does it show me pictures of donkeys when I use my school portrait for the search?
Non, je ne veux pas coucher avec toi ce soir.
see http://www.oracle.com/technology/products/intermed ia/index.html (oracle)
This document was written in 2002 (and that version is as old too)
I can remember some sales guy saying "You can look for a couch that's like this one. But blue."
So that means it should do a bit more than just color patches..
and then there is Google images..
nice thong and an A-340 -- Same beach.
747 landing
This famous pic is the GIS for 747 and beach.
This Photo ID has not been indexed in the similarity database yet.
It just keeps coming up with photos of airplanes.
^^vv<><>BA
Purdue also has a 3D shape search. More can be found at Here .
I'm pretty sure altavista had this feature several years ago, but removed it. I remember that it worked fairly well. Does anyone know what happened to it?
I'm sorry. The number you have reached is imaginary. Please rotate your phone 90 degrees and try again.
I took Professor Wang's networking class this Semester and have him for databases in the fall. He is a very interesting professor and was in the same class as the people who invented google.
quite right. Color-based retrievals are much easier and cheaper to implement, you know.
I did my BS thesis in CS/CE on CBIR back in '99 and decided to use interest points instead of color statistics in order to explore the options.
While color-based CBIR gives very "good" results as far as color match is concerned and human context is therefore many times hit on the spot and sometimes missed altogether, a simple interest points calculation (filtering the image with a double Gaussian etc.) yields "interesting" results, sometimes returning nice hits and sometimes grouping images with shapes that no human soul could count for similar.
Note that I used and tweaked a method that successfully used when checking for identical images (against forgeries, for example) as a foundation, deliberately, to explore its usefulness in searching for similar images.
Searching for similar images or checking if identical are actually two quite different things.
I imagine you would be just as disappointed using my thesis C program, as the field of computer vision, as we called it, was relatively fresh on the subject at the time. I admit I haven't done any further research since, having to make a living, so I'm not sure what's up and cookin' these days...
Naturally, there are entirely different methods that could serve CBIR purposes (machine learning, neural networks, metadata, text-tags-assisted stuff), but that be a different focus, not the computer vision approach.
Of course, given the usual course of things, it will instead be deployed at JFK's formerly-TWA terminal, assigned facial recognition tasks, and immediately declare everyone to be among the 10-most-wanted terrorists. I can't wait.
Village idiot in some extremely smart villages.
This is great and all, but I would love for a way to upload a graphic file to a search engine and have it report back any identical (or nearly identical.. perhaps a "threshold" setting) copies on the web.
This would be useful to me as a photographer to see if anyone out there is using my photos.
-David
> This is just asking for trouble. As most of you would probably imagine, some self-proclaim "comdeian" would post either porn pictures, or pictures that resembles porn body position.
Everyone around here is a "comdeian". I wish everyone on this site would grow up and take things seriously for once.
python -c 'print "butter me".join(["\0"*n for n in range(100,1,-1)*2+range(1,100)])' >
About a year or so ago, I and three other Masters students worked on a similar project at the University of Southampton.
I've not RTFA (not had the time), but our approach was to split the images into segments (based on colour and texture) which were assumed to be objects. The segments would then be analyzed for various feature vectors, such as shape, texture, colour etc. These vectors would then be added into a database of numbers, and finally the segments grouped, giving a collection of classified sections which (hopefully) represent similar objects.
From related metadata such as keywords, you could then hope to build up an idea of what keyword matches which section. You could also come up with a relevance between two images, and thus search for similar images.
We didn't have enough time to make it bulletproof by any means, but our limited results were very promising.
Sorry I can't find the paper, but we've got some screenshots of the application here and here (you can see false colouring applied to the original image to display the segments)
We Build Beautiful Websites
I've just found some more here and here
We Build Beautiful Websites
Image search will kinda work for airplanes in this database,as there are a very limited set of airplane model numbers, which are going to be attached to each photo.
But if the database didnt have these text clues the image search is going to be unlikely to see the similarity between an 747 in the air, as seen from the ground, with a head-on view of a 747, or one at the gate, or one in a hangar, or one in twilight, or one of a different color.
Maybe it could be done by outsourcing the task to India?
I was trying this out a bit, and have to admit that it's cool that something like this exists at all.
However, I think it would be better if it were able to realize what the 'background' was and filter it out. (Though I couldn't begin to guess how you'd do this.)
For example, I searched for this image. Many of the results are of something completely different, such as a white jet. Which is nothing like a camo helicopter. But the sky and the ground are pretty similar, and I think that's how it's matching.
It's incredible that we got this far, but I think there's still a long ways to go before it's at the stage where you put in an image, and are awed at how quickly it works.
Also note that I'd tend to think this would exponentially more difficult than searching HTML files, so it might be much more expensive to implement large-scale.
________________________________________________
suwain_2
(And yes, I'm posting this anonymously even though I have a registered account...)
Whistle or hum a tune in your microphone and have google fetch it.
Great for when you hear a tune on the radio and want to know the name.
Someone will make it some day, and when they do I want the royalties I am due.
The first thing I though of when seeing this was the next wave of law-suits.
Search for images you created, and you will find all kinds of similar/duplicate images.
Lots of web-sites uses images gotten from other sites, may modify them slightly, however much of the images is based upon images found on the net.
A program like this is likely to be popular among lawyers and design power houses.
Its almost like telling someone to go to whitehouse.com
"It's".
Also, whitehouse.com isn't a porn site any more. It's now some stupid real estate site.
Actually, this story (the veracity of I do not know) predates our modern concepts of "neural networks" - that is, multi-layer networks of nodes (typically three - input, output, and intermediary layers), in which the nodes simulate neurons via weighted thresholds and other mechanisms for "firing" an output based on inputs aggregated over time and/or frequency - coupled with back-propagation "learning"...
Instead, the story seems to have popped up soon after the introduction of, in the late 1950's to early 1960's - of the ideas behind the perceptron, the direct precursor to modern neural networks. Historical perceptrons could be conceptually visualized as simple, single layer neural networks.
So, your original statement applies, but I wanted to clarify where and how this story seems to have originated (computer science, on the whole, seems to be one area of research where almost nobody knows, remembers, nor seems to care - about its history, thus we seem to be forever reinventing the wheel in many areas). You find it brought up as an apocryphal story everywhere in liturature about neural networks, seemingly no matter how far back you go, until you get into the Minsky era (late 1950's - early 1960's) of such machines.
Reason is the Path to God - Anon
It will be interesting if we ever get to a stage where we can just search for a random object (or person) in a database of photos. Then you could take pictures of everything with an always-on camera and if you need to find where you put your car keys, just do a search.
#1660: [DigiGnome] Real life should have a fucking search function, or something.
[DigiGnome] I need my socks.
In general image understanding is equivalent to general AI. We won't get a CBIR system that works well before we get an AI that works and vice versa, because people expect to be able to match the *content* of the image they submit as template and not the general appearance of the image. The problem is then too unspecified.
Even in the restricted context of aeroplanes this is not a trivial problem. Someone in the list of replies submitted an image of a warthog (A-10) and got nonsensical results. Somehow the CBIR system would need to be able to infer a model of the A-10 from a given random 2D projection, and match it to the other 2D projections of the same A-10 model that they do have in the DB. This doesn't sound impossible but it is hard, and I suspect the Penn State people didn't do that. Instead they are probably matching on colour, texture, general appearance, etc.
This is not to say that CBIR is not a nice problem to apply new image processing/image analysis algorithms to, which are developed all the time.
When this becomes a little more refined, it will be nice to apply to frames of video to assist in creating video loops. For example, you mount a camera near the coast, and get 3 hours of footage of the waves crashing against the rocks. You want to make a nice loop of that, which can play indefinitely, but you want to avoid that 'jump' that always seems to occur. Find small sections of the movie where say...10 frames in a row are similar to 10 other frames... It'll take out a lot of the usual pain.
Surviving America
Like seemingly everyone else heree I also worked on an Image Searching tool for my group project during my Undergrad at Glasgow.
For possibly the worst report of all time, in both web presentation and actual content, see here: http://tlc.dcs.gla.ac.uk/students/level3/imagedb/
this will work great for photo stiching mosaics.
It finds pictures looking similar to the pixel ranges you want.