Searching by Image Instead of Keywords

Content based image retrieval (CBIR), the technique to search for images not by keywords, but by comparing features of the images themselves has been the focus of much research ever since the web emerged. Consider for instance adding CBIR to Google Images, where you would be able to search for images similar to a query image instead of using keywords. A research project at Penn State University has recently been applied to the biggest aviation photo database in the world with close to 800,000 images. You can search for images similar to a photo already in their database (click "View similar photos") or submit your own query image. Some queries generate better results than others but CBIR is certainly here to stay and will be standard in many image applications of the future.

Comments

Think of the greatness to society!

[qewl]

I can't wait to put a nipple into it!

Location?

[poopdeville]

What an awful beach.

Wow

[themoodykid]

I was just thinking about this the other day. I think content-based image search is one of the Next Big Things. Cameras are so ubquitous now (for better or worse), but having to rely on metadata to give meaning to images requires lots of effort up front.

It will be interesting if we ever get to a stage where we can just search for a random object (or person) in a database of photos. Then you could take pictures of everything with an always-on camera and if you need to find where you put your car keys, just do a search.

Re:Wow

[theguyfromsaturn]

If you are only interested in searching for images on your own computer, have a look at imgSeek. http://imgseek.python-hosting.com/

It's been around for some time now. You can not only use an existing image to search, but also do a rough sketch. Check the screenshots:

Nice complement to what has been presented in this article.

Re:wtf?

[Rei]

Because it still has problems - you'll note that the pictures seem to be compared simply based on color similarity. That's the same thing imgSeek does (a great program for sorting and searching your photos) on photo searches. It works wonderfully if you're searching a very limited picture subset (say, airplanes), but if you search a wide variety of pictures, the results can be quite amusing.

And for 'lynx' users...

[mikael]

... the search engine will support ASCII art image searches.

Some relevant research papers

[FleaPlus]

There's a bunch of interesting papers out there on content-based image analysis and retrieval. Below is a sampling from my bibtex file. Does anyone else have others they'd like to share?

* Finding Naked People (Fleck et al, 1996)

* Video google: A text retrieval approach to object matching in videos (Sivic & Zisserman, 2003): web page demo here

* Names and Faces in the News (Berg et al, 2004)

* FACERET: An Interactive Face Retrieval System Based on Self-Organizing Maps (Ruiz-del-Solar et al, 2002)

* Costume: A New Feature for Automatic Video Content Indexing (Jaffre 2005)

and then we have reverse "Googling" for images..

[dotpavan]

Here is a google game which is reverse of google's image search:

One has to guess the search word which generated a given set of 20 images in google's image search

When things are moving forward, we have soomthing to talk about "those good ole days" but frankly the game is interesting initially but later gets boring due to the frequent repetitions..

Is it just colour?

[Bifurcati]

I just did a quick search based on this image of a Qantas logo (that's the main Australian airline, in case you're wondering...) It's red, with a white kangaroo in the middle. My theoretical aim was to find photos of Qantas planes.

What I got was an awful lot of red planes - some of which were actually Qantas planes, but I think more by coincidence (i.e., they're red) than design. Many images had nothing to do with Qantas, or even a red plane - they simply had a lot of red in the image.

This is impressive in some ways, but in others it seems like it's simply looking for similar patches of colour. I haven't done enough testing to see what happens if,say, I gave it a half red half green image.

Interesting, but not ready for public consumption just yet. A bit like A.L.I.C.E. the artifial intelligence system actually - neat, but not practical. Yet!

Great!

[SetupWeasel]

Now I can find all the other naked pictures of Bea Arthur on the web!

There is a GNU project related to this GIFT

[capedgirardeau]

From gnu.org:

The GIFT (the GNU Image-Finding Tool) is a Content Based Image Retrieval System (CBIRS). It enables you to do Query By Example on images, giving you the opportunity to improve query results by relevance feedback. For processing your queries the program relies entirely on the content of the images, freeing you from the need to annotate all images before querying the collection.

GIFT It worked pretty well for me in the demos they linked too. I have been waiting for this type of application to gain momentum.