Recognizing Scenes Like the Brain Does

Roland Piquepaille writes "Researchers at the MIT McGovern Institute for Brain Research have used a biological model to train a computer model to recognize objects, such as cars or people, in busy street scenes. Their innovative approach, which combines neuroscience and artificial intelligence with computer science, mimics how the brain functions to recognize objects in the real world. This versatile model could one day be used for automobile driver's assistance, visual search engines, biomedical imaging analysis, or robots with realistic vision. Here is the researchers' paper in PDF format."

Comments

adverse effects

[prelelat]

If my computer could "see me" I think that it would BSOD its self to sleep. Long long sweet slumber.

Interesting, but what comes next?

[The Living Fractal]

I understand the reasoning behind modeling these systems on our own highly-evolved (ok, maybe not in some people) biological systems. What I want to see, however, is something capable of learning and improving its' own ability to learn. If our intelligent systems are always evolution-limited by the progress of our own biological systems then I can't see how A.I. smarter than a human will ever ben achieved. But if we are able to give these systems our own abilities as a starting point and then watch it somehow create something more intelligent than we are... then we really have something. Whether or not what we have is good at that point I can't say, though there are many people and communities in the world who are working on making sure this post-human intelligence doesn't essentially destroy us. Foresight for example.

I'm not knocking the MIT research, I think it's amazing. It just seems to me like imitation rather than imagination. Granted, highly evolved and complicated imitation. But does it even have the abilities of a parrot?

TLF

Re:Interesting, but what comes next?

[zappepcs]

It is interesting to consider the problem of AI researchers. How to create intelligence when it is not really understood. In the time between now and when we do understand it, we'll have to develop systems using logic and software that approximates how we understand it. A simple example is to ask yourself how many times that you had to learn that fire is hot? An AI system may have to learn this every time that you turn it on.

There is software systems that can approximate the size and distance between objects in a picture with reasonable accuracy, and if the scope of scenery presented to the system is limited, then that ability combined with sensing motion of objects is enough to determine a large percentage of what is desired. This is not the trouble or the hard part. The hard part is determining object classification and purpose in those times when it is not simple.

Each of us can almost always look at a scene and determine the difference between a jogger and a purse thief on the run or a businessman late for an appointment. For computers to do so takes a great deal more work. It is only a subtle difference and one where both objects maintain similar base characteristics.

The point? Even mimicking human skills is not easy, and fails at many points without the overwhelming store of knowledge that humans have inside their heads. This would point to the theory that if more memory was available, AI would be easier, but this is not true either. Humans can recognize a particular model of car, no matter what color it is and usually despite the fact that it might have been in an accident. The thinking that comes into play when using the abstract to extract reality from a scene is not going to happen for computers for quite some time.

The danger is when such ill prepared systems are put in charge of important things. This is always something to be wary of, especially when it is used to define/monitor criminal acts and identify those who are guilty whether that is on cameras at intersections or security systems, or government surveillance systems.

Re:Interesting, but what comes next?

[suv4x4]

If our intelligent systems are always evolution-limited by the progress of our own biological systems then I can't see how A.I. smarter than a human will ever ben achieved.

You know this is pretty misleading so you can't take any blame for thinking so. Lots of people also think that we're also "a hundred years smarter" than those living in the 1900's, just because we were lucky to be born in a higher culture.

But think about it: what is our entire culture and science, if not ultra sped-up evolution. We make mistakes, tons of mistakes, as human beings, but compared to completely random mutations, we have supreme advantage over evolution in the signal/noise ratio of the resulting product.

Can we ever surpass our own complexity in what we create? But of course. Take a look at any moderately complex software product. I won't argue it's more complex than our brain, but something else: can you grasp and asses the scope of effort and complexity in, say (something trivial to us), Windows running on a PC, as one single product? Not just what's on the surface, but comprehend at once every little detail from applications, dialogs, controls, drivers, kernel, to the processor microcode.

I tell you what: even the programmers of Windows, and the engineers at Intel can't.

Our brain works in "OOP" fashion, simplifying huge chunks of complexity into a higher level "overview", so we could think about it in a different scale. In fact, lots of mental diseases, like autism or obsessive compulsive disorders revolve around the loss of ability to "see the big picture" or concentrate on a detail of it, at will.

Together, we break immensely complex tasks into much smaller, manageable tasks, and build upon the discoveries and effort we made yesterday. This way, although we still work on tiny pieces of a huge mind-bogglingly complex puzzle, our brain can cope with the task properly. There aren't any limits.

While I'm sure we'll see completely self-evolving AI in the next 100 years, I know that developing highly complex sentient AI with only elements of self-learning is quite in the ability of our scientists. Small step, by small step.

Re:Interesting, but what comes next?

[Xemu]

Each of us can almost always look at a scene and determine the difference between a jogger and a purse thief on the run or a businessman late for an appointment.

Actually, we can't, we just base this recognition on stereotypes. A well known Swedish criminal called "the laser man" exploited this in the early 90s when robbing banks. He would rob the bank and then change clothes to a business man or a jogger, and then escape the scene. The police would more often than not let him pass through because they were looking for a "escaping robber", not for a "business man taking a slow paced walk".

The police caught on eventually and caught the guy. Computers would of course have even greater difficulties to think "outside the box".

nothing new

[Anonymous Coward]

After scanning this paper, their model extends nothing in the state of the art in cognitive modeling. Others have produced much more comprehensive and much more biologically accurate models. There's no retinal ganglion contrast enhancement, no opponent color in LGN (or color at all), no complex cells, no Magno/Parvocellular pathways, no cortical magnification, no addressing of aperture problem (seem to treat scene as a sequence of snapshots, while the brain... does not) the object recognition is not biologically inspired. Some visual system processes can be explained with feedforward only mechanisms, but all visual system processes can't.

Re:nothing new

[kripkenstein]

I agree that the paper isn't revolutionary. In addition, it turns out that, after the 4-layer biologically-motivated system, they feed everything into a linear SVM (Support Vector Machine) or gentleBoost. For those that don't know, SVMs and Boosting are the 'hottest' topics in machine learning these days; they are considered the state of the art in that field. So basically what they are doing is providing some preprocessing before applying standard, known methods. (And if you say "but it's a linear SVM", well, it is linear because the training data is already separable.)

That said, they do present a simple and biologically-motivated preprocessing layer that appears to be useful, which reflects back on the brain. In summary, I would say that this paper helps more to understand brain functioning than to develop machines that can achieve human-like vision capabilities. So, very nice, but let's not over-hype it.

I'm not getting it, why it is significant ?

[S3D]

Gabor wavelets, newral networks, hierarchical classifiers in some semi-new combination - there are dozens image recognition papers like this every month. Why this exact paper is special ?

research done at cyberdyne

[macadamia_harold]

Researchers at the MIT McGovern Institute for Brain Research have used a biological model to train a computer model to recognize objects, such as cars or people, in busy street scenes.

this is, of course, the first step in finding Sarah Connor.

Re:Does anybody know where to find the actual pape

[gardyloo]

There was. You didn't recognize it.

My own two cents

[MillionthMonkey]

I've written here before about epileptic seizures I have that start somewhere in the right occipital lobe possibly near V1, based on the nature of the aura and a recent video EEG last month. These things started for no reason when I was a teenager and now involve these interesting post-ictal fugue states where only chunks of my brain seem to be working but I'm still able to run around and get in trouble. I've developed a talent over the years for coping with brain trauma and sort of bullshitting my way through it.

Usually I'm not forming long term memories during fugue states, but when I do, I remember some pretty interesting stuff. One thing that is typically impaired is object recognition, since this mostly seems to be handled by the right occipital lobe. I can see things but can't immediately recognize what they are, unless I use these left-brain techniques. The left occipital lobe can recognize objects too, but the approach it takes is different and more of a pain in the ass to have to rely on. It's more of a thunky symbolic recognition, as opposed to an ability to examine subtle forms, shapes, and colors. I have to basically establish a set of criteria that define what I'm looking for and then examine things in the visual field to see if they match those criteria. I'll look for a bed by trying to find things that appear flat and soft; I'll look for a door by looking for things with attributes of a doorknob such as being round and turnable; I'll find water to drink by looking closely at wet things. My wife says I make some interesting mistakes, like once confusing her desk chair for a toilet (forgetting for a moment that part of a toilet has to be wet, but at that point memory formation and retrieval is disrupted to the point where I could imagine forgetting that it's not enough to just be able to be sat on, toilets have to have water in them too). I have trouble recognizing faces, and she says I'm sometimes obviously pretending to recognize her. Recognizing a face using cold logic can be tricky even when you're not impaired. Recognizing familiar scenes and places becomes difficult. I drove home in a fugue state once, back in my twenties, and while I didn't crash into anybody or have any sort of accident, I did get lost on the way home from work. I ended up driving miles past where I lived. Even as a pedestrian, getting lost in familiar areas is still a problem.

People have been trying to come up with image processing algorithms that mimic cortical signal analysis for decades. I remember reading papers ten years ago like this. It's amazing to see they're still mistaking road signs for pedestrians. I don't think even I could make an error like that. The state of the art was totally miserable back then, too. Neuroscience has got to be one of the sciences most poorly understood by humans.

Earlier work 1989-1997 on street scene analysis

[Wills]

Apologies for blowing my own trumpet here, but there was much earlier work in the 1980s and 1990s on recognizing objects in images of outdoor scenes using neural networks that achieved a similarly high accuracy compared to the system mentioned in this article:

1. WPJ Mackeown (1994), A Labelled Image Database, unpublished PhD Thesis, Bristol University.

Design of a database of colorimetrically calibrated, high quality images of street scenes and rural scenes, with highly accurate near-pixel ground-truth labelling based on a hierarchy of object categories. Example of labelled image from database

Design of a neural network system that recognized categories of objects by labelling regions in random test images from the database achieving 86% accuracy

The database is now known as the Sowerby Image Database and is available from the Advanced Technology Centre, British Aerospace PLC, Bristol, UK. If you use it, please cite: WPJ Mackeown (1994), A Labelled Image Database, PhD Thesis, Bristol University.

2. WPJ Mackeown, P Greenway, BT Thomas, WA Wright (1994).
Road recognition with a neural network, Engineering Applications of Artificial Intelligence, 7(2):169-176.

A neural network system that recognized categories of objects by labelling regions in random test images of street scenes and rural scenes achieving 86% accuracy

3. NW Campbell, WPJ Mackeown, BT Thomas, T Troscianko (1997).
Interpreting image databases by region classification. Pattern Recognition, 30(4):555-563.

A neural network system that recognized categories of objects by labelling regions in random test images of street scenes and rural scenes achieving 92% accuracy

There has been various follow up research since then

Hope most folks realize, once they get down vison

[Maxo-Texas]

It's going to change everything.

Robotic vision is a tipping point.

A large number of humans become unemployable shortly after this becomes a reality.

Anything where the only reason a human has the job is because they can see is done in the 1st world.

Why should you pay $7.25 an hour (really $9.25 w/benefits & overheard for workers comp, unemployment tax, etc.) when you can buy a $12,000 machine to do the same job (stocking grocery shelves, cleaning, painting, etc.).

The leading edge is here with things like roomba's.

Re:not like the brain does.

[odyaws]

Disclaimer: I work with the MIT algorithms daily and know several of the authors of this work (though I'm not at MIT).

This paper's claim to recognize scenes like the brain does, is overdrawn. As far as i can tell from their paper (it is a journal version of their cvpr paper) only their low-level Gabor features are similar to what the brain does.

Their low-level Gabor filters are indeed similar to V1 simple cells. The similarity between their model and the brain goes a lot further, though. The processing goes through alternate stages of enhanced feature selectivity with roughly Gaussian tuning (the S layers) and pooling over spatial location and scale via a max operation (the C layers). If you read more papers from their lab, there is a significant amount of biological plausibility in both of these operations, and a great deal of effort has gone into tuning the various layers to behave in accordance with physiological data.

The rest of the paper uses the currently popular bag-of-features model, which is a model that discards all spatial information between image features, which i don't think the brain does.

The model is roughly equivalent to a bag-of-features, but with the nice feature (from a biologist's perspective) that it builds the bag in a biologically plausible way. The features themselves are picked randomly from natural images in a training stage that takes the place of human development. Discarding spatial information makes the model a lot more tractable, and it isn't clear what role spatial information plays in the processing of the ventral visual system, which is what their algorithm models.

Furthermore, for classification algorithms they consider a Support Vector Machine and Boosting. Both of these classifiers are certainly not comparable to what the brain does. Why not use a neural network if they aim is to mimic the brain?

They use these classifiers on top of their algorithm simply to determine how good the model was at extracting relevant feature information. Since they want to quantify how much information is there, it is wise to choose the best method they can to locate the information.

Furhtermore, they only conside feed-forward information, where research shows that there is at least as much information going back as there is going forward.

Feedback is definitely very important (this is what my own research is about), but feedforward accomplishes a lot with a vastly simpler computational model.

Don't get me wrong, it is still a nice paper, with good results. (however, all Caltech datasets are highly artificial, with objects artificially rotated in 1 direction) So, nice paper, but to compare it with the workings of the human brain is too much.

Here are the Caltech datasets they used: vision.caltech.edu. I think the "artificial" datasets you refer to are the "3D objects on turntable," which are a bit artificial. However, the images they refer to in the paper discussed here are from the Caltech-101 dataset, which consists of real-world images of objects from 101 different categories - most of the images are not at all artificial.

Fine paper, but why not quote all of PAMI ?

[HuguesT]

This is a nice paper by respected researchers in AI+Vision, however pretty much the entire content of the journal this was published in (IEEE Pattern Analysis and Machine Intelligence) is up to that level. Why single out that particular paper ?

Interested readers can browse the content of PAMI current and back issues and either go to their local scientific library (PAMI is recognisable from afar by its bright yellow cover) or search on the web for interesting articles. Often researchers put their own paper on their home page. For example, here is the publication page of one of the authors (I'm not him).

For the record, I think justifying various ad-hoc vision/image analysis techniques using approximations of biological underpining is of limited interest. When asked if computer would think one day, Edsgerd Dijkstra famously answered by "can submarine swim?". In the same manner, it has been observed that (for example) most neural network architectures make worse classifiers than standard logistic regression, not to mention Support Vector Machines, which what this article uses BTW.

The summary by our friend Roland P. is not very good :

This versatile model could one day be used for automobile driver's assistance, visual search engines, biomedical imaging analysis, or robots with realistic vision

• There already exist working automated driving software. The december 2006 issue of IEEE Computers magazing was on them last month. Read about the car that drove a thousand miles on Italy's road thanks to Linux, no less.
• Visual search engine exist, at the research level. The whole field is called "Content Based Retrieval", and the main issue is not so much to search, but to formulate the question.
• Biomedical image analysis has been going strong for decades and is used every day in your local hospital. Ask your doctor !
• Robotic vision is pretty much as old as computers themselves. There are even fun robot competitions like robocup.

I could go on with lists and links but the future is already here, generally inconspicuously. Read about it.