Microsoft Seeks Patent For "Search By Sketch"

theodp writes "So, how does one search for images that aren't tagged with keywords? Google does offer its sometimes-spotty search by image, but what if you don't have an image handy that looks like what you're searching for? Microsoft, reports GeekWire, offers a solution that's 'a little like playing Pictionary with a search engine — drawing a sketch and seeing if the algorithm can return pictures that match it.' That's the concept behind Microsoft Research's patent-pending 'MindFinder' project, which has already been incorporated into a Windows Phone app called Sketch Match. A patent application made public Thursday notes that touch computing makes sketching easier than ever, making one wonder if we'll be 'giving Bing the finger' with Windows 8!"

Comments

Testing would be interesting

[Rogerborg]

Could they make it work for the 5% of sketches that aren't boobs or dongs?

Re:Testing would be interesting

[mwvdlee]

They'll just return the same nude picture on every search and claim 95% accuracy.

Re:Testing would be interesting

[Anonymous Coward]

if you draw a penis you might get a doom rocket launcher

Re:Testing would be interesting

[Samantha Wright]

Go find out for yourself! Here's some prior art. It's not very good (it doesn't understand the concept of edges like MS's stuff does), but unfortunately the article doesn't have a link to the actual patent to tell if it's specific enough... the link that should go to the patent goes to the app on the Windows Phone site. (Certainly, the algorithm is impressive, though.)

Re:Testing would be interesting

[dotancohen]

Could they make it work for the 5% of sketches that aren't boobs or dongs?

This feature already exists in Digikam, from KDE:
http://digikam.org/

Interestingly, the feature works fine for things that aren't boobs or dongs, because boobs need circles and dongs need straight lines. It fails on normal geometric shapes, but works great on more abstract drawings.

Re:Testing would be interesting

[Canazza]

or half the output of b3ta.com

Re:Testing would be interesting

[Anonymous Coward]

Hate to break it to you, but prior art or not, it's legit purely because of what you've mentioned. You can license prior art, and patent your improvements, if they substantial enough. This (understanding edges, therefore making it actually work) is substantial enough to be patentable, provided it was actually derived from the earlier work, and was licensed -- OR -- is the prior art is sufficiently specific to show that this method was derived independently and using different methods.

This isn't a popular opinion in these parts, but if this actually works, this is pretty bloody cool, and really rather useful. Stuff like this is why the patent system exists, and it's certainly patent worthty.

Re:Testing would be interesting

[Samantha Wright]

Actually I absolutely agree with you—I just wish we could see the patent to make sure that they were actually patenting their innovations, not trying to claim all forms of sketch-based image search input.

Re:Testing would be interesting

[hairyfeet]

Easy way to find out. Hey are any of the 3 guys that have a WinPhone here? Hey could you load up Sketch Match for us and draw a sweaty monkey and see if it gives you a picture of Ballmer? i know its not a pretty sight but its for science, so be a pal, take one for the team. Of course i'd STRONGLY suggest you do NOT write "Take one for the team" in Sketch Match as the images you get back...well you just don't want those.

Re:Testing would be interesting

[Noughmad]

Yes, and it works quite well I noticed.

However, this patent will most likely include "in the cloud" and/or other buzzwords.

Interesting concept, but...

[sidthegeek]

...don't most smartphones have cameras already? Wouldn't it be faster and easier for the user to just take a photo of whatever he/she wants to identify or search? (Of course, if they don't have a photo of the object available at the time, then this idea would make more sense.)

Re:Interesting concept, but...

[zill]

Sounds like you're talking about Google Goggles.

Re:Interesting concept, but...

[leptogenesis]

No, Google Goggles does nothing like this. Google Goggles (and Google search-by-image) is, from the experiments I've done, instance-based image retrieval. That is, it can match objects with exactly the same shape (given a picture of the Eiffel Tower, it will return other images of the Eiffel Tower). However, given a drawing, even a good one, the contour shapes won't match quite well enough, and the algorithm will return garbage. The same can be said for 'deformable objects' like dogs and people.

In fact, I'm quite sure that nothing like this exists. I'm not sure about the actual search engine part of all this, but I did see a talk last fall by one of the researchers who worked on ShadowDraw, which I'm reasonably sure is going to be a component of the final system. The real problem that *they* had to solve was the simple fact that the average person is a horrible, HORRIBLE artist. Ask them to draw a rabbit and for 90% of people, it will come out as a blob that might be an animal, but that's about all you can tell. The algorithms they talked about that actually make the system work as well as it does were actually quite impressive--extremely fast contour indexing, contour combination, converting real photos into convincing sketches--it all sounds easy, but I dare you to actually try implementing it.

Now--and let's see what happens to my karma for saying this--I actually kinda think they deserve a patent for this. Not for coming up with the idea of drawing-based search; that idea is obvious. However, making a system that works as well as ShadowDraw is quite an achievement, and more importantly, Microsoft Research would never have released the algorithm to the public unless it could be patent-protected. Patents in this case aren't about protecting Microsoft's innovation; it's about motivating Microsoft to publish for the sake of other innovators.

Overaged business model

[vikingpower]

Microsoft, once more, is one the very large corporations that still think they can base their business, in the 21st century, on patents. I wonder how long it is ( yet ) going to taken before corporations of this size realize that such practice is old-fashioned ?

Already Done

[Anonymous Coward]

Already done here for LaTex symbols: http://detexify.kirelabs.org/classify.html

Please please please don't approve the patent. I haven't read it, but according to the summary there's nothing new about it.

Re:Already Done

[rveldpau]

I agree... it really is just an image search anyways, it's just much looser in its algorithm

Re:Already Done

[JasterBobaMereel]

3 minutes and we are up to 4 systems that show prior art already, do the Patent office actually do any searches ...?

oh sorry this is "...on a mobile device" so they don't count ..?

Re:Already Done

[Anonymous Coward]

I'm sure you can access Detexify from a mobile phone as well.

Re:Already Done

[Miros]

You're citing prior art for the concept, which is not the same thing as prior art for what is actually being patented. There are 1000s of patents for "mousetraps." The fact that a mousetrap had been invented by someone first did not mean that other people could not invent other kinds of mousetraps. Just because people have built sketch based search technology before does not mean that Microsoft (or anyone else for that matter) can't invent an improved version of sketch based image search and then patent that method. That appears to be what they have done here.

Re:Already Done

[Miros]

Also, as a note, there are already dozens if not hundreds of patents for this concept as well: https://www.google.com/search?q=sketch+image+search&btnG=Search+Patents&tbm=pts&tbo=1&hl=en

Re:Already Done

[Anonymous Coward]

Using sketches to find actual persons...good thing the police or the FBI didn't patent that!

Re:Already Done

[Anonymous Coward]

uh, also, thats not an image search. thats drawing text symbols. which has been around for years. and even according to the programmer, the software has to be "trained". i'm pretty sure microsoft is using something 100% different than this to do image searching based on a sketch. the software will have to analyse the sketch for edges and determine shapes that are similar to shapes in their image index. they aren't going to sit there and train the software and tell it to link to specific images if someone droes this specific shape.

however, there is program that does pretty much exactly this only i think its better, becase you can label parts of your sketch, and because it includes a composite algorithm as well. And i've been excited about this possiblility for a while. http://cg.cs.tsinghua.edu.cn/montage/main.htm

Re:Already Done

[Nerdfest]

There are also actual implementations, like this one for Linux.

Re:Already Done

[Miros]

Indeed, implementations of the concept but are these implementations of the specific claims of Microsoft's patent application? That's what matters for prior art.

Re:Already Done

[Anonymous Coward]

which is not the same thing as prior art for what is actually being patented.

The PTO basically handwaves when deciding whether two ideas are the same or different. This means that whether something has been invented before is a pointless question because the PTO can always find some arbitrary difference. "Prior art" and the PTO's idea of whether an idea is new is almost meaningless. They can't even reliably separate idea from terminology let alone define what a substantive difference is. It's all handwaving that costs billions.

digikam had it for a long time

[Anonymous Coward]

if anybody is interested in testing the concept...

Re:Overaged business model

[Robert Zenz]

As long as it works: Never.

I wonder

[jeesis]

Will Microsoft block pornographic searches using it? I feel like drawing two circles with dots in the center instead of searching for "porn".

My first thought

[subreality]

Oh please [sketch sketch] let there be porn like this [scribble]

Re:My first thought

[Guppy]

Oh please [sketch sketch] let there be porn like this [scribble]

Now pipe 4chan to it, and watch Microsoft's server commit suicide by setting itself on fire.

Re:My first thought

[shutdown -p now]

This so needs to be integrated into http://rule34.paheal.net/

Already exists

[Anonymous Coward]

That functionality was integrated a few years back into digikam.
How can those corporations always get away with patenting stuff under prior art?

Re:Already exists

[__aaltlg1547]

That functionality was integrated a few years back into digikam.
How can those corporations always get away with patenting stuff under prior art?

There are two ways.

1. Simply ignore prior art and claim everything as your own invention.

2. Make a trivial change and then claim that and bamboozle the patent office into believing you have therefore invented the wheel.

Re:Already exists

[Stewie241]

or
3. Recognize that there is legitimate room within patents for doing something that has already been done but doing it in a non obvious, novel way.

I'm not saying this is the case for this particular patent, but just pointing out that in my understanding patents aren't always about doing a certain thing but rather doing a certain thing in a certain way. Obviously this limits the scope of the patent to that one certain way.

So, to put it in slashdot terms (i.e. a car analogy), pretty much every car sold today has shock absorbers, and the standard way of doing this is to use metal coil springs. If somebody were to put time and research into improving the design of shock absorbers and came up with a design that used marshmallows instead of metal coil springs, I would consider that novel and I think you would agree that it is non-obvious. To me, if we have the patent system that we have, that is at least a legitimate use of it. You aren't patenting shock absorbers, you are patenting shock absorbers that use marshmallows to absorb the shock of the impact.

Re:Already exists

[FreeUser]

1. Simply ignore prior art and claim everything as your own invention.

2. Make a trivial change and then claim that and bamboozle the patent office into believing you have therefore invented the wheel.

You left out:

3. Simply pay the US Patent Office Bribe, I mean "Fee". Patent approved, no questions asked.

(And quietly, from one patent lawyer at the USPTO to another, at Corporation X, or defending defendent Y: go forth and make money!. We win, the only losers are the corporations who filed, and the innovators who "violated" the bogus patent. Extra points if you get a judge who was once a patent attorney. Any way you slice it: Profit!).

Re:Already exists

[SadButTrue]

Nerds! You made me unmod :(

A shock absorber is a piston in a fluid column. It's force is dependent on the velocity through the working fluid.
A spring on the other hand produces a force proportional to its displacement not its velocity.

Confusion could arise because modern cars don't use separate units but instead use a strut assembly which is a shock + spring.

Re:Already exists

[andydread]

I think i this case his analogy would be replacing the fluid in the shocks/struts with marshmallows since the springs mainly is what maintains ride hight.

Re:Already exists

[__aaltlg1547]

or
3. Recognize that there is legitimate room within patents for doing something that has already been done but doing it in a non obvious, novel way.

I'm not saying this is the case for this particular patent, but just pointing out that in my understanding patents aren't always about doing a certain thing but rather doing a certain thing in a certain way. Obviously this limits the scope of the patent to that one certain way.

So, to put it in slashdot terms (i.e. a car analogy), pretty much every car sold today has shock absorbers, and the standard way of doing this is to use metal coil springs. If somebody were to put time and research into improving the design of shock absorbers and came up with a design that used marshmallows instead of metal coil springs, I would consider that novel and I think you would agree that it is non-obvious. To me, if we have the patent system that we have, that is at least a legitimate use of it. You aren't patenting shock absorbers, you are patenting shock absorbers that use marshmallows to absorb the shock of the impact.

Most patents are about doing something a certain way. They ALL should be. But many patents and even more so claims of infringement involve unjustified extension of the patented invention to anything that's resembles its function at a much higher level.

And now the patent office is allowing people to patent *ideas* that have never been reduced to practice, so that the invention that's patented can't actually be demonstrated until somebody else invents an enabling technology that doesn't exist yet. This only feeds patent trolls without producing any useful technology.

Re:Already exists

[__aaltlg1547]

Nerds! You made me unmod :(

A shock absorber is a piston in a fluid column. It's force is dependent on the velocity through the working fluid.
A spring on the other hand produces a force proportional to its displacement not its velocity.

Confusion could arise because modern cars don't use separate units but instead use a strut assembly which is a shock + spring.

Nerds! You made me unmod :(

A shock absorber is a piston in a fluid column. It's force is dependent on the velocity through the working fluid.
A spring on the other hand produces a force proportional to its displacement not its velocity.

Confusion could arise because modern cars don't use separate units but instead use a strut assembly which is a shock + spring.

That's not the only possible kind of shock absorber. For instance, one could make magnetic shock absorbers that dissipate energy by moving a powerful magnet by a metal plate. Or you could have electromechanical shock absorbers that sense motion and acceleration and use fast-acting linear motors to actively absorb the shock. The energy could be captured and used to recharge the batteries in your car.

Like digiKam has had for the last few years?

[Anonymous Coward]

And several other "search by sketch" technologies I've seen in the last decade or so...
What's the big claim in the patent that differentiates it?

Re:Like digiKam has had for the last few years?

[DrSkwid]

it used to be that they added "on the internet", now they add "on a mobile phone", looks like this one adds "with fingers on a mobile phone"

Fuzzy Users

[garthsundem]

Hopefully sketchers will be more accurate than hummers or whistlers -- a la Midomi.com. But I doubt it -- can't wait to see if this survives beta.

Prior art

[jason_g_haines]

I remember seeing a research paper over 12 years ago describing exactly this concept.

Re:Prior art

[Anonymous Coward]

Digikam has had something similar for at least a year, though their system seems to be based on colours rather than shapes.

Re:Prior art

[Anonymous Coward]

imgSeek has this since 2003...

Re:Prior Art

[lucidlyTwisted]

That's what I was thinking. I am not sure how well TinEye copes with "fuzzy" searches and my crap drawing, but even if TinEye is poor at it, MS's latest wheeze is just an obvious refinement of a pre-existing service.
Still, this is the USA and they will allow patents on anything.

Re:prior art

[lucidlyTwisted]

Wow - now that is good!

Patent the concept or specific algorithm?

[rollingcalf]

I oppose software patents in general, but if this patent covers their specific algorithm I don't really have a problem with that.

The trouble is that regardless of whatever algorithm they describe in the patent, the patent can be used to sue others who use a different algorithm to implement the same concept.

So they can get a patent on an ineffective algorithm for image search, wait for somebody else to create a better algorithm that is actually effective, then sue the implementor of the better algorithm. The patent effectively covers the concept, not the algorithm, hindering innovation by preventing others from implementing their own different algorithms for the same concept.

Re:Patent the concept or specific algorithm?

[rveldpau]

The problem is that you can't actually patent an algorithm, there are laws that prevent that; but that does stop sleazy lawyers from working around that problem, and what that gives us is overly broad patents. If we let people patent algorithms, we might actually be better off.

Re:Patent the concept or specific algorithm?

[rveldpau]

I meant doesn't stop... I should really proof-read a bit better before I post

Re:Patent the concept or specific algorithm?

[Anonymous Coward]

It's ok. I'm sure MS has some proof-reading/grammer checking patents too. We all just assumed you didn't have permission to use those.

Re:Patent the concept or specific algorithm?

[Theaetetus]

I oppose software patents in general, but if this patent covers their specific algorithm I don't really have a problem with that.

The trouble is that regardless of whatever algorithm they describe in the patent, the patent can be used to sue others who use a different algorithm to implement the same concept.

So they can get a patent on an ineffective algorithm for image search, wait for somebody else to create a better algorithm that is actually effective, then sue the implementor of the better algorithm. The patent effectively covers the concept, not the algorithm, hindering innovation by preventing others from implementing their own different algorithms for the same concept.

Not really... As others have pointed out, the general concept was already done by ImgSeek. So, Microsoft's patent will have to be narrower than the entire concept if they want it to be granted - they'll have to patent a specific implementation. That leaves others free to make different implementations, too. Additionally, if someone does use Microsoft's implementation but makes it better, more efficient, more effective, that someone can get a patent on that improvement, and Microsoft would have to pay them for a license if they want to use it (or cross-license their patent and the improvement patent). This encourages innovation, by (a) leaving avenues open for new implementations; (b) allowing improvements; and (c) requiring Microsoft to publish the details of their implementation so that people can do (a) and (b).

Re:Patent the concept or specific algorithm?

[rollingcalf]

That's fine in theory, but the reality is that many do get sued and lose for implementing a similar concept using a different algorithm.

Re:Patent the concept or specific algorithm?

[Theaetetus]

That's fine in theory, but the reality is that many do get sued and lose for implementing a similar concept using a different algorithm.

Agreed... but that points to the problem being in the litigation stage, not in the patent application stage.

Re:Patent the concept or specific algorithm?

[gl4ss]

just implement this 1:1 . and put a "on a mobile device" at the end of your patent application. I guess there's going to be already 100 such patent applications by tonight evening though..

Re:Patent the concept or specific algorithm?

[Anonymous Coward]

Actually, you can patent an algorithm. GIF had a patent on part of the algorithm which PNG was designed to get around. RSA had patents on the encryption algorithm which I think lead to problems for PGP. You cannot patent math. I am not sure where the dividing line is.

What about false positives?

[Anonymous Coward]

I'd be much more concerned about false positives.

Say you're a university lecturer, delivering a lecture to 400 mechanical engineering students. During this lecture, you wish to show your students an image of a mechanical part or tool that vaguely resembles the shape of a penis. Let's assume it's a wrench or a bracket of some sort, where there's a long, straight, relatively thin shaft with a large bulbous D-shaped end with a hole in the middle.

Now, you forgot to find an image beforehand, but your laptop is hooked up the lecture room's projector and the university's wireless network. So go to do a search for some pictures of said mechanical device, and you use a sketching service to find a picture of it. Unfortunately, your wrench or bracket is mistaken for a fat hairy penis.

You're now in a pretty bad position. You've just unexpectedly shown 400 people a grid of penis photos. Some are laughing, but that's the least of your worries. The real problem will be those who are recording it on their cell phones. Soon it'll be all over the Internet. It'll likely go viral.

Unfortunately, a small minority of those in the lecture will take great offense to being shown pictures of penises during a mechanical engineering lecture. They'll launch lawsuits against you and the university.

Basically, your career could be ruined, all over a drawing of a wrench or a bracket that looks vaguely like a penis.

Re:What about false positives?

[Anonymous Coward]

Is this serious or a really stupid joke? Obviously "Safe Search" options already exist. Also obvious, it's never a good idea to search for anything of visit webpages you've never been to before in front of 400 other people. Neither of these things are new, and "Search by Sketch" changes neither of them. For the record, you hit porn on page 2 on Google Image Search for "Wrench or bracket" if Safe Search is off. So far, I've yet to find a search term that doesn't produce porn on page 2 or earlier if Safe Search is off.

Re:What about false positives?

[Anonymous Coward]

Basically, your career could be ruined, all over a drawing of a wrench or a bracket that looks vaguely like a penis.

AKA "The Ballmer Squirt"

Re:What about false positives?

[mjjochen]

Um, two things:

1. Lesson Plan
2. Prepare before going into the classroom

Seriously, if you are planning your lecture while doing it, you're doing a huge disservice to your students. That's not to say that open discourse and exploratory learning aren't good in the classroom, this can be great - let the discussion go where the students take it. But on the technology/course material side, I would be very concerned with adding material to my class at the last minute. What happens if that image/movie/website that you were counting on weren't there? What if your connectivity tanks when you wanted it? What if you get that huge phallus instead of the ICBM that you were looking for? I download anything that I want to use ahead of time.

If you are doing this last minute, to me that demonstrates that you are not taking your planning/prep seriously (or that you are lazy), and that you are putting your laziness ahead of the learning of your students. They (the students) deserve better than that.

Now that I've nibbled at the troll-bait, I think this is seriously off course - search by sketch, search by term, they all have the same possibilities to return content that someone out there may find offensive. So I fail to see how this is any different. That's the glory & risk of the Internet - use at your own risk.

Re:What about false positives?

[hobarrera]

Almost any keyword is bound to show at least two naked women in the first couple of pages. We tried with the oddest of keywords at my last job, and 9 out of 10 times, this was true... so there's nothing new here.

how about the copyright ?

[Anonymous Coward]

But do I automatically sign over my copyright to the sketch over to Microsoft ?

link to actual patent application?

[Miros]

The source article links to a page on the windows phone marketplace where it claims to be linking to details about the actual patent. Does anyone have a link to that?

Rtriever

[cyrano.mac]

Rtriever has been doing exactly the same thing since 2006. http://labs.systemone.at/retrievr/

Re:Rtriever

[Anonymous Coward]

Also, imgseek has been doing this for a long while now. http://www.imgseek.net/

Re:Rtriever

[Anonymous Coward]

...and the long-defunct Bulldozer Software's Diggit was doing the exact same thing in 2001. I can't link to them any more since they no longer exist, but here's a citation noting the sketch search function, and that it worked:

http://www.imaging-resource.com/NEWS/993764344.html

They went out of business just a few weeks later, it seems:

http://searchenginewatch.com/article/2067963/Image-Search-Faces-Renewed-Legal-Challenge

Prior Art

[Anonymous Coward]

http://www.tineye.com/

Sketch image, upload to tineye, done.

How about searching by dimensions?

[WillAdams]

I've been trying to find a plastic organize case to fit into an archery case I've just finished and am _not_ having any luck finding an organizer which is ~4" x 10.5" x 1.5"....

Prior art by IBM?

[Tal Cohen]

I remember IBM DB2 ads from the 90s about, well, sketch-based DB searches. IBM called it "Query by Image Content", or QBIC. It was easy to find one using Google Books -- in this case, CIO Magazine, Sept. 1, 1995: http://books.google.com/books?id=AAcAAAAAMBAJ&lpg=PA25&ots=GGDbllo74W&dq=ibm%20db2%20ad%20bottle%20%22perfect%22&pg=PA24#v=onepage&q&f=false

Re:Prior art by IBM?

[bill_mcgonigle]

Yeah, and I did a short presentation on it for a grad class in '94 (one of those classes which is everybody doing a short presentation on a new topic each week). Information Retrieval was a geeky/esoteric topic a the time.

  The idea certainly isn't new.

Patent will be invalid - It's been done before

[Etylowy]

Patent is invalid if there is prior art. Well, imgSeek has had that function since at least 2008 (can't find changelog for desktop version - it might have been there already in 2006).

Re:Patent will be invalid - It's been done before

[Anonymous Coward]

Every implementation of this that I've seen (and there are several) has been based on this algorithm: http://grail.cs.washington.edu/projects/query/

Re:Patent will be invalid - It's been done before

[Theaetetus]

Patent is invalid if there is prior art. Well, imgSeek has had that function since at least 2008 (can't find changelog for desktop version - it might have been there already in 2006).

Patent is invalid if there is prior art that teaches or suggests each and every element of the claims. Microsoft's claimed implementation may be very different from imgSeek's implementation, and as long as Microsoft isn't claiming something broad like "searching via sketching," then imgSeek's implementation may not anticipate the claims.

For example, if I was a Toyota engineer and invented the hybrid gas/electric transmission that they use, I could get a patent on that, and it wouldn't necessarily be anticipated by Panhard's 19th Century manual transmission. I couldn't claim "a transmission" generally, because that would be anticipated by any existing transmission, but I could start adding in additional new elements of a gas/electric hybrid transmission that aren't in existing transmissions.

Re:Patent will be invalid - It's been done before

[drinkypoo]

I couldn't claim "a transmission" generally, because that would be anticipated by any existing transmission, but I could start adding in additional new elements of a gas/electric hybrid transmission that aren't in existing transmissions.

sure, whatever wasn't in the 1906 Auto-mixte, or H. Piper's 1905 patent...

Re:Patent will be invalid - It's been done before

[Etylowy]

I did in fact read the claims in patent application and a lot of them look like the wavelet based method used by imgSeek. I seriously doubt much will be left off this patent in result.

Re:Patent will be invalid - It's been done before

[rastoboy29]

Yes, but...

Someone has to file an objection, which takes time and money.

Or they bring up prior art when being sued over the patent, which takes LOTS of time and money.

As usual, no justice without money in this country.

Re:Patent will be invalid - It's been done before

[Etylowy]

It's not that bad. ImgSeek author has been informed of the patent application filled and there's always http://www.softwarefreedom.org/

prior art

[Anonymous Coward]

http://detexify.kirelabs.org/classify.html

This reminds me of....

[djsmiley]

some weird application that appears while ago where you could draw a "shape" and name it - i.e. "Car" and then somewhere else on teh canvas you'd do "sun", "cloud" etc....

Once you'd done it it'd attempt to search and find images to make up your "master" image.

Probably not as simple as it seems

[Miros]

This is probably not as simple as Microsoft trying to patent something which has already been done before. You may want to check out some of the actual research they have done here: http://research.microsoft.com/en-us/projects/mindfinder/ I seriously doubt that they are not familiar with all of the prior art examples that have been brought up here, and they would not have spent the money on a patent if they did not think they had improved on the existing methods sufficiently for it to stand up to even basic scrutiny.

Re:Probably not as simple as it seems

[Miros]

Retrieving images to match with a hand-drawn sketch query is a highly desired feature, especially with the popularity of devices with touch screens. Although query-bysketch has been extensively studied since 1990s, it is still very challenging to build a real-time sketch-based image search engine on a large-scale database due to the lack of effective and efcient matching/indexing solutions. The explosive growth of web images and the phenomenal success of search techniques have encouraged us to revisit this problem and target at solving the problem of web-scale sketchbased image retrieval. In this work, a novel index structure and the corresponding raw contour-based matching algorithm are proposed to calculate the similarity between a sketch query and natural images, and make sketch-based image retrieval scalable to millions of images. The proposed solution simultaneously considers storage cost, retrieval accuracy, and efciency, based on which we have developed a real-time sketch-based image search engine by indexing more than 2 million images. Extensive experiments on various retrieval tasks (basic shape search, specic image search, and similar image search) show better accuracy and efciency than state-of-the-art methods

http://research.microsoft.com/pubs/149199/0630.pdf

Re:Probably not as simple as it seems

[Anonymous Coward]

You are one naive mofo.

Re:Probably not as simple as it seems

[oakgrove]

Assuming they get this patent granted, if and when it goes to court they're going to need something better than it "shows better accuracy and efficiency than state-of-the-art methods". If it were that easy patents would be worthless as you could make a few changes on anything and be home free. Samsung: "our slide to unlock has a shorter distance to travel and a bigger slide :)". Of course, maybe the insanity is to such a level that it really is that easy but if that were the case I'd use your logic and say "This is probably not as simple as $CORPORATION trying to patent something which has already been done before." but then I'd ask why isn't everyone doing it?

Re:Probably not as simple as it seems

[Miros]

My point is that they are patenting something which may not have been done before. The ends may be similar but the means are different, and the means are what matter for patents - as they should. The mechanism is specifically designed to maintain an incentive to come up with new and better ways of achieving the same purpose.

Re:Probably not as simple as it seems

[oakgrove]

Based on my observations of how patent law is applied in the realm of software these days I think you're confusing theory with reality. As things are now software patents are enforced on the end not the means. Of course legal arguments can get very nebulous almost immediately. A good lawyer could ask the court, "what is the 'end' of what we're talking about? The end is to get a search result. Therefore the patent is invalid as you are patenting means." And on and on. Just look at the shenanigans between Apple/MS and the various Android OEM's. They are not arguing 'means' by any stretch. One of the patents MS is wielding against Barnes & Noble is one where the text is displayed before the background in a web browser. Is that the means or the end? It could easily be argued both ways. I'd say it is the end as the algorithm itself is the means. But that isn't what MS is saying. They are saying if you implement text before image then pay up. I mean, wtf is that? When it's all said and done though, it's all up for interpretation so your opinion is valid and so is mine. My opinion is that this patent will not stand up to scrutiny despite your unquestioning faith in MS research as you mentioned in another post. They may be smart and they may have many resources but MS has lost in court before so blind faith is a bit misplaced on your part.

Re:Probably not as simple as it seems

[Miros]

I don't have any greater faith in Microsoft research than I have in any other organization (including tiny start-ups who also choose to file patents). One of the protections is litigation. Many silly patents that actually make it to that step end up defeated because greater amounts of resources are put into finding prior art that specifically conflicts with the method which was patented (and litigated over). I would not be shocked or troubled if someone immediately posted valid prior art, but it is a little troubling that few of the people contributing to this discussion understand what that actually means. Fight the battle on terms which actually give you a shot at victory!

Don't just post, Act

[Bucc5062]

Now all you guys claiming prior art, please, send it to the USPO so they can at least ignore it publically instead of what they usually do, ignore by default. Even better, contact the people in this article from Slate "Stamping out patent trolls" and they, Article One Partners,will take it from there. It is one thing for us to think the King has no clothes, or whisper it behind "closed doors" to friends, it is another to shout it out in public or to proper authorities so either action is taken, or you discover the public doesn't really care.

Common misunderstanding

[Theaetetus]

it used to be that they added "on the internet", now they add "on a mobile phone", looks like this one adds "with fingers on a mobile phone"

Common misunderstanding, but no. This came up because many patent applications have dependent claims that may say things like "2. The method of claim 1, wherein the network is the Internet" but that's not independently claiming the Internet... Or even claiming that novelty is due to 'the internet'. Rather, it's a doctrine called claim differentiation: dependent claims are subsets of the independent claims from which they depend from. Like, if Claim 1 is a large box on a Venn diagram, then Claim 2 would be a circle within that box. So, when Claim 2 says "wherein the network is the Internet," all that means is that the 'network' of Claim 1 must be broader, and include both the Internet and any other type of network.

But, for claim 2 to be patentable, claim 1 has to be patentable. And adding "on the internet" doesn't actually make claim 1 patentable... rather, claim 1 must already have enough new and nonobvious elements to be patentable on its own.

So, this misunderstanding about claim differentiation also meshed with a misunderstanding of patent titles and abstracts... They have no legal weight, the only important part is the claims, but people look at the title and say, for example, "'a network operating system'? Networks are known! Operating Systems are known! This is a bullshiat patent!" Add that to a claim saying "wherein the network is the Internet" and suddenly it looks as if someone was trying to patent an operating system... but on the internet! But no, that's simply not true at all.

In summary, there are no patents that claim "A method, comprising [X, Y, Z] on the Internet" where X, Y, and Z are all known and the only claimed novelty is 'on the internet'.

Prior art is really a red herring here

[Miros]

There are already dozens if not hundreds of patents in this same area.
https://www.google.com/search?q=sketch+image+search&btnG=Search+Patents&tbm=pts&tbo=1&hl=en

prior art?

[Suchetha]

Supposedly this patent is from 2010..

but according to this article "[Search by sketch] was mentioned in years 2007, 2010 and sometime in the late 90s"

if that is the case then there should be enough prior art to invalidate the patent for this concept

Useful?

[firewrought]

Everyone's commenting on whether this is patentable or not, but I'd be interested in discussing whether it is useful or not. Outside of some very narrow use cases (recognizing letters/symbols or maybe chemical structures?), will people actually find this useful? When we know the name of something, it seems easier to type that in (or speak it) instead of trying to sketch it. When we don't know the name... hmm, well it still seems easier to type in related words instead of trying to draw a sketch. :-\

Granted...

[Anonymous Coward]

Having the Etch-a-Sketch from Toy Story replace the search dog would be kinda neat.

Prior Art

[Anonymous Coward]

heh, I saw search by sketch at IBM's Almaden Research Center back in '96 or '97. Nothing new here

This is totally prior art

[efalk]

This patent application is rubbish. I implemented this myself nine years ago as a portfolio piece while I was interviewing at Google. And I did it based on a 1992 Siggraph paper.

Another example of prior art is called GEOLUS

[Anonymous Coward]

http://www.plm.automation.siemens.com/en_us/products/open/geolus/demos/index.shtml

Lots of info, online demo, FAQ, etc.

M$ Rips-off PhotoSketch ?

[sl4shd0rk]

http://www.zdnet.com/blog/weblife/photosketch-better-than-sliced-bread-photoshop/965

Re:M$ Rips-off PhotoSketch ?

[Anonymous Coward]

Hurr durr durrr... u r clevar with teh M$. hurr.

Prior art (more of it)

[Anonymous Coward]

Digikam does this already

Why are software patents allowed again?

[Trogre]

These silly clowns. I expect this article will quickly fill with examples of prior art. Here's another one:

Looks at digiKam.
Looks at patent application.
Looks back at digiKam.
Folds arms.

Re:Why are software patents allowed again?

[shutdown -p now]

How do you know that digiKam uses the same algorithm that's being patented here?

Re:Overaged business model

[Anonymous Coward]

Microsoft bases its business on selling software, not on patent license fees. It only takes a quick glance at the quarterly report to see this - almost all income is from Windows and Office sales.

Wasn't this on big bang theory?

[davecason]

They had a more nuanced idea, the ability to take a scientific formula scribbled somewhere and understand/translate it, but essentially it was the same. They basically substituted f of x for a bicycle.

Prior art

[efalk]

I wrote a search-by-sketch app in 2003 as a portfolio piece when I was interviewing with Google. And I based it on a 1992 Siggraph paper. So whatever Microsoft has patented, it had better be more specific than "search by sketch".