Slashdot Mirror
Microsoft Lost Search War By Ignoring the Long Tail
Art3x writes "When developing search engine technology, Microsoft focused on returning good results for popular queries but ignored the minor ones. 'It turned out the long tail was much more important,' said Bing's Yusuf Mehdi. 'One-third of queries that show up on Bing, it's the first time we've ever seen that query.' Yet the long tail is what makes most of Google's money. Microsoft is so far behind now that they won't crush Google, but they hope to live side by side, with Bing specializing in transactions like plane tickets, said Bing Director Stefan Weitz."
I worked on MSN Search (later "Live Search") so I can answer a few of these for you: 1) There was very little collaboration with the MSN teams. MSN is generally despised at Microsoft, and to get people to come to Search we had to reassure them that it wasn't "really" part of MSN. For their part, the MSN people seemed to try really hard to live up to their "it can't be done" reputation. For example, the MSN team controlled the UI, and even though a top customer complaint was that there wasn't enough space for users to type their queries, no force in the Universe was powerful enough to make the MSN guys widen it. (Their design rules required it be usable by people whose display was a TV set.) 2) Yeah, the MSN data was worthless. First, there wasn't that much of it; rather than saving the raw data, they had a process for computing digests of it, and that's all we could get. Also, that digest process was full of bugs. For example, for years it told us the top queries were "google," "internet explorer" and "yahoo"; it was obvious this was a bug, but our management couldn't get the MSN team to do anything about it. 3) As Yusuf suggests in his article, the cumuative Search and Click data is NOT what you need to produce a good search engine. One of the most frustrating things about working on Search at Microsoft was Management's obsession with head queries. They had several articles of faith that didn't accord with reality, but this was one of the worst. Good news for Microsoft if they've finally figured this out. Of course, almost all the people responsible for the original mess are long gone now. 4) The Google-worship was nauseating. We wasted all kinds of effort trying to duplicate features that obviously didn't work even for Google (news being an obvious example) whereas new features that might have been helpful consistently got killed with "Google doesn't do that." In many cases, this argument was used for technologies where no one had any reasonable clue what Google actually did. --Greg
Google has decided that they WILL NOT censor the web for 1/4 of the world's population,
Well, to be precise, Google went along with the censorship until they caught the Red Dynasty fucking with their servers, and decided that they'd had enough.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
Remember Cuil? They were originally talking about the "long tail"; they wanted to have a bigger index than Google. Cuil is mostly ex-Google people, and they thought they could re-do Google at lower cost.
Didn't help Cuil.
There's ongoing effort in search engine development. Unless you pay close attention, though, it's invisible. A few years ago, around 2007, Yahoo introduced about fifty specialized search sub-engines. These understood weather, stocks, sports, celebrities, movies, and similar popular search topics. They focused on areas that have a strong structure, and need a lookup engine that understands that structure. For about six months, Yahoo was way ahead of Google on such searches.
Didn't help Yahoo. Google implemented something similar and caught up. Now everybody does that.
It's not clear that the Twitter search is a win. Bing announced they were going to do Twitter and Facebook searches, and a day later, Google announced they'd do that too. Google implemented Twitter search, and apparently Bing didn't. Twitter search just seems to clutter up Google results.
In the last year, Google has become much more aggressive about interpreting queries. Google tries hard to infer from the query words what the user is really looking for. This tends to work for popular queries (since it's based on statistics from other queries) and doesn't work too well for unusual queries. For hard queries, you need to use explicit operators ('+' and '"') with Google more than you did a year ago.
The big search engines are still doing badly at de-rating sites which are basically link farms. When you're searching for a product, and you get a hit that's just some site with ad links to other sites, that's a fail. Search for auto parts, and you're likely to get "parts.com", "thepartsbin.com" and "who-sells-it.com", which are just "portals". They don't even return pages that are actually about the part in question. ("thepartsbin.com" pages are all essentially the same, except for keywords inserted for SEO purposes.) Search engines need to look at the business behind the web site. If a business has a million commercial-looking web pages, and a total business volume of a few million dollars, they're probably bogus. That's a part of the "long tail" you don't need to visit.
An interested person might start here: http://google-opensource.blogspot.com/
This is interesting reading: http://socghop.appspot.com/
Chrome and/or Chromium browser: http://en.wikipedia.org/wiki/Google_Chrome
Whatever your interest is in open source, try googling it. Not everything in the labs is open source, but some is - check that out: http://www.googlelabs.com/
Want code to play with? You'll get more from Google than you'll EVER get from Microsoft. Maybe I exxagerated with the word "most" - but they have given away a lot of stuff, and they help with a lot more. One of the things you'll see when you click the links above is Gnome. They contribute, but, of course, Gnome doesn't belong to Google - that capital "g" is just coincidental.
So, go look around.
"Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br