Microsoft To Launch Homegrown Search Engine

Mr. Christmas Lights writes "While Google is currently the king-of-the-hill in search engines, Microsoft continues to lag in market share and uses Yahoo's technology/results. But Cnet reports that they'll launch on Thursday their own homegrown search engine , although it appears this is mostly a face-lift (despite a year of development and $100 million investment). According to Bill Gates, they 'will introduce a homegrown web crawler and algorithmic search engine ... later this year,' which is almost certainly their tech preview (you can look at this now) -- but will that be ready for prime-time in less than two months?"

Comments

About time

[lightdarkness]

I've seen their cralwer (their new one I presume) around for at least a month, without any indication on where the results were being shown. At least I have another spider to add to my list of robots that steal my bandwidth.

Re:About time

[b374]

Is this their crawler?

Re:About time

[FireFury03]

The msnbot has been around for many months. I have seen many complaints about the amount of bandwidth it uses and I know many web masters (me included) have blocked it's access because of this so I dunno how useful the search results will be. I've seen reports of it sucking gigabytes off a site in a day, and then doing exactly the same again the next day, which is really quite serious for those people who have a reasonably small bandwidth limit on their web space.

For me it was sucking several gig a month off my site, and was obviously very badly coded since it was refetching the same pages over and over (cachable pages, non-cachable pages and 404's). So in the end I gave up and outright blocked the damned thing - yet another bit of shoddy MS code out to break the internet.. :(

Re:About time

[AndroidCat]

Blocked it with what? Is it playing nice with robots.txt and meta-tags, or did you have to get rough?

Re:About time

[FireFury03]

It pays attention to robots.txt directives (finally, a small amount of standards compliance!)

Re:About time

[AndroidCat]

It looks like it checks for meta tags too. (Useful when /robots.txt isn't convienent.) MSNBot page and other info. Also note that it only checks /robots.txt once a day, so policy changes might not take effect right away.

Re:About time

[flithm]

Actually it says it pays attention to robots.txt, however my test results show that it does not behave as expected. After noting the amount of bandwidth it was consuming, I created a robots.txt based from the examples on their web site, since I noticed it wasn't following the rules I had already specified that other crawlers obey nicely.

Unconvinced... here's some stats from my logs:

MSNBot hits: 10217+77 bandwidth: 441.67 MB
Googlebot hits: 116+90 bandwidth: 16.13 MB

This is after the modifications of the robots.txt file, and this is only for a 2 week period in October. MSN bot was drawing nearly 1 gigaBYTE of upstream per month, just from my lowly site! No thank you... I prompty did this:

iptables -A INPUT -p all -s 65.54.0.0/16 -j DROP

I encourage all other webmasters to do the same.

Re:About time

[FireFury03]

It payed attention for me with:

User-agent: msnbot
Disallow: /

iptables -A INPUT -p all -s 65.54.0.0/16 -j DROP

Or even better, if you have the TARPIT module:
iptables -A INPUT -p tcp -s 65.54.0.0/16 -j TARPIT :)

So

[jbartone]

Who's actually going to use this instead of Google?

Re:So

[MoonFog]

Since it'll probably end up being default start-up page in IE, lots.

Re:So

[Draveed]

You would be amazed. This week I discovered someone in my office who knew nothing about google.

Re:So

[zakezuke]

Since it'll probably end up being default start-up page in IE, lots.

Fortunately IE has enough in the way of exploits so the default start-up page gets hijacked often enough.

Re:So

[Technician]

Since it'll probably end up being default start-up page in IE, lots.

You mean the same people who use the default favorites? I looked at the default list once, then deleted it. It looked like a paid list from the yellow pages of the travel and media sections in the phone book.

Re:So

[Titusdot Groan]

Did your company fire them?

I've got a running gag with my team that if I can find the answer to their problem in 3 google searches I get their pay for that week. The number of dumb question I get is WAY down.

Re:So

[jimicus]

This week I discovered someone in my office who knew nothing about google.

What's google?

Re:So

[Anonymous Coward]

He's moved on to the infamous step 3.

PROFIT!

congrats, I'd mod you, but I don't have any mod points.

Seriously, who doesn't use google to search for a microsoft error message instead of microsoft's knowledgebase search.

And, what does everybody think about them being able to retrieve hits at pay for registration sites like experts-exchange?

Re:So

[Spoing]

1. I've got a running gag with my team that if I can find the answer to their problem in 3 google searches I get their pay for that week. The number of dumb question I get is WAY down.

Smart. Back when I worked as a Tech Support manager (pre-WWW, post 'net), the techs would constantly come to me with the same questions... I'd fire back to them "did you find anything in IZE?" (a simple but useful outline database back then). If they said no, I'd look...and about 1/3 of the time found the answer there.

The only difference between then and now is that it used to be that if the search came up empty, I would tell them to write up a note on what they learn so that the next person searching would find something. Now, with Google, that is rarely necessary.

Search for Windows.....

[alistair]

No Results Found

Needs some fine tuning before it's ready for the prime time, me thinks.

Re:Search for Windows.....

[jai0]

Just searched for "where do you want to go today".. the first result was "Where do you want to go birding in Africa today?"!

Microsoft's problem

[ArbiterOne]

Microsoft is branching out too much. Without ripping off Google, I don't really see how they can pull this off. In order to reverse the current trend in market share, they'd have to have a better algorithm than Google, a massive ad campaign, and the popular opinion on their side. Oh, and start giving things away for free (Google: Blogger, Picasa, etc.)

Re:Microsoft's problem

[KiloByte]

... or ignore quality and just put it as the default (and only) search engine for a certain well-known OS...

Re:Microsoft's problem

[geg81]

It's not a "rip off" when multiple companies offer similar products, it is competition. Competition is a good thing: it lowers prices and improves products. And while Google has some good technology, they company didn't come into being in a vacuum and they don't own the idea of citation and reference analysis, either legally or scientifically.

"Lower what prices?" you might ask. Well, Google isn't a charity. I expect Microsoft will compete with them on advertising rates and whereever else Google makes money.

The only thing that may be considered unfair about that is that Microsoft can afford to make losses for many years on this before driving Google out of business. But that's a problem everybody faces when Microsoft enters a new market.

Re:Microsoft's problem

[Anonymous Coward]

The answer to the question "how does Microsoft intend to destroy Goggle" has been well known for a while. They will use the same old anti competitive tactic they used for Netscape and imbed the search capability direct into Windows probably via Windows Explorer. By upping the convenience, they think, the fact that the search is inferior probably doesn't matter much. It worked very well for IE, Office and the Windows platform itself. It hasn't worked so well for them recently with neither the Xbox or .NET or whatever their iPod killer is called all looking rather lame. I don't think it will work here eihter. After all you already can search MSN from Windows Explorer and nobody bothers on account of the fact that the search results are so useless.

The weird thing about all of this, at least to me, is that MS had a really good opportunity to get Windows into the data centre between about 1999-2002 when Linux wasn't quite there and customers were looking for a cheaper alternative than Solaris/AIX etc. They missed it because they were so busy trying to be Sony and Sun and are now compounding the problem trying to be Google. In the mean time Linux has plugged the hole and is firmly established as the low cost data centre OS of choice.

The browser war was won when Microsoft removed any value from PC based browsers (you couldn't sell them anymore since MS gave their one away for free) - but that means that you end up with something that costs you money to maintain and support for no good reason, so you don't bother, and end up with something that looks as tiered as Internet Explorer (or a very tired thing, make up your own simile). Whilst I'm sure they could make a lot of money selling advertising space on Windows (the base Goggle profit stream) I can't help feeling that customers might not like it very much. And, of course, you don't need an expensive search engine R&D project to do this, you could just say randomly change everybody's wallpaper to an advert every 5 minutes or something.

Re:Microsoft's problem

[0x461FAB0BD7D2]

Let us not forget that Google is foraying into Microsoft's domain.

Gmail and Google Desktop Search are not accidents. Hotmail is extremely popular, and with Google Desktop Search, Google has learned that people will install replacements to Microsoft's own built-in services if they are better, and if they are branded.

And the browser war was never won. That was just the browser battle. By removing value from browsers, all Microsoft did was reduce the incentive for it to update its own browser after it gained near-ubiquity. Open-source has no problem with zero value, and this is why Mozilla has no problems working tirelessly on Firefox and Seamonkey. The browser war has just begun.

Re:Microsoft's problem

[irc.goatse.cx troll]

Yeah! And whats with AOL using AIM as the default messaging service? And aol's mail for the default email? Its so monopolistic! They should license YIM and embed it into aol to be fair to their competition.

Seriously, its their browser, why shouldnt they make the homepage their search? Moz's default is a moz branded google, how is this different?

Re:Microsoft's problem

[jsebrech]

Microsoft may have won the browser war against netscape, but netscape lost it. Had netscape not put out such crappy products in the 4.x timeframe, they wouldn't have dropped below the 20 percent marketshare treshold, and people would have designed sites to work in both.

For microsoft to win the search engine war, google would have to lose it, and that's not very likely.

Prediction for most popular queries on day 1...

[seanellis]

"mozilla firefox" download
google

Algorithmic search engine? WTF?

[jagripino]

Don't *all* search engines have to have, hmm, some kind of algorithm in them?

Marketing speak confuses me! Please stop!

this article is oooooold

[evil_one666]

This article is from june 30th

Comment removed

[account_deleted]

Comment removed based on user account deletion

Doubtful this will take any ground.

[Ambient_Developer]

I say it time and time again.. Microsoft is not a company of innovation (besides user interfaces), they are a company that aquires other companies. It is doubtful that a home-grown engine will beat the likes of google.. Especially being so late in the game, not only that what good will a face lift do? Google is already one of the easiest things out there, how can Microsoft make search even easier? THAT is the 100 million dollar question!

Tech Preview?

[beders]

No results for my name or a random word. More worryingly no result for "porn". Got a long way to go chaps, $100 million seems a little steep for a input box linked to an error page...

Search engine wars

[dauthur]

It seems that Microsoft might just be trying to cut in on the business that Google, Yahoo, AskJeeves and all those other engines are making. I don't know what kind of a fool would use a Microsoft search engine anyways, the index would have to be built from scratch, instead of the years of data that Google and Yahoo have accrued.

Re:Search engine wars

[northcat]

It seems that Microsoft might just be trying to cut in on the business that Google, Yahoo, AskJeeves and all those other engines are making.

No, Bill Gates is just trying to impress his wife. "Look at my search engine honey, search for 'bill gates nude'. I bet you have never seen THAT before!"

Dutch?

[tompercival]

Fantastic... dutch search results - just what I was after too.

Maybe..

[StormyWeather]

their 100m would have been better spent to stop the bleeding they are about to recieve at the hands of Mozilla before folks realize they can add specialized search engines in the search toolbar instead of just google. Once folks find out how wonderful this ability is I think it will even slap Google upside the head a bit. For real research I have found this an invaluable as using google tends to give me search results that are too broad, often from sources that are more difficult to document.

The best part of all...

[Anonymous Coward]

is that when I searched for "Windows XP crack" it found a great page about an underground piracy ring called the SPA, they even gave me a number to call 1-800-388-PIR8. Thanks Microsoft!

Similarities

[ArbiterOne]

Check out the similarities:
http://search.yahoo.com/
http://search.msn.com/
http://www.google.com/

Best search engine

[Libor Vanek]

Just try to search "best search engine" and enjoy what comes out:
http://www.search.msn.com/results.aspx?FORM=SRCHWB &q=best%20search%20engine

Re:Best search engine

[jimicus]

That looks remarkably like a Google results page, in terms of structure rather than content.

Example:

http://www.search.msn.com/results.aspx?FORM=SRCHWB &q=best%20search%20engine
http://www.google.com/search?hl=en&ie=ISO-8859-1&q =best+search+engine&btnG=Google+Search

Surely Microsoft haven't chosen to rip-off their design based on the market leader?

lack of trust

[vinsci]

Microsofts search engine lacks the most important feature: trusted results.

In the past, it has been shown that Microsoft blocks search results that are contrary to its own business interests.

W3C page for HTML nowhere in the first 20 results

[northcat]

I searched for 'html' in their "preview" search engine and the w3c page for HTML was nowhere in the first 20 results. I didn't look beyond 20 results. The w3c page should have been in at least the first 20 results. Is this search engine really that good?

If their past strategy is any guide...

[DonDiablo]

... after getting enough users from directing all windows searches to their engine, they will create "search extensions" for all the sites hosted in a Microsoft server, and "special html/jsp search tags" for sites developed using their tools, which will produce a better placement on their search results.

Almost there...

[Luigi30]

They have 99 million matches for Linux. Google has 162 million.

Looks just like google

[ChrisMDP]

Sadly, Microsoft are up to their old tricks again...

1. Release a sub-standard product which looks like the better original.
2. Rely on their massive brand penetration to increase market share.
3. Throw enough cash at something to make it worthwhile.

It irritates me that they do this - it slows the rate of internet progress down by duplicating other peoples ideas. Why not invest in google and build on what someone else has done, rather than trying to completely monopolise all areas of the internet?

wtf?

[Mr_Silver]

although it appears this is mostly a face-lift (despite a year of development and $100 million investment)

I thought it was only marketing that didn't understand that just because it looks the same, doesn't necessarily mean you've done nothing under the hood.

THE bot?

[knipknap]

I wonder whether that's the bot that has been scanning my website for three days by attempting to "crawl" through all session ids and causing more then 1 GByte of traffic.

"msnbot/0.11 (+http://search.msn.com/msnbot.htm)"

It was only stoppable by blocking the IP. (robots.txt was only read once before it started) Great, smart bot, really.

3 bad results.

[caluml]

Orange. No results for Orange, the mobile phone company.
Linux. No pointers to linux.org.
Google. Returns the Dutch/Belgian version of the page. Why?

Re:3 bad results.

[BuilderBob]

Orange. No results for Orange, the mobile phone company.
Linux. No pointers to linux.org.
Google. Returns the Dutch/Belgian version of the page. Why?

These are no longer true. I know it used to do this but now ...

'Orange' returns Orange.co.uk.
'Linux' returns linux.org
'google' returns google.com
'microsoft sucks' returns fuckmicrosoft.com
'abu graib' returns the photographs of inside the prison.
'lindows' returns lindows.com

This is from Firefox 0.8 on Redhat Linux.

BB

Re:3 bad results.

[caluml]

I ran those 3 searches a minute before I posted them. If they used to be true, they still are. I am in the UK. Maybe it's messed up over here?

Country-specific redirection borked

[OblongPlatypus]

Clicking on the tech preview link in the blurb redirects me to a French version of the page, at techpreview.search.msn.fr. The problem, you ask? I'm in Spain.

Minor detail, sure, but add it to the shaky performance of the actual search, and this product would seem to require more than a couple of months of fine-tuning.

Re:Do they have to ?

[ssj_195]

Yes, this is fully in accordance with Microsoft's mindset - whenever a new pie appears in the horizon (a floating pie lol wtf?!) Microsoft cannot stand not having a piece of it. To be honest, I think the fact that they cannot have all of it rankles them. I hate to sound like a typical anti-M$ "slashbot", but whenever I think of adjectives for Microsoft's whole corporate mindset, the only ones I can think of are "broken" and "diseased".

Filtering out queries

[ttys00]

Will they be filtering out queries with this engine as well (eg. xfree86 being filtered as discussed here a while back)?

Of course. And while they do that, I won't be using it.

MSN Bombs

[Nuskrad]

My three favourite googlebombs still work! Also interesting are Best operating system and Worst operating System

This isn't insightful

[SimianOverlord]

Google have built up a market by offering ad views alongside their search engine, thus making money. It isn't about the search, it's about that market, and Microsoft moving into new markets is what it is all about.

They do not need a better algorithm than Google (which is becoming increasingly gamed by shady companies and not as good as it was anyway), they just need something "good enough", like their OS is "good enough".

As for reversing the trend, Microsoft have 1) leverage in the form of their existing OS userbase (as others have mentioned, using MS search as default), 2) Massive cash surplus 3)Brand recognition. They do not have to give things away for free. They have to fight against a competitor with a larger market share, something they have done in the past quite successfully. Do not confuse the Slashdot echochamber with objective reality.

Search Results

[chia_monkey]

Oh my. I can just see the search results now:

Search entry: "Antarctic Penguin"

Search Result: A paperclip pops up on your screen and says "It appears you are searching for a penguin. Did you know Microsoft servers are cheaper to run than Linux? Would you like to buy one now?"

Re:Netcraft says the hosting servers run on Linux

[alib001]

Mostly wrong.

...the DNS directs us to a server operated by Akamai... Akamai's http caching servers run Linux, and so we report Linux as the operating system. However Akamai also forwards the http Server: header from the original server as part of the cached content, and so we report "Microsoft-IIS/6.0" as the web server.

No faith in their own product

[BlackHawk-666]

I notice they're using Akamai instead of a cluster of cheap Windows servers. Nice of them to recommend to everyone else to use their technology, but then not trust it for their own stuff.

subversion:~# telnet techpreview.search.msn.com 80
Trying 213.253.9.73...
Connected to a213-253-9-73.deploy.akamaitechnologies.net.
Esca pe character is '^]'.
HEAD /

HTTP/1.0 400 Bad Request
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 161
Expires: Wed, 10 Nov 2004 12:56:31 GMT
Date: Wed, 10 Nov 2004 12:56:31 GMT
Connection: close

Nice and clean

[jmcmunn]

I can't say that this will be my first stop when searching (Google will be until they stop being the best) but often times, if my result is not in the first few pages of Google, rather than figure out the exact phrase I need to search for to find the site I am looking for, I just hit a few other engines to see if my original phrase does the trick.

I can see how this new MS search page would become stop number 2, in front of Yahoo as long as they keep it clean and light like Google is. Then I'll move along to Yahoo of Lycos or wherever.

So yeah, I think this is a good improvement for my general searching needs, but it is going to take something amazing to replace Google as my number one choice. It's sort of a brand loyalty at this point.

Comment removed

[account_deleted]

Comment removed based on user account deletion

haha

[deander2]

at the bottom of my search:

"Results 1 - 10 of about 8 containing deander2"

first off: "about 8"?!?
second, WTF? can't they check for 10 results?

It's hardly brilliant...

[nick8325]

Search for 'Search' on Google:
search.com
AltaVista
Yahoo
Excite
All TheWeb
Lycos ...

Search for 'Search' on new MSN:
Vault: the most trusted name in career information
Destiny Group
CareerBuilder
Realtor.com
Lycos People Search

So, the fifth link on MSN is nearly - but not quite - relevant.

Incidentally, Google doesn't list itself until 20th when you search for "search" on it. Which is interesting... maybe it's because of its minimalistic website which doesn't mention searching very much.

how did you find such a person?

[jxyama]

>You would be amazed. This week I discovered someone in my office who knew nothing about google.

i imagine you had to use google to find such a rare person? :P

no NEAR operator

[Sai Babu]

When will these guys take a cue from altavista.com and incorporate the NEAR operator. For example Pussy NEAR cat returns a much higher density of feline related URLs.

More tinfoil hat fodder.

[saur2004]

Anyone remember what the infamous Registration Wizard did when upgrading from Windows 3.1 to Windows 95?

I wouldnt put it past them to code a phone home feature for clickthroughs for this.

Even if they get caught doing it, I can just hear the argument now, "Heck Yahoo, AltaVista and others collect aggregate data. Just look at what the URL becomes when you mouse over the link from thier lists."

I for one, love that google STILL does not do that.