MSN Search Engine Favors IIS

Scud writes "It appears that if you want to rise up in the rankings over at the MSN search engine you would do well to host your page on IIS. Ivor Hewitt has done a study and it appears that by using IIS, you are likely to increase your odds of a higher listing by several percent."

Comments

FTFA

[Reignking]

So what's going on? I have no idea, I doubt it's all a big conspiracy... but some possible explanations spring to mind: Perhaps the MSN search has simply been coded by developers used to talking to IIS machines and so it just does that job better? Perhaps the MSN spider is taking advantage of some specific IIS features to provide enhanced indexing?

In other words, there are some explanations out there other than "MS is biased and there's a conspiracy and they are trying to take over the world"...

Re:FTFA

[misleb]

Well, MS *is* biased. That much should be obvious. There is really no way around that one. Is there a conspiracy? Why not? Would it be all that astounding? I mean, all it takes is for one exec to say, "ya know, I think we ought to favor IIS because IIS (insert some lame justification here)."

-matthew

Re:FTFA

[coolGuyZak]

"ya know, I think we ought to favor IIS because IIS (insert some lame justification here)."
Maybe... "ya know, I think we ought to favor IIS because IIS is our product"?

Re:FTFA

[WoodstockJeff]

Is there a conspiracy? Why not?

No, there is no conspiracy. There may be a company policy , but conspiracies require more than one party. MSN is part of Microsoft, so this isn't the case.

Now, if Yahoo or Google were doing it, too, that could be a conspiracy.

Re:FTFA

[Otter]

The different results for "Linux" were noted here the day the new MSN beta launched and explained a minute or two later -- the MSN ranking algorithm weights the domain name much more heavily than Google does, which is why MSN favors Linux.com and Google goes for Red Hat.

My guess is that the server differences are somehow correlated with something weighted differently in their rankings. As someone else noted, the real test would be switching the server on which a site is hosted and seeing if its rank changes.

Or if that's too much work, one could also argue that Google ranks IIS down!

Re:FTFA

[badfish99]

In other words, the other explanation is "Microsoft are yet again using undocumented features of their own software to give themselves an advantage over their competitors"

Re:FTFA

[aneurysm36]

How does it benefit them to favor IIS if they arent advertising that fact?

Re:FTFA

[coolGuyZak]

It makes the hit counts on their servers go up, which is something they can use to fuel the argument that IIS is better than Apache/thttpd/etc.

Also, if you are a "decision maker" who uses MSN Search, you'll see IIS everywhere. It will influence your opinion: you'll think it is more ubiquitous than it is.

Re:FTFA

[.com b4 .storm]

Or if that's too much work, one could also argue that Google ranks IIS down!

The problem with that is that Google (for now?) has zip, zilch, nada, and nil to gain directly by ranking any given server up or down. Google does not distribute or sell web servers, nor have any direct stock in any particular server and its success or failure. Microsoft, on the other hand, makes a web server - and if their search engine adjusts ranking in any way based on the presence or absence of that web server, that is rather fishy.

One could argue, of course, that Google has a stake in certain web servers (i.e. ones not controlled by companies like Microsoft) by virtue of them keeping the WWW open, and thus providing a viable arena for Google's search technology and money-making adverts. That's a bit different, though, and I'm not aware of any indication that Google favors open source web servers (or whatever) in their results.

Re:FTFA

[caluml]

Why not just modify the Greeting banner in Apache to read as if it was IIS.

Yep, I'm sure that's what they want. So suddenly on Netcraft, it looks like IIS is gaining huge numbers at the expense of Apache.
"See - webmasters are taking advantage of our superior blah, foo, and duh."

it's foolish...

[hruske]

... to think ms wouldn't use all it has. Obviously it hasn't yet learned from google, that being evil is bad. And bad guys get punished.

Re:it's foolish...

[MoonFog]

In what world? Seing how MS has been allowed to do basically whatever they want in the US .. why would they suddenly change their ways? The bad guys clearly do not get punished.

I laugh at Microsoft.

[the_mutha]

... and they still think they can beat Google to the game. When are they going to realize that what made Google so successfull was the fact that is has been so unbiased in all ways imaginable, including not accepting payments to get higher rankings.

Google makes money by prioritising quality. Microsoft makes money by prioritising money.

Go figure.

Re:I laugh at Microsoft.

[mattmentecky]

Google makes money by prioritising quality. Microsoft makes money by prioritising money.

As a publicly traded company, Google's priority is delivering returns to investors, just like Microsoft's, don't fool yourself for a second into believing that not to be true. What I think you are trying to say is how Google and Microsoft get to that common priority.

Re:I laugh at Microsoft.

[Anonymous Coward]

You're joking, right? Google search results have been getting worse and worse. Have you not noticed just how many of their top results are nothing more than other search engines or ebay affiliates?

Re:I laugh at Microsoft.

[dustmite]

Amazing how quickly people seem to forget though, and how nobody actually learns from history, even recent history. Microsoft can (illegally) crush any competitor they like by simply tying a product very heavily to Windows, and search engines are no exception. Yet people here think Google is somehow immune to this because they care about quality. When a monopoly in one market is illegally abused to force domination in another market, quality doesn't even enter into the equation. What a naive view. MS have already set their sights on Google; they're dead in the water. And nobody here even seems to know what "antitrust", "product tying", "Sherman act" etc. mean anymore - terms that everyone seemed to know about during the antitrust trial regarding Netscape.

Re:I laugh at Microsoft.

[bergeron76]

Yes, and in the day of Enron and MCI Worldcom, could it be possible that Investors see value in being "responsible"?

Investors aren't stupid either, they see the trend that people trust Google and they're putting their money on it.

Re:I laugh at Microsoft.

[mattmentecky]

Yes exactly.

I am not saying Google is some animalistic beast that fights for money legally or illegally, simply that their priority is delivering returns to investors. That interpertation can be cut many different ways, including yours, if their is value in being responsible, and there is, then it falls under the main priority of a public company.

Think about it: in the long run, if Enron/Worldcom/MCI whose stocks if still existing, are performing horrible, wouldnt it have been cost effective for them in the long run to be 'responsible' instead of illegal and profiting in the short run?

Re:I laugh at Microsoft.

[killjoe]

All companies exist to make money. Google has the phrase "don't be evil" in their vision statement. I presume they put that there to remind all their employees that it's possible to make money and not have to resort to evil tactics.

Re:Silly, silly boys (and girls)

[Binestar]

IMO the true test would be to take his site which is hosted on Apache, move it to being hosted on IIS and watch and see if his ranking goes up or down after the next time it is indexed.

wow

[SComps]

Apparently it's a very slow news day. In the interests of being remotely on topic.. (yes my karma will suffer dearly for this)

Why would this be any real surprise to anyone? MSN being MS is obviously going to give preferential treatment to their own products. This may be by design or strictly because IIS servers respond to some proprietary (yes I said it) requests that other servers won't.

I don't necessarily see it as an evil thing, but it's not entirely philanthropic either.

Re:Absolutely

[zero_offset]

MSN has your example listed as #2 at the moment.
It comes out #6 on AskJeeves and Teoma, and #5 on Gigablast.

My god, CONSPIRACY!

In fact, the only place I could find where you come out #1 is on AOL.

Re:Why would i want to do that?

[dAzED1]

1. Why would i want to improve my rankings at an "unknown", "unworthy", "bug-ridden", "slow" search engine?

They're not suggesting you should. They're saying the rankings in MSN search are affected. And guess what - millions of people use MSN search. Therefore, it will have a significant economic impact.

2. Why would i want to install IIS, when i have a better alternative Mac OS X?

Because you're a business that wants to maximize your profit-making potential. Most places with web sites care about income flow. If you don't have such pages, then don't worry about it.

Re:IIS imperial domination

[secolactico]

MSN Search should be banned for being dishonest.

Banned from where and by whom?

MSN search can do whatever they like. I don't know anybody who actually uses it. Even non-tech oriented people that use IE (against recommendations) set their startup page to something else. Google, mostly, but also "My Yahoo" and their webmail or portal of preference.

"Conspiracy"

[Anonymous Coward]

conspiracies require more than one party

So you're saying that "government conspiracy" should actually be "government policy"? Interesting.

Re:"Conspiracy"

[morcego]

When people mention a "goverment conspiracy", it is related to several agencies, or at least should be.

The IRS is not conspiring to get all your money. It is just company policy. :)

Re:"Conspiracy"

[m50d]

A "government conspiracy" normally involves the government conspiring with oil companies, local steel or similar.

You don't need to assume malice.

[argent]

Remember, the guys working on the MSN search engine certainly use IIS to host their intranet sites, and whatever internal webservers they use to test against are probably IIS as well, at least in the most cases. They are likely to consider bogus results for their own sites (both internal and external) more critical... that's not malice, that's just human nature. Even if they consciously work against that, they're more likely to notice problems there first.

And search engine tweaking is more an art than a science. It's an evolutionary process, with feedback loops and strange attractors. So if there's any difference in the behaviour or design of Apache or IIS that would be visible to a search engine, it's likely to lead to a slight bias in favor of the server software that the servers they pay more attention to run.

Do we see a significant effect? Is it just chance?

[davids-world.com]

Sounds plausible at first, but if you look at his figures you see that the author didn't run a lot of queries, and that the difference between the google/Netcraft and the MSN ratios for Apache vs. IIS is not huge (68:20 vs 64:26).

Leads me to think: is it significant? That is, can we exclude (to a reasonable certainty, that is, p>0.95) the possibility that the effect seen cannot be attributed to chance or some other criterion MSN uses?

Ivor says at some point The initial set of words indeed showed a significant difference between the results from Google and the results from the Beta MSN search..

But what does he mean? I would be interested in what kind of significance test was applied, what the exact results were. Just looking at the ratio of percentages doesn't tell me enough... One should go back at the original data (seems provided, good) and check if the effect is actually trustworthy or just, in Ivor's words, "Odd. Pure coincidence perhaps."

Before seeing some analysis of significance, I don't believe anything...

News?

[WilyCoder]

Sadly, this fails to surprise me. This news story doesn't sound too different from the one posted yesterday about Kerry-contributors being banned from some engineer gathering.... Why should a search engine give a flying fuck about what http server a box is running?

No, you THINK about TFA

[FunWithHeadlines]

"Perhaps the MSN spider is taking advantage of some specific IIS features to provide enhanced indexing?"

In other words, there are some explanations out there other than "MS is biased and there's a conspiracy and they are trying to take over the world"... "

It's called plausible deniability. "Why, no, we had no idea this would happen. You say it's an interaction with an IIS feature that causes this to happen? Heavens to Betsy, we never thought of that."

Microsoft people aren't stupid, and they ARE trying to take over the computer world, or haven't you been paying attention to what they say and what they have done? The engineers that built MSN Search would certainly be aware of any interaction that fits with IIS features to provide enchanced indexing. They would have been all over it from the beginning. And a side-effect means that IIS sites come out higher? Great! It's a feature that benefits us, they would think.

Of course MS is biased. Of course they would have noticed this. Of course they like it.

Six Sigma Geek Says:

[jacks0n]

Where's the paired t test?

Re:Silly, silly boys (and girls)

[b1t r0t]

Actually, what would be even better is to configure his web server to report itself as IIS in the headers it returns. That's the only real way to know what a web server is running, unless you want to parse server-created error messages, or exploit vulnerabilities in the server itself.

selection effect

[coyote-san]

Apache freely issues advisories and patches. It will issue an advisory if even one user faces a minor risk.

Microsoft (and nearly all other proprietary software companies) tries to hide problems to protect their perception in the marketplace. You usually only see advisories for major problems that will become public knowledge anyway, and numerous other fixes are piggybacked on the big ones.

But beyond that advisories don't really address the quality of a product. They're one metric, but nothing more.

Re:Nobody uses MSN. This is a perfect example of w

[DoktorTomoe]

Then, in 2007, came Longhorn, with integrated web search using not Google, but MSN. Joe Sixpack didn't care, but MSN was so damn convinient he forgot about Google - effectively forcing Google Inc. with its costly development department out of business. Later - oh surprise - all results you got for "Linux" on MSN were advisories to ditch it for Windows. He who controlled the search result, controlled the industry. (Maybe I should put some fake Frontpage-Meta-Header to my webpages to increase Rankings on MSN ... just to be sure)

Re:IE bias too

[sycodon]

Dog Bytes Man.

The only thing more pathetic than M.S. doing this kind of thing is the "news media" acting surprised over it.

Re:IE bias too

[birge]

This is pretty ridiculous. There is no way to account for the million other variables that could confound this, such as:

1) Maybe it is Google discriminating *against* IIS, not Microsoft for.

2) Maybe there is a correlation between things like website type (i.e. corporate vs. .org) and use of IIS, and MSN is discriminating for or against that other, correlated variable.

IIS is already irrelevant.

[SgtChaireBourne]

IIS is already irrelevant.

In major countries like Germany, IIS is already down to around 3% of the server market. Even world wide, most people have the sense to run Apache. You can look at the percentages, but every time an IIS farm is rolled out, shortly thereafter, they wise up and drop it for Apache or any other product actually suited for being connected to the network.

Frankly, I'm not sure why this article even made it to Slashdot. Is slashdot or OSDN participating in this year's marketing tsunami by doing product placement ads? Please let's go a week without MS articles, there's enough shilling going on in the discussion without them.

Re:Conspiracies of one

[StillNeedMoreCoffee]

"BTW, this isn't a "conspiracy" in the legal sense since it's not a crime to give preferential service on the basis of web server. It's sleazy unless it's fully disclosed, but it's not a crime unless they actually sell the search engine as an unbiased tool."

This would not be true if it having a higher status in the search engine meant more hits which meant more business. It could be argued that that was a discriminatory practice that could be quantified as a loss in dollars. Or possibly a monopolistic practice, and we know there has never been any suggestion that MS had any issues with those types of descriminator practices. Pattern of behaivour comes to mind.

Re:Just have your Apache report that it is IIS !

[bogado]

True what we need is :

if (UA == MSN_SPIDER) THEN
REPORT JUST LIKE IIS
ELSE
REPORT DEFAULT SERVER GREETING

Re:why would u use MSN to search ?

[sk8king]

Unfortunately most people still think that to get to a page on the web they use MSN search. That default home page for explorer is very sneaky. Customers that I've spoken with don't even know what the address bar is for. They hit the 'home' button and type the URL in the search. Some things that appear obvious to computer users are apparently not obvious to everyone else.

Re:A possible explanation

[davids-world.com]

Cool, you're giving a couple of good reasons that illustrate that even a statistically significant correlation does not imply a certain direct causality :-)

Re:Silly, silly boys (and girls)

[robertjw]

TCP signatures also vary from one TCP stack to the next, allowing you to identify a Linux box and a Windows box

I'm sure that's true, but do you think their spider is checking the TCP stack on every connection. It's probably just looking at the header the server sends like the grandparent stated. Why look at anything else, until of course everyone hacks their Apache servers to say they are IIS...

Re:Just have your Apache report that it is IIS !

[sremick]

Which makes Microsoft incredibly happy, since such info is used all the time to compound web statistics.

Increasing marketshare statistics increases your marketshare further. What could be nicer that having your competitors fudge the numbers in your favor at the beginning to give you a head start?

This is why I'm against browser user-agent spoofing as well. UAs are like votes. Stand up, be counted, and leave your UA alone so that the stats work in YOUR favor, and not against you.

Re:IE bias too

[kperson]

"...mostly irrelevant blogs and pages full of adverts..."

Nice concise description of the Internet!

Re:Gee...

[spacecowboy420]

Along your line of hardware and services, the oems work for them [thus they do not need their own hardware division], and they do [or have a "partner" that does] all of the consulting/services you could ask for - with one caveat, you must buy their products.

You are forgetting a couple of things. While your arguments are indeed valid, MS will continue to exist due to their insulation from the Karmic Wheel by HUGE PILES OF CASH. So, even if everyone said "fuck MS, I will not give them another dime, I'm moving to Linux" MS will dump a small portion of their HUGE PILE of cash into something that will generate revenue. Even if they did nothing viable, it would take a long time time to deplete their cash stash. I believe they would even make a MS/Linux before we saw their demise. If MS got into the linux game, I believe it *could* hurt many of the distros out now. Could you imagine a linux kernel wrapped in ms proprietary bs? Then they would have most of the advantages of linux [aside from being open] and the advantages of windows [manufacturer support]. Yeah, it would hurt at first [kinda like a skinned knee] but they could get right back into the game. They may be taked down a few pegs, but you are NOT going see MS die anytime soon.

Now I'm convinced it's nothing...

[birge]

Thanks for the link to the original. However, now I'm even more convinced it's nothing! Look at the variation between the four engines: the MSN results actually don't stand out, even though they are the lowest for Apache. For example, there is more difference between Google and Teoma than between Google and MSN. So, are we going to accuse the other search engines of manipulation, too? They exhibit the same level of variation from the apparently unquestionable Google reference.

Re:A possible explanation

[Keeper]

Because his theory makes logical sense, and yours is rooted in paranoia.