Slashdot Mirror
MSN Search Engine Favors IIS
Scud writes "It appears that if you want to rise up in the rankings over at the MSN search engine you would do well to host your page on IIS. Ivor Hewitt has done a study and it appears that by using IIS, you are likely to increase your odds of a higher listing by several percent."
It's clearly biased towards Internet Explorer too, the results I get back in Firefox are mostly irrelevant blogs and pages full of adverts.
"Those who cast the votes decide nothing; those who count the votes decide everything." (attrib. Joseph Stalin)
So what's going on? I have no idea, I doubt it's all a big conspiracy... but some possible explanations spring to mind: Perhaps the MSN search has simply been coded by developers used to talking to IIS machines and so it just does that job better? Perhaps the MSN spider is taking advantage of some specific IIS features to provide enhanced indexing?
In other words, there are some explanations out there other than "MS is biased and there's a conspiracy and they are trying to take over the world"...
One man's Funny is another man's Offtopic.
... to think ms wouldn't use all it has. Obviously it hasn't yet learned from google, that being evil is bad. And bad guys get punished.
Is there any truth to the rumor that having a picture of Bill Gates on your site makes you #1 in your category?
I'm a big tall mofo.
And who is the silly person who would expect it to be otherwise? Have you actually been listening to the news at all over the past decade? Have you learned nothing? The real story would be if the ranking did not rise if it were housed on an IIS server. Otherwise it's a nothing, I would have assumed that.
... and they still think they can beat Google to the game. When are they going to realize that what made Google so successfull was the fact that is has been so unbiased in all ways imaginable, including not accepting payments to get higher rankings.
Google makes money by prioritising quality. Microsoft makes money by prioritising money.
Go figure.
My page is titled "San Andreas Radio" and if you Google it, comes out #1 or #2 every time.
MSN it and it comes out about #7. Either they're being paid to reduce its rank (it's a bit subversive), or they don't like the fact I'm hosted on Linux, or they simply don't have a very good search engine.
If I put the exact unique title of a page into an engine, I expect that page to be #1.
Mirror of Ivor Hewitt's site
The control over what webserver you will use is typically limited by your hosting provider. While many provide the choice between Unix-based servers and Windows-based servers, many do not.
For those who use hosts that do not provide these services, I don't think it appropriate to think that they are simple SOL. Rather, the better quality your website provides, the more relevant it is to the topic you discuss, the better it will fare in any search engine. The type of webserver you are using becomes nothing more than the tiniest fraction of your search ranking.
To be conclusive, it needs to be a controlled experiment with the same text and same outgoing/incoming links.
Just the webserver alone changing. This can happen by taking a popular site and then changing what it reports to the MSN search robots.
But until such an experiment is done, the data is open to too many interpretations.
The link to MSN search on the main story links to beta.search.msn.com. It should be noted that MSN Search is out of beta for a while now - the correct links should be http://search.msn.com. It's not like it's Google or something - trying to keep everything in beta for years to escape criticism.
My site is first or nearly first in google using relevant search terms. But in MSN it never shows (even if listed). Maybe also the use of PHP is harmful for MSN ranking? M.
For years now, the company where I work has had all it's Apache systems reporting that they are IIS 5.0 systems. Just a quick change in a single file before compiling and there you go!
MS software installs MSN as the default search engine in many places, and there are an awful lot of users who don't bother to change anything, so when they hit 'Search', it's going to be MSN.
- Crow T. Trollbot
Just change the server response line if the GET or POST comes from Redmond, WA to say you are some version of IIS. I can already see the recommendations coming from the SEO folks.
Most of those useless keyword, domain parking/hijacking, and spam sites out there run on Linux+Apache because the owner can host thousands of those domains fairly inexpensively, and that's the key to all spam: minimization of operating expenses so you only need 1 out of 100,000 users to click/buy to turn a profit.
These sites don't have any real content, they just point to other sites and/or exist to spam you with advertisements. Some of them have googlebombed their way higher into the rankings.
My guess is that MSN does a slightly better job of filtering those useless sites out of the index at the present time, OR the "googlebombing" techniques they use aren't as effective with MSN's indexing. Since they almost exclusively use Apache that would have the false appearance of favoring IIS.
This is just a guess, but it seems plausable.
Natural != (nontoxic || beneficial)
MSN Search should be banned for being dishonest.
Banned from where and by whom?
MSN search can do whatever they like. I don't know anybody who actually uses it. Even non-tech oriented people that use IE (against recommendations) set their startup page to something else. Google, mostly, but also "My Yahoo" and their webmail or portal of preference.
No sig
...Googled for anything using MSN!
If you disagree with me on social issues, then it's pretty clear that you are a narrow-minded bigot.
- Google 7873
- Yahoo 3163
- MSN 199
- AOL 65
- Dogpile 44
- Unknown 41
- Earth Link 28
- AltaVista 16
- Excite 14
- A9.com 9
- Others 77
...which comes out to about 2% MSN.
Remember, the guys working on the MSN search engine certainly use IIS to host their intranet sites, and whatever internal webservers they use to test against are probably IIS as well, at least in the most cases. They are likely to consider bogus results for their own sites (both internal and external) more critical... that's not malice, that's just human nature. Even if they consciously work against that, they're more likely to notice problems there first.
And search engine tweaking is more an art than a science. It's an evolutionary process, with feedback loops and strange attractors. So if there's any difference in the behaviour or design of Apache or IIS that would be visible to a search engine, it's likely to lead to a slight bias in favor of the server software that the servers they pay more attention to run.
Leads me to think: is it significant? That is, can we exclude (to a reasonable certainty, that is, p>0.95) the possibility that the effect seen cannot be attributed to chance or some other criterion MSN uses?
Ivor says at some point The initial set of words indeed showed a significant difference between the results from Google and the results from the Beta MSN search..
But what does he mean? I would be interested in what kind of significance test was applied, what the exact results were. Just looking at the ratio of percentages doesn't tell me enough... One should go back at the original data (seems provided, good) and check if the effect is actually trustworthy or just, in Ivor's words, "Odd. Pure coincidence perhaps."
Before seeing some analysis of significance, I don't believe anything...
The only difference in the HTTP response is just that IIS adds headers and that IIS has that stupid HTTP Continue on handling SOAP via ASPNET.
Just telnet to almost any Apache web server and type GET / and then to an IIS server and do the same thing. Look at the top. Almost all non-IIS web servers return no default headers.
Microsoft.com:
redhat.com
Have they gone ahead and implemented that thing about assigning you a hosting provider at birth then? What a shame. Back in my day, we used to be able to pick our hosting provider based on what they provided and what they charged for it.
Ah, the good ol' 1900's.
--MarkusQ
"msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
"Time is an abstract concept devised by carbon-based lifeforms to monitor their ongoing decay." - Thundercleese
You can have a "conspiracy of one" if that person acts in multiple roles.
As an example, let's say that one person is a company's bookkeeper and CFO. (This isn't uncommon in small companies.)
As a bookkeeper she cooks the books to cover her embezzlement.
As CFO she prepares false financial documents for her company and its investors.
One person, criminal acts in two roles, so in many states she can be charged with conspiracy in addition to embezzlement.
BTW, this isn't a "conspiracy" in the legal sense since it's not a crime to give preferential service on the basis of web server. It's sleazy unless it's fully disclosed, but it's not a crime unless they actually sell the search engine as an unbiased tool.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
In other words, there are some explanations out there other than "MS is biased and there's a conspiracy and they are trying to take over the world"... "
It's called plausible deniability. "Why, no, we had no idea this would happen. You say it's an interaction with an IIS feature that causes this to happen? Heavens to Betsy, we never thought of that."
Microsoft people aren't stupid, and they ARE trying to take over the computer world, or haven't you been paying attention to what they say and what they have done? The engineers that built MSN Search would certainly be aware of any interaction that fits with IIS features to provide enchanced indexing. They would have been all over it from the beginning. And a side-effect means that IIS sites come out higher? Great! It's a feature that benefits us, they would think.
Of course MS is biased. Of course they would have noticed this. Of course they like it.
It's just coincidence that there happens to be a bias that makes IIS-hosted sites measure higher by this metric. ;-)
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
Then, in 2007, came Longhorn, with integrated web search using not Google, but MSN. Joe Sixpack didn't care, but MSN was so damn convinient he forgot about Google - effectively forcing Google Inc. with its costly development department out of business. Later - oh surprise - all results you got for "Linux" on MSN were advisories to ditch it for Windows. He who controlled the search result, controlled the industry. (Maybe I should put some fake Frontpage-Meta-Header to my webpages to increase Rankings on MSN ... just to be sure)
Screw the FSM - Real geeks believe in the Invisible Pink Unicorn
Believe it...
First off, I looked at the difference in means for Apache rankings in MSN and Google. 61.5% (MSN) vs. 64.3% (Google) for 970 observations Right there, you ought to be able to eyeball it and see significance. But, to make sure, here are the results of a t-test which checks the likelihood that two matched sets have different means (forgive the crappy formatting):
M G
Mean 0.615061856 0.642948454
Variance 0.01100624 0.008740111
Observations 970 970
Hypothesized Mean Difference 0
df 969
t Stat -10.51551356
P(one-tail) 7.26569E-25
t Critical one-tail 1.646427658
P(two-tail) 1.45314E-24
t Critical two-tail 1.962415113
As you can see, the P is 1.45 x 10^-24, which at least makes us think the results are not pure coincidence. I don't intend on speculating on the causality, though...
You could just change the HTTP Server header that Apache sends out. Someone should try it for a few weeks and see if it really makes any difference.
If you have mod_header installed, just add the below line to httpd.conf:
Header set Server "Microsoft-IIS/6.0"
Add something like this pseudocode to your server:
if $Browser = "MSNSearchBot" then $Server = "Microsoft-IIS/6.0"
One line blog. I hear that they're called Twitters now.
Try typing "online music".
On Google the top two references are iTunes and iTMS. On MSN you'll have to go through a few pages before you'll see anything about iTunes.
Yeah, I trust Microsoft to provide unbiased search results. Sure I do.
m.m.
Yeah the trouble is that the person who submitted the story linked to the old "original" result set i.e. the "/orig/" in the url rather than the more complete more recent results at: http://www.ivor.it/goog
I guess the "MSN against Google" report is more attention grabbing.
Thanks for the link to the original. However, now I'm even more convinced it's nothing! Look at the variation between the four engines: the MSN results actually don't stand out, even though they are the lowest for Apache. For example, there is more difference between Google and Teoma than between Google and MSN. So, are we going to accuse the other search engines of manipulation, too? They exhibit the same level of variation from the apparently unquestionable Google reference.