Netcraft Web Server Stats Challenged

kolchak writes "An article in The Age has an interesting analysis of the Netcraft Web Server Usage Reports. According to Port80 Software, Netcraft's surveys are biased towards domain name parkers and very small web sites, not taking into account how popular a site may be - there's some interesting results in the competing Port80 survey." However, it should be pointed out that Port80 "develops software products to enhance the security, performance and user experience of Microsoft's Internet Information Services (IIS) Web server."

Comments

A bit more than the average MS bias

[SeanTobin]

This is wrong on soooooo many levels. I could understand trying to twist the truth by redefining what a webserver is... but thier sampling method is straight out wrong.

Want proof? Here it is. Go to the linked article, (or click here) and where they have the box to check your server header (about half way down the page) type in www.microsoft.com - you will see its running IIS/6. A nice happy IIS server.

Now, type in my web server - http://www.isthatdamngood.com - its a nice Linux/Apache server. My server will CRASH thier app! Actually, a lot of linux servers will crash it...

Kinda hard to claim your results are more indicitative of the market when your scanning technology is flat out broken.

Re:A bit more than the average MS bias

[ejaw5]

Check out the ad below the detection test:

Note:
No matter what the above results show, this company may be running Microsoft IIS and protecting its Web server identity with ServerMask.

Try ServerMask FREE for 30 days. Download Now!
Buy ServerMask for only $49.95 today!

Re:A bit more than the average MS bias

[_xeno_]

Worked for me. I tried "slashdot.org" and "www.theregister.co.uk" - both of them worked just fine. However, "www.isthatdamngood.com" did indeed cause a scripting error - but I doubt it would effect their actual surveying, it's just an ASP error, not an actual "crash."

Anyway, it's long been known that Netcraft's methods are flawed, since it counts individual web servers multiple times for each virtual domain. It should only count unique sites. (For example, Slashdot counts for something like 13 sites - the individual sections (like apple.slashdot.org - I'm not listing all of them), slashdot.org, www.slashdot.org, images.slashdot.org.)

It's still debatable what the correct survey method is (and whether Port80's method is any better), but Netcraft is biased towards sites with lots of virtual domain names. (I'd imagine SourceForge gets counted many times, too...) Of course, it's also questionable if individual servers in a round-robin load-balancing solution should be counted, so counting by IP instead of domain name is questionable too.

As is often said, "there are lies, damned lies, and statistics" - any counting method has issues.

Blah, I can't preview because Mozilla is f***ing broken and won't display the preview page, so please pardon any typos.

Re:A bit more than the average MS bias

[panaceaa]

The parent poster's point is that their site grabber program can get IIS sites but crashes on some Apache sites. Port80 Software may use the same code to run their surveys since both the grabber and survey programs need the core feature of analyzing a site's HTTP headers.

So if their survey script also returns invalid data for Apache sites, then the IIS numbers would be much higher than they actually are. I would at least like to see some actual numbers rather than pure percents before I believed their data. They surveyed 1000 sites -- how many sites are included in the survey's data?

Another thing that seems odd to me is Netscape iPlanet usage is higher than Apache. Where's the primary data to support that?

Like that's going to work

[BigRedFish]

a product .... to confuse script kiddies

I am running Apache on Linux, and I still get 1000 hits a day trying to crack MSADC with buffer overflows, and FrontPage exploit attempts. It's not like the script kiddies check the server ID or pay any attention to it even if they do.

A good methodology

[cgenman]

If you are conducting a survey to find out what is the "best of the best" in server software, why survey Family Dollar Store? Or Land 'O Lakes? You should be choosing technically savvy, solution neutral companies are likely to choose the best. These are the actual companies that have a big web presence and you would not expect them to choose a platform which would affect their bottom line badly... As opposed to Sears Roebuck, whose online presence can be compared to Amazon's retail presence. Would we ask Amazon how to organize endcaps? Let's pick a few technically adept companies at random here...

Amazon - Apache
AT&T - Netscape
Bell South - Apache
Cisco - Unix
Dell - IIS5
Earthlink - Netscape
E-Bay - IIS4
HP - Apache
Intel - IIS6
Lucent - Netscape
Motorola - Apache
National Semiconductor - Netscape
Nextel - Netscape
Qualcomm - Netscape
PC Connection - IIS5

I can't survey any more companies, because Port80's IIS6 server is slashdotted. However, if is apparent from this data that nearly 1/3rd of all websites that count are hosted on Netscape platforms. Apache and IIS share 1/4th each, and Cisco's odd unix variant wrapps up the rest.

Personally I'm amazed that Netscape is holding on to a lead... I would have expected them to be out of the running long ago. I'll have to check them out.

Re:Corporate Web Servers

[Sevn]

As a career admin who has worked for 15 fortune 100 company as either an employee or a consultant in the past decade, and currently as the project lead replacing an aging proprietary UNIX solution for a telecom spanning an ENTIRE STATE you are on crack. To dot the I's and cross the T's I hired FIVE independant firms to do cost benefit analysis on proprietary versus open source even though I already knew the answer. The long and the short of it is, over a 5 year period for our particular needs the BEST case scenerio for cost with the cheapest possible proprietary solution factoring in maintenance, upfront costs, and scale was 10 million dollars. The highest price for an open source solution was 4.3 million and that was because it was a hybrid solution that was about 50 percent proprietary and not purely open source. The solution I went with was 90 percent debian based (since redhat is doing it's thing, and SuSe is uncertain because of the merger) and 10 percent Solaris/Oracle and will cost an estimated 2.3 million. And for the record I freaking HATE debian but it makes the most sense for this particular situation.

Re:Where's Google?

[ChaosDiscord]

I could not help but notice that Google, Yahoo, and Slashdot are omitted from their "top 1000" list.

The "top 1,000" list is based on the Fortune 1,000. Google, Yahoo, and Slashdot aren't on the Fortune 1,000. The theory is that the Fortune 1,000 indicates Real Companies, and that this is what Real Companies chose. However, many of these Real Companies are holding companies or target highly specialized audiences (like people needing drilling supplies). Many of these Real Companies are actually running what we would consider toy web sites: almost no content, entirely static pages, very few pages, and almost no visitors. So while this may represent what Real Companies chose, it does not necessarily represent what people with Real Work chose.

Free Software Wins again.

[Anonymous Coward]

and what would that one line be?I want my $50 worth on my apache server

• Unpack the Apache distro file (apache_1.x.xx.tar.gz) and run the configure script.
  
  Now do the following commands:
  
  
• cd src/os/unix
  (With Apache 2.x, cd os/unix)
  
• vi os.h
  
• Search for:
  #define PLATFORM "Unix"
  
• Replace "Unix" with whatever you want your OS identification to be. (Some of the more creative ones I've done are 'NachOS,' 'PathOS,' 'StratOS,' 'ZerOS,' and 'WinDos'...anything.)
  
• Save the file.
  
• cd ../../include
  
• vi httpd.h
  (With Apache 2.x, vi ap_release.h)
  
• Search for:
  #define SERVER_BASEVENDOR "Apache Group"
  #define SERVER_BASEPRODUCT "Apache"
  #define SERVER_BASEREVISION "1.x.xx"
  
• Replace "Apache" and "1.x.xx" with whatever you want your Server and version number to be. (I recommend "Port80Software-Is-A-Fucking-Ripoff" and "Holy-Jumping-Jesus-This-Was-Easy", respectively.)
  
• Save the file.
  
• cd ../..
  (With Apache 2.x, cd ..)
  
• make
  

You're done. Congratulations. You just saved yourself $49 dollars!!!

Re:Free Software Wins again.

[ivan.ristic]

If you're using mod_security on your Apache server then you only need to add one line to the configuration file:

SecServerSignature "MyServer/19.5.1"

Re:This makes sense..

[Eivind]

Except if you'd bothered to check you would notice that Netcraft is fully aware of this, and thus produce different numbers for "web-servers" and "active web-servers" the latter excludes domains which are only parked somewhere.

http://news.netcraft.com/archives/2003/11/03/novem ber_2003_web_server_survey.html Is the latest survey, apache has 67.41 of all domains (well, all that Netcraft knows about anyways) at 30298060 domains.

If you look only at "active" domains, apache has 68.60%, so actually even a *higher* market-share. Of a total of 14370515 active domains. (so according to Netcraft, about half of all registered domains are "active" and the other half are "parked"