Web Log Analyzers?

sammy.lost-angel.com asks: "What's the best web log analyzer out there today? It's time to upgrade our horribly out of date one and I'm not sure what's good out there at this time. Our site receives about 50,000 hits a day, so things like remembering what's already been analyzed can save a lot of time." What about log analyzers that can work on more than one type of web server? An analyzer that could parse access data for, say, IIS and Apache would be a nice tool!

Comments

Analog and Webalizer

[rakerman]

http://www.analog.cx/

http://www.webalizer.com/

Re:Analog and Webalizer

[Stephen]

...it can be hard to dig through the [analog] documentation.

I (the author) have some sympathy with this; but the main problem is that it's so configurable that there just are a lot of commands.

I have done some work recently on presenting the documentation in different ways. As well as the main topic-based documentation, there's now a page with only the most basic commands for beginners; a comprehensive index; all the commands on a single page with a BNF-type grammar; and two sample configuration files with all the commands in, one in topic order and one in report order. There's also the beginnings of a collection of third-party HOWTO's (for which I need more volunteers, HINT HINT!).

I do take a lot of time and trouble over documentation, I suspect much more than most open source projects. My rule is that no change can be committed until it's fully documented. So you will never find the documentation lagging behind the reality, or options missed out of the documentation. I also spend a lot of time rephrasing the existing documentation.

AWstats rocks!

[OctaneZ]

I have been running AWStats since July, and I absolutely love it. It does not provide the fine-grain detail that many people need, and which can be provided by Analog. But it does provide exactly what 90% percent of us need, in an easy to view package. It creates an easy to understand page about many aspects of your site, including, users, page hits, countries, languages, OS, browser, spiders/robots, access times; it's great! It is also a GPLed perl script! The developement team is over at Source Forge and is actively releasing new code all the time. It also has the added benefit of allowing cgi updating through a web page; simply putting the script in your /www/cgi-bin/ directory and adding appropriate permissions allows you to get up to the second information about your sight without having to dig up a terminal! Definately check this package out!
-OctaneZ

Analog

[Stephen]

I'd like to plug analog. I'm the author, so read my comments in that light. :-)

First, as others have commented, the commercial programs suck, especially Webtrends.

Analog is over six years old, but it's still actively developed, and I think it's still the leading free log analyser. The main contender is the Webalizer. To some extent it depends what you want (why not try out both?). The Webalizer's biggest advantage is that it produces prettier pictures. Some of analog's advantages are that it is more configurable; that it runs on any OS (the Webalizer is Unix only); and that it can analyse logfiles from any web server.

Besides, analog's author reads Slashdot.

Re:Analog

[frankie]

I use Analog exclusively (well, after DNSTran for name lookups and Perl to sort out sub-logs) and I have found little reason to complain. As Stephen mentioned, you can use ReportMagic to prettify the output. I don't bother.

My only complaint is Stephen's dogmatic insistence on not performing any form of speculative analysis. For example, he refuses to even attempt visitor counting, path tracking, etc. The sort of stuff that bosses like to see, whether or not it's strictly accurate.

Stephen could put WebTrends out of business with a couple hours of coding, but he has his principles.

Re:What's it matter what server generates the logs

[Stephen]

Last I checked, both IIS and Apache generate (or can be set to generate) W3C standard format logfiles. Part of the reason for having/using that standard is so that you don't get locked into a proprietary tool.

You might think so, but IIS breaks the standard in several ways. And it's not even really a standard, just an early working draft that was never finished.

In my opinion, a good logfile analysis tool should be able to recognise and analyse all commonly-used formats, and provide a means to specify custom formats. In other words, it should work with what the server has already produced, rather than force the server administrator to reconfigure the server and ignore old logfiles. My program analog does all this, but most programs don't.