Geographical Mapping of Website Traffic?

inputsprocket asks: "Having gotten into a project that involves doing some geo-statistic data mining analysis on Apache log files, I couldn't believe how there seems to be such a large chasm, in open source's vast library, for such a powerful analytical tool. Unless I'm wrong, then there is no such open source project available for *nix variants. There appears to be a great deal of brand software (heh — only for Windows off-site analysis though), yet there is are great open source tools for generic mapping tools (GMTs). The available GMTs seem to be overkill for mapping of weblogs and require a lot of in-house programming to map IP addresses to cartographic data using free geo-location databases. The closest I found was Palentir. Does one really have to rely on the powerful needs of Map Server, MapTools, Carto Web, or Map Builder to draw simple maps with statistics on web usage, given the extra programming needs required to do this? "To see what is possible, have a look at DeepMetrix's LiveStats.XSP — a commercial option, which isn't a problem. It's just a shame that it's only for Windows and for of-site browsing.

Somebody, please tell me there's more to open source geolocation of web traffic than Analog."

Comments

Try hostip.info

[Space cowboy]

I created the hostip.info website a couple of years ago which is an open-source geo-location system. I've since passed it to others to maintain.

On their front-page right now is a geographic map of website traffic, which is a "public-beta" of a mapping system (upload IP addresses, get a map).

They don't actually appear to give out the code (from only a quick look on the site, that may not be correct), but this ought to be pretty trivial to do yourself if you wanted... The public API of hostip.info returns the lat/long of any IP address it knows, and plotting lat/long on an image of the world ought not be too hard.

I designed the whole system so you could download and set up your own geolocation database, slaved from the master at hostip.info with regular updates. Then you can query IP addresses locally and generate charts however/whenever you wanted...

Just a thought,

Simon

NeXT

[2.7182]

I seem to recall that the NeXT machines that I used in the 90's could do stuff like this with some piece of software. Anyone recall the name ?

perl and cpan are the answer

[stry_cat]

Like most things, just use perl and a cpan module. Apache2::Geo::IP

Google?

[CodeMonkey22]

Google Analytics does a decent job of this. I wonder if they have an API you could hit against it...

what's the website with the pinpoint on the map?

[Anonymous Coward]

on a slightly offtopic topic, does anyone know the website where you can enter peoples zip code (in US) and it will leave a pinpoint on a map for their location?

Can that website, or any others, allow you to enter a large number of zip codes at once (say exported from database) then map a large amount of zip codes with a pinpoint so that we can see where all of our customers are located?

thanks in advance

Azureus

[slim]

Azureus can display the location of your peers on a world map.

That's Open Source -- so look at their source.

ip2location

[Utopia]

http://ip2location.com/ is the best database out there.

Re:ip2location

[Anonymous Coward]

Yes it sure is, but it's also 200$/server (or much more if you want more details, up to 1099$/server)

OpenLayers.org, OSGeo Foundation and GeoRSS

[Lord Satri]

I don't know the answer, but I can direct you to OpenLayers.org and to the Open Source Geospatial foundation to look for answers. I'm interested to the answer because we want to geolocate visitors for slashcode itself, and I copied your question on Slashgeo.org, which may help attract more answers to this /. story.

While your at it, GeoRSS is also open and, in my opinion, the future of RSS. Learn more about it here.

Invented On Slashdot

[Doc Ruby]

As seen on Slashdot: Map-O-Net maps IP#s on a fractal-ruled IP#-space map, including geography and orgs.

Re:ip2location: accurary != precision

[DaveCar]

I can't see much point in anything that is supposedly more "accurate" than by country. Using the demo site there I got either just "UK" or "UK, London" for a whole bunch of addresses dispersed around the UK. Seeing as most home users will be on some xDSL connection which is probably peered/registered in London/$MAJORCITY, but they are perhaps a few hundred miles away it doesn't seem very "accurate" just more precise (but wrong). Maybe for the US you'll at least get an idea of which time zone they might be in?

You can go get the Geo::IP database for nothing, so I think I'd be tempted to just use that.

Perl Module Geo::IP and MaxMind Database work ...

[xmas2003]

A while back, I wrote a quick little Perl script to do geolocation and other misc. info - see the page for more info, but it uses the free Perl Module Geo::IP and the MaxMind database. The free version of the later used to only provide country data, but they now provide lat/long and city info ... seems reasonably accurate, although with all of these things, the results can sometimes be way off.

So with that, it wasn't too hard to write another Perl script that parses the IP address from the Apache logs, acquires misc. geolocation info, and then dumps out the lat/long data in Javascript for the Google Maps API - see this graphically shown on my christmas maps page.

Re:ip2location: accurary != precision

[poot_rootbeer]

Agreed, attempts to map IP addresses to geographical locations are so imprecise as to be largely useless. I don't know if it's still the case, but the last time a company I worked for attempted to do geotargeting by IP discovered that $SOME_HUGE percent of our users were all apparently living in Virginia. Turns out those users were all coming through an AOL traffic relay located there. They could have been anywhere in the country in reality.

If you want to find out where your site's users live, ask them! Set up a sweepstakes where they have to fill out a form giving their zipcode to be eligible to win. Divide your content into distinct sections by geographical region and see which ones your users visit most often.

Azureus-RIAA.

[Anonymous Coward]

"Azureus can display the location of your peers on a world map."

Thanks

Re:Perl Module Geo::IP and MaxMind Database work .

[LanMan04]

Yes, I'm sure we can *really* control your xmas lights....weren't you around a year or two ago when all similar sites turned out to be BS?

Re:ip2location: accurary != precision

[bcrowell]

Hee hee ... I just went to http://ip2location.com/ , and they informed me that I was located at "(US) UNITED STATES, VIRGINIA, HERNDON." Mm...no, I'm in Los Angeles, and no, AOL isn't even my ISP :-)

I have at least two misgivings here:

1. The precision seems to be horrible.
2. It might not even be a good thing if this kind of capability was reliable and easy. I recently got my site DDoS'd by a subdomain that reverse DNS revealed to be a block of servers owned by the government of a certain province in Spain. I e-mailed their webmaster (in my best high school Spanish), and got no response. One of the first things I thought of was, "Hell, I don't need these foreign freakos hosing my server! What if I just use geolocation to lock out people from outside the US from using this particular CGI script." Well, once I calmed down and thought about it, it became clear that I didn't want to turn away 90% of the world's population just because some psychologically disturbed IT worker in Spain was hassling me. But that's exactly what could happen if geolocation of http requests became cheap, easy, and accurate -- you'd get the internet becoming fragmented along national lines. CI can just imagine the state government of /(Georgia|Utah|Iran|Singapore)/ saying, "We're going to prohibit discussion of homosexuality on the internet, because it might reach young people. It's not an unreasonable burden on webmasters, because they can just use geolocation to determine that the user is in $1."

phpmyvisites

[inputsprocket]

Replying to my own question, but I think I found the most complete OS project out there for geolocation log statistics....

I've been playing with phpMyVisites and it looks a very nice and thorough implementation of what I was looking for.

Although it's method for country geolocation is a bit *cough* rough, I think a module to hook into Space Cowboy's hostip.info shouldn't be too difficult...

The only problem with it is it uses mysql to store the logs *grrrrr*

OpenLayers looks slick!

Re:phpmyvisites

[slowbad]

The statistics from Ritecounter [free invisible web counters under 3000 per day] ties itself into Google Maps, but so far it only does 10-20%

They don't distinguish between Time-Warner customers in Chicago versus New York, so you get a lot of DC-area longitude and lattitude URLs.

ClustrMaps

[Anonymous Coward]

Why don't you try http://clustrmaps.com/ you can tag your site to it. Easy really.

Re:Perl Module Geo::IP and MaxMind Database work .

[inputsprocket]

Xmax -
please get in touch if you wish, but I did some Apache benchmarks with Geo::IP and Pear's GeoIP, both using Maxmind's free database and while Pear was quering at 1/5ms on 100 concurrent connections, Geo::IP was answering 1:180ms but with apache balking after 2000 request...

This really disappointed me and as such we only run PHP on the maxmind dbase....

I never looked into why the delay was there, but care to comment?