Unique Visitors = 1/10th of Unique IPs?

Max Fomitchev submitted a little blog entry where he proposes that the ratio of unique IPs to actual unique users is 10:1. This flies in the face of the numbers you usually see attached to these sorts of things. I'm not sure about the logic he uses to come up with these numbers either.

Comments

10 was arbitrary

[op12]

The 10 was a hypothetical...the only point was that you can't trust the number of recurring visitors that a site reports because they users come back with a different IP (obvious) and get counted twice. Couldn't one use cookies and IPs in combination to get a better gauge? The IP may change but the cookie would not. Sure some may delete it, but it'll still improve accuracy at least a little bit.

Re:10 was arbitrary

[MikeFM]

My experience is that a lot of users use cookie killing software that removes cookies every time the browser is closed or just reject cookies altogether. Also many users seem to use multiple browsers and computers even within small time periods. Counting unique visitors is really quite difficult. Still, if all site's play by the same rules on counting the number still has some meaning. Unique IP address within a given timeframe is probably a decent metric still.

Re:10 was arbitrary

[bbsguru]

Oh Puh-lease! EVERYONE knows the REAL number is 11.32019 per IP address. This is just silliness! There is no magic number that works everywhere; fuggidaboudit. If it matters that much to KNOW the real number of unique visitors, ask each one for a scan of their right thumbprint, and then create a database. I thought so.

Re:10 was arbitrary

[op12]

My experience is that a lot of users use cookie killing software that removes cookies every time the browser is closed or just reject cookies altogether.

My bet would be in the grand internetscape, this number of users is actually quite small and that most do not reject or remove cookies (at least not often).

However, the point was a better gauge, not a perfect one. Requiring login would resolve most issues of users from different locations or even multiple users on the same computer, but few people are going to like a website they can't browse anonymously.

I think it's just interesting data for site owners and nothing more. I wouldn't overanalyze some site touting 3000 unique visitors a day or something.

Re:10 was arbitrary

[Bogtha]

and honestly if you site has SSL support why not redirect your users to that at all times (i do if they are loged in - even for message boards)

It totally eliminates public caching (e.g. ISP caches). A waste of bandwidth, a waste of CPU, and slower speeds (it can be a lot faster for users to get stuff from an intermediate cache than from the origin server).

Also, this isn't an option for anybody using name-based virtual hosting, which is incredibly common. There have been specifications published for getting around this, but browser support isn't there yet.

Re:10 was arbitrary

[ADRA]

How many user are sitting behind my company's single routable IP address? Hit: Its over 1000.

This sways the number in the opposite direction. The number the story's based on is completely baked. You could attempt to statisticly estimate the number of unique users/ip on your site with some effort, but you can't get a real concensus between one sight and the next. The reason is demographics. If you take a mobile enabled sight, you're almost always guaranteed to get at least 2 IP's per user(one mobile usability, one desktop ease), but if you take a corp-LAN, you're almost guaranteed to have 1-1 user/IP.

As for sites worrying about anonymity-type scatter IP, I think that most anonymity-type solutions are quite easy to detect if a site maintainer really tries to. Remember, the referrer record is your friend here. If the user decides to piss around and blank it out, just loop them back to the main site page or something or tell them to login. So, if user X is behind a proxy that drops referrer headers, tell them to login or 'goto hell'. If someone's using an open relay, drop them from your site all together. A hit doesn't count as a user if you don't let them in =)

Re:10 was arbitrary

[Amouth]

"this isn't an option for anybody using name-based virtual hosting, which is incredibly common"

this isn't true.. i have never had any issue with this - you just need to have it set up right which is quite easy.. what are you using that has this problem?

so...

[gEvil (beta)]

So, he's saying my website has 1/10th of a visitor?

Neighbor's Wireless

[Anonymous Coward]

I help keep this in balance by using my neighbor's wireless, that IP has a load of unique users.

Re:nothing much here

[GigsVT]

I'm on DHCP on Cable. I don't think I'm the only one. I guess maybe the article wanted to show by demonstration how most statistics are made up on the spot.

Maybe not...

[Anonymous Coward]

I've hosted several servers from home for years at a time without my dynamic IP address ever changing, and I've known many others in the same situation. I think this 10x rule might be a bit extreme...

But...

[xpird]

Can he find a formula for the number of /. articles posted vs. the actual unique articles?

WTF?

[giorgiofr]

This argument is flawed. Logging to Slashdot now from my house and two hours from now from my friend's house should count for two visits, and so it rightfully does. The article writer seemed to have a problem with this? ZOMG 2 different IPs...
And if my IP has changed but I'm still here... that's because I haven't surfed for many hours at least otherwise the lease will be renewed and the address will stay the same. So it should still count for two visits. Duh.

Re:Already considered.

[Nos.]

Because its none of your business (you being a web"master"). I understand that my Ip is broadcast, but I'm not going to use a web browser that sends personal information about me or my computer, every time I hit a site. I can use an anonymous proxy now, and refuse/delete cookies, and know that I am not being tracked. Besides, MAC addresses can be altered as well.

The author left out surfing from work

[Trails]

I do most of my news browsing at work, where several hundred people show up as one IP (home computer is exclusively for WoW).

Besides, the assumption that stated unique visitors = actual unique ip's is innacurate. Lots of companies track users with some kind of UID cookie, for more accurate stats. True, this isn't perfect either, and will reset when users wipe their cookies or it expires, but is probably closer to the real number than ip's.

Re:nothing much here

[zippthorne]

Well it appears to make the assumption that the visits are sparse enough for the DSL Ips to change every time (and also that cable IPs are static. Mine is not, though it doesn't change for months at a time)

So, what IS the typical holding interval for a DSL ip?

as for properly estimating, If there are good enough statistics to have separate numbers for both {known, relatively static IPs over a month} and {known dynamic IPs} you could find the ratio of returning static IPs and normalize the dynamic ones to match that ratio.

In fact, I'd be surprised if this wasn't already being done for many sites.

wait a sec

[hurfy]

I forgot something.
What about the other way?

Do they see the 10 people on the office NAT as one IP ?!?

That would skew it in the other direction and average things out wouldn't it? Now 10 is definately excessive.

Re:nothing much here

[hotgigs]

Last line of article: "So do not trust stats, they ARE inflated." Like I have always heard in regards to statistics: "Statistics are like loose women. You can do whatever you want with them" Although I don't suppose most of us on Slashdot would know much about women, let alone loose ones...

I thought it was the opposite.

[Black Perl]

First of all, a DHCP server is typically going to give you the same IP address each time your computer requests it, unless there are more users than IP addresses, in which case there will be some shuffling. But that tends to be when there are more users than available IPs.

There are entire domains hidden behind a NAT device of some sort. This would be many users per IP address. TFA didn't mention this at all.

So I think TFA is indeed arbitrary, and also wrong.

Re:I thought it was the opposite.

[the melon]

Yeah, here at Sun there are nearly 40 thousand people that connect through 40-50 different proxy servers. That is a thousand to 1 in the opposite direction the article claims.

And yes, he seems to have no idea how DHCP really works. Even if your lease is expired you will get the same IP address unless the pool has been exausted and your address re-used. I see that as an extremly unlikely thing to happen because it would mean, as you say, that your pool is smaller than your installed base. If you pool is smaller then you will start having issues because x number of customers will always be without a connection because they can't get an address.

Had he mentioned Dialup users then I would be more inclined to agree because you are very likely to get a different address every time you connect.

Re:I thought it was the opposite.

[XorNand]

True, but some broadband providers intentionally hand the client a different IP when their lease expires in order to:

a. Prevent subscribers from running servers without paying for a static IP. While dynamic DNS services can be a workaround much of the time, it doesn't work very well with SMTP or other cases where DNS caching can cause issues.

(or, if you ask the provider)

b. To decrease the likihood of crackers breaking in your computer.

Re:I thought it was the opposite.

[ranson]

Hmm, well TFA was speaking to DSL users, who generally connect via PPPoE. PPPoE, while it does dish out IPs a la DHCP, does not have a lease file that associates user mac addresses with the IP they were handed on previous connections; users generally will get a different one every time they connect just like dial-up (aka PPP). I would say router-based PPPoE connections drop and reconnect at least once per week on average.

Re:I thought it was the opposite.

[the melon]

Actually the oposite should be true. The bigger the pool the better your chances are for getting the same IP. When you release your DHCP lease from the server or it expires before your client renews it the server still has that address assigned to your MAC address. The only difference is the lease is inactive. If there are addresses in a pool that have never been assigned and a new client connects it will use those addresses first. If there are none then it will start checking the inactive leases. It will take one, not sure if it does them in order form oldest expiration or randomly, and ping the address to make sure it is not still in use. If it is not then it assignes it to the new client. If it is in use then it moves on to the next one and marks that address as unuseable. ISC DHCP will revisit unuseable address once it has exausted the inactive pool. Solaris DHCP will not revisit them until they have been marked good again by an admin.

So the longer you are inactive the greater your chances are that you will get a different address and the bigger the pool is the lower your chances are.

The above is of course null and void if the ISP deliberatly expires leases and forces users to new addresses. It has been my experience though that very few do this. Even my dealings with PPoE based DSL has indicated that as long as you stay connected via a keepalive then you will retain the same address for a while. If you restart your router with PPoE you are likely to change though.

I wonder what percentage of people with DSL and Cable do not use a firewall/NAT box?

Re:I thought it was the opposite.

[spyrochaete]

If you don't want to change settings every time your coworker's IP changes check out No-IP's dynamic DNS service. I've been hosting many live services from my home network for years including POP3\SMTP email and it's been impeccably stable, and totally free!

Re:I thought it was the opposite.

[TheRaven64]

This sig has been in use for a while. I've seen it before, but it's obviously not a real comment; the link text isn't coloured and there bullet is not the right character. When I saw it first (a year ago?), it seemed an obvious joke, and I laughed a little. Perhaps you need to lighten up a bit?

Re:I thought it was the opposite.

[Matilda the Hun]

Yes, well, if I had mod points I'd give you +1 Overreacting. I guess we can't all get what we want.

AOL uses HTTP caching

[QuietLagoon]

All of the millions of AOL users visit websites via a couple of hundred cache servers. You won't see a lot of different IP addresses for the AOL visitors to your site.

I wonder if the other major ISPs do the same.

Re:AOL uses HTTP caching

[drinkypoo]

I wonder if the other major ISPs do the same.

No, most of the major ISPs just have an agreement with someone like Level3.net that handles dialin for them, and they only do caching for customers who pay for "high speed dialup" which is to say browsing through caching proxies that degrade image quality in order to reduce bandwidth consumption due to page loads.

Or, to completely skew his numbers...

[Dynedain]

We have 54 employees going through one firewall, and having one external IP address. On our company website, only that one IP address shows... So for that IP, it is not 1/10th of a unique visitor, it is 54 unique visitors. His numbers are baseless and skewed.

Re:NAT? 10 is too high!

[RingDev]

No joke, we have 800 people going out over one IP from here. Kinda a pisser when I hit the 'slow down cowboy! you just posted' message. As much as the stats are inflated by dynamic IPs and multiple logon points, they are deflated by NAT and Proxies.

-Rick

Crazy article

[TheLastUser]

You can't trust web stats, that much I agree with. The rest is a bunch of hand waving.

DSL customers do not get a new IP every time they turn on their computer. Maybe some do, but my IP changes maybe once every few months, max.

He fails to mention the effect of NAT'ing and mega proxies, both of which are in heavy use and have the OPPOSITE effect. All of AOL emerges through a small number of IP addresses, clearly more eyeballs than IPs.

I agree that IP != eyeball, but that's it, there could be more eyeballs than ips or less, who knows, and it probably varies from site to site, based on demographic. There is no way to know for sure. Cookies will only tell you the number of computers.

Re:Already considered.

[dietrollemdefender]

...can have something more unique to identify a unique customer by than an IP address

So says the AC.

Re:Just count visits.. why count ip's?

[ceoyoyo]

Sounds accurate to me. Many hands, one brain. Or near enough.

Re:nothing much here

[level_headed_midwest]

From what I hear, most cable users are on DHCP. In fact, my cable provider doesn't even allow the option of a static IP whereas the DSL provider will (for an extra fee, of course.)

Re:This is pointless

[Churla]

Yes...

But 37.5% of all stats presented by people are made up on the fly.

Only about 2.31% of people know that by adding numbers after the decimal point the average person considers the number "more credible".

Re:Already considered.

[level_headed_midwest]

That's called Internet Protocol Version 6, where some of the address is your MAC address.

So, what he's saying is...

[zen611]

"Don't trust Stats. Except mine..."

bad maths here

[Ankh]

There are so many factors here that focusing on one probably isn't sensible.

Some examples:

1. Even DSL users can often keep an IP address for several weeks, depending on the ISP.
2. On the other hand, any sensibly configured home network (OK, that's almost none of them perhaps) has a hardware firewall, or has multiple users with "connection sharing", so multiple users per IP.
3. Most offices use a firewall with NAT, so all of $BIG_COMPANY appears to be one IP address.
4. Some ISPs run an HTTP proxy -- AOL is one example -- so that static pages will only be fetched once per Expiry period (or once per day) even if everyone on AOL looks at them.
5. In any case, numbers of users is not the same thing as number of IP addresses; sites are reporting based on cookies or on login codes.
6. small numbers like 10 sometimes take on different values. Er, OK, no they don't but I'm bored.

I don't really know why it matters in any case. For advertising, clickthrough rate is more important than number of users, and they are not very closely related. Sadly, the poorer your site's navigation the higher the clickthrough rate (and the fewer pages on your site people will see each visit, as the ads take them away sooner).

Visitors not trusted in the Web Analytics Industry

[boscowall]

I'm not sure about his article and his formula, but it is already a debate in the web analytics industry. http://www.omniture.com/blog/ Even using cookies it's nearly impossible to get correct unique visitor counts and that is why the industry is moving more towards unique visits, because a visit is a visit, it doesn't matter who the visitor was... The only way to really measure how far off visitor data could be is comparing unique customers (cusomter id) to the number of unique visitors they create (the customer id coming from a login). That way they could see the affects of multiple customers on multiple machines and browsers and also see the affects of multiple customers on a single machine and browser.

What about proxies?

[wbhauck]

Proxies could, especially ISP proxies (AOL, anyone) can hide potentially 10,000's of unique users.

Also, as far as i've seen DSL IPs don't change that often.

The key is not the Ratio, it's the Revelation

[EvilMagnus]

It's that this is a Marketing Person who has realised that IP != Unique User.

That places him amongst a tiny minority of marketing people, even if his reasoning and ideas on methodology are just as batshit insane as the rest of his kin.

Re:The key is not the Ratio, it's the Revelation

[imidan]

Actually, if you go back and look at the rest of his blog, you'll find that he claims to be a developer. In fact, when he attended University of Tulsa, he was apparently surprised to find that some few students there were actually smarter than he was! So he's clearly a very smart developer!

No, but really, if you browse the rest of his blog, he just comes off sounding like a dumbass. Well, more of a dumbass than he sounds like just from this nonsense about unique visitors to his web site.

Re:The key is not the Ratio, it's the Revelation

[robertjw]

comes off sounding like a dumbass.

Appearantly not too much of a dumbass. Managed to get his blog slashdotted. He's probably buying a new car with his adsense check about now.

Re:On the contrary...

[ranson]

AOL user's actual IPs do not change mid-session, but the web proxy server that is making requests on behalf of the user changes all the time; my server logs even show certain objects on a single page for the same user are requested by different cache-*.aol.com servers.

Lots of info about that is here.. including the proxy IP list, etc... http://webmaster.info.aol.com/proxyinfo.html they say specfically "When a member requests multiple documents for multiple URLs, each request may come from a different proxy server. Since one proxy server can have multiple members going to one site, webmasters should not make assumptions about the relationship between members and proxy servers when designing their web site."

My own stats say very different things.

[CFD339]

I did a quick analysis of a 250,000 line entry server log. I counted unique ip addresses, unique useragent cgi values, and then the number of unique combinations.

A useragent value looks like this: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; .NET CLR 1.1.4322)

Although even this is hardly reliable since useragent can be faked, and useragent isn't unique enough to be a client fingerprint -- its still helpful in this context.

One can make the assumption that a given user's "useragent" value isn't going to to change much on a day to day basis, though it will not stay the same over time as vesions get updated. GENERALLY speaking, the same IP address but different USERAGENT values would indicate different people from behind the same NAT firewall, or different users assigned the same DHCP address.

Here's what I got for results -- it looked like counting only unique IP's gave you only about 85% of the unique hits.

Total Hits Looked At: 249861
Unique IPs: 10309
Unique UAs: 1578
Unique Combos: 12232

I've ALWAYS had the same IP (going on 15 years)!

[sd.fhasldff]

My IP has been 127.0.0.1 for a really long time now. Ever since I got my first internet connection, actually. That must be why it's such a "nice" number and not those horribly complicated ones other people always seem to have.

Re:Use cookies

[gstoddart]

Unfortunately, some users disable cookies. And then all you can do is fall back on their IP address.

Of course we block cookies. Because most of the cookies you get offered are 3rd party to the site you're visiting and just crap so gator and all of that other junk could keep track of you. I only accept cookies coming from the site I'm visiting, and then only if I say YES. It took a very long time to teach a lot of people they needed to be more cautious with cookies, because there were a lot of privacy issues with them. If I could block my IP address, I would do that too. =)

It would be nice to see cookie-tracking support in Open Source stats engines like awstats.

But we don't want to be tracked. That's why we disabled the cookies in the first place. I got so damned tired of things like doubleclick trying to set cookies back in the day -- if you need to give me a cookie to make your site work, *maybe* I'll accept it. Giving me third-party cookies? No flippin' way!

Course, I'm pretty anti-advertiser and the like. So I'm probably not a good example. :-P