Modeling Linking on the Web

An Anonymous Coward writes "Amazon has a much greater market share among online bookstores compared to the greatest market share for offline stores. How is this possible? Because the web changes how people find information. There are millions of links to Amazon on the web, which makes it more likely for people to find Amazon when surfing the web, or when using search engines which typically use link popularity in ranking. This makes it harder for new businesses to compete. Researchers have discovered that across the entire web, links are distributed according to a "power law" which leads to "rich get richer" or "winner's take all" behaviour where a small number of sites get the vast majority of links and traffic. A new study just released by NEC shows that this behaviour varies in different communities, and shows how to predict competition in different areas. For example, you can see how much tougher competition is among booksellers compared to photographers."

Comments

Proving dot-commers weren't so crazy

[michaelmalak]

Wouldn't this prove that the quest for eyeballs was no more crazy than the quest for a starlet to become a Hollywood star, the quest for a high school quarterback to make it to the NFL, or the quest to win the lottery?

Gee, Name recognition should have some bearing...

[Shivetya]

Amazon does a lot to get their name out. So its very reasonable that most people would tend to look towards them for books.

Ask people off the street where they would buy books on the internet... and your bound to get many replies of Amazon or "I don't know".

I suppose by their logic that the only place to be on the web is AOL... but on the street you could get that response as well.

Advertising does pay. Links on the net may lean towards one provider over another, but most of them were bought by one method or not.

One reason the rich get richer is because they are optimist, they are willing to do it now. The best time to start a business is always TODAY... the best time to get your name out there is ALWAYS today.

trust

[selderrr]

the internet is a very young medium and most online buyers just don't trust much... yet. I'm a regular buyer, but wouldn't really consider buying a CD from a company that just showed up and is hosted on some obscure domain.

Wait until online buying settles down a bit, and everyone gets used to buying stuff everywhere. Then wait for VISA to become more secure (read : Hardware cardreader devices at home, which is inevitable) or for banks to do direct transactions between you and the book reseller. THEN we can start to evaluate models.

Re:trust

[swillden]

How would a hardware card reader at home help anything? Havn't you even followed the whole satellite-stealing saga? If you give the end user control of the hardware, you can't trust it, period.

Geez, I get tired of correcting this misperception on /.

The pay TV systems are the worst thing that has ever happened to smart cards. There are numerous reasons why they're insecure, and those reasons do not apply to other systems, particularly not credit card systems.

First, pay TV engineers have an essentially intractable problem. No security system is unbreakable and the problem is particularly difficult when the attacker has unlimited access to key components of the system (as is the case with all smart card systems). However, the pay TV guys have the additional complication that satellite television is *broadcasted*. That means that every card is doing the same thing with the same keys and the same data, which in turn means that breaking one card essentially breaks all of them (or at least a large number of them).

To try to manage this risk, the pay TV designers use flashable cards and a security-by-obscurity approach. In most environments flashable cards are a very bad idea, because you want to make sure that attackers cannot fiddle with your software. In this situation, it permits the system designers to modify their system frequently and remotely, to make it obscure again and shut down all the existing pirate cards. OTOH, flashable cards are easier for attackers to fiddle with. The result is that every update is broken within a few months, but most breaks are disabled within a few months, and that keeps the whole thing in the news all of the time.

Nearly every other smart card system can take a different approach, one in which (a) the card is expected to be secure for a few years and then replaced and (b) the break of a single card does not compromise the whole system.

Card-unique keys, key rotation, audit mechanisms, hotlists and moderately frequent on-line validations are all tools used by careful designers to ensure that breaking one card does not break the whole system, and that broken cards can quickly be detected and shut down.

When you add those tools to the security measures offered by the card itself (such as S/D PA resistance, environmental monitoring, security-sensitive layering, chemically-similar protective shields, etc.) the result can be a highly secure system.

The job of a smart card system designer is easy to state: Design a system such that the cost of breaking a single card is much higher than the value of breaking that card. A good estimate for the cost in expertise and equipment to break (extract the secrets from) a single, modern, well-designed card is about $250,000, give or take a little. As the card ages and new attacks are devised, that cost goes down. Meanwhile, card manufacturers make changes to defeat the new attacks, raising the bar again for new cards.

For credit cards, this is an easy thing to do. Not only are the cards unique, used on-line frequently (whether at home or at a merchant) and replaced regularly, the possessor of the card is generally motivated to maintain the secrecy of the information in it. Further, the job of the card is identification and authentication, rather than decryption, which is a completely different problem from that of pay TV cards.

Secure smart card credit cards are easy for a variety of other reasons that I won't go into, but perhapes the most important is that there is already a significant and mostly workable security infrastructure in place that doesn't need the cards themselves to provide any security. This model breaks down to some degree for on-line transactions and a chip card is a perfect way to close that loophole.

It's true that nothing is completely secure, but it's also true that nothing has to be completely secure. Success in security is when the attackers decide to find an easier target.

Scale-Free Networks

[bonoboy]

This is also called a scale-free network, and the research on it, by Albert-Laszlo Barabasi (currently at Notre Dame U) is in this week's New Scientist. (Apologies, it's not on their site yet - www.newscientist.com) He's applied it to many systems other than the web as well, from viral transmission on the net and human populations to the vulnerability of "hubs" in genetics (a few, like p53) would take out damn near everything due to their pervasiveness and even quantum mechanics.

Re:Scale-Free Networks

[ngibbins]

This is also called a scale-free network, and the research on it, by Albert-Laszlo Barabasi (currently at Notre Dame U) is in this week's New Scientist.

There are a quite few papers on this topic (behaviour of disordered networks) by Barabasi and one of his research students, Reka Albert (now probably graduated), most of which are available from his research group's website or from arXiv.

Particular highlights:

A-L. Barabasi and R. Albert, Emergence of scaling in random networks, Science 286, 509, (1999)

A-L. Barabasi, R. Albert and H. Jeong, Scale-free characteristics of random networks: The topology of the World Wide Web, Physica A 281, 69-77 (2000).

A-L. Barabasi and R. Albert, Topology of evolving networks: local events and universality, Physcal Review Letters 85 5234 (2000).

This work is an interesting counterpoint to the 'small world' networks of Watts and Strogatz:

D.J. Watts and S.H. Strogatz, Collective dynamics of 'small-world' networks, Nature 393, 440-442, (1998).

D.J. Watts, Small Worlds, Princeton University Press, (1999).

What about the biggest e-commerce of them all?

[Anonymous Coward]

Where does the pornography industry fit into the chart?

And can you provide a few links there as well? Just to "sample"

Google?

[cetan]

I don't know if this applies or not, but what about the rise in popularity of Google?

Clearly they came in late, after all the other search engines were established, but they ate up market share and eye balls because they were (and are) better.

So I could imagine that there are some Amazon-killers out there or that at least there could be...

Re:Google?

[selderrr]

I really hope there are a lot off Googles out there to kick Amazons fluffy butt : Google has proven that ad-free online information (and in a way commerce) actually works. Works better even than the ones with Ads.

Amazon gives me headaches with all their fsking commercials and screaming colors. I don't go there unless I really can't find my info anywhere else. The opposite with Google : their layout is so nice I sometimes find myself just googling away for fun... like the old days when the internet was still fun to explore.

Re:Google?

[Lee Bottemiller]

So I could imagine that there are some Amazon-killers out there or that at least there could be...

As much as I want to agree, I can't. Google's quick rise to power was because Google offered something much better for very little extra effort.

When Google hit the scene, search absolutely sucked so there's was an existing scratch that needed itching and to scratch that itch, all you did was search from a different page.

So the user got a massive reward from a tiny change. But with Amazon, there's not that much pain in the mind of the online consumer. One-click ordering. Recommendations. Reviews. A very usable site by many standards.

But switching from Amazon to another bookstore requires a good amount of hassle... MUCH more effort than switching search engines and offers a much smaller reward than Google offered to frustrated internet searchers.

Re:Google?

[wfrp01]

Google is interesting for another reason also. If you have a high Google rank, you increase your probability of getting linked to. Which will increase your page rank. Ad infinitum.

I love Google, but I wonder sometimes if at the same time it makes it easy to find good sites, it also inherently keeps other good sites down (relative to the high ranked sites).

Re:What's new?

[Jim Efaw]

Isn't this a law in all kinds of business? The rich do get richer. What can you do about it?

It's happened over and over again in the economy in the U.S. elsewhere. We're already seeing it happen again fairly rapidly in the U.S. (and probably other places) in a lot of service and utility businesses-- although some of that is a result of deregulation. Online business just speeds the process up.

The big benefit of the public Internet isn't the speed at which the rich get richer-- it's the speed at which something else can come along. The (relatively) low cost of having a presence online, and the speed with which one can establish a satisfying niche, is what made the Internet the "equalizer" in the late '90s. And everyone will still have a good shot at creating a niche and possibly overtaking this year's "big guns" on the Internet, unless a few organizations get enough power to control whether one "free" person can comunicate with other "free" people arbitrarily.

Feedback Loops & Linking

[inKubus]

Well, in "Cybernetics and Society: The Human Use of Humans" by Norbert Wiener, the author talks about messages--communication. Links to a web site are messages given to people using the web that a given site/page/whatever exists. They are easy to make use of, since all you have to do is click it and whatever is on the other end is given.

This is all fairly obvious. The neat thing is how these messages and the messages they point to interact. Dr. Wiener says that the more unique a message, the more "important" it is. This is simply because an overused message (a cliche) quickly becomes filtered by the human mind, and loses its meaning.

Take the Amazon example, for instance: How often do you click a LINK to Amazon? Yes, there are hundreds of thousands of links to Amazon, but I would guess most of them NEVER get clicked. Why? Because there are too many of them. The first time I saw one, I followed it, but now I just ignore them. I almost never click on Amazon links because I know it goes to some bookstore.

When I do go to Amazon, I just type the DOMAIN name into my browser, and go directly there, and do my own searching, follow my own links.

So, basically, there is an upper limit to the number of links before they essentially become useless. Of course, this upper limit is dependent on the total number of users who haven't seen the links, which is increasing every day as new people come on to the web. As the number of links reach this critical mass, more and more people are just typing in the domain name rather than following a link.

This is Google's essential flaw. It does not recognize that a site like Amazon does not need an entry in a search engine. There are enough links out there already for just about anyone to find it. Google should instead group searches around a bell curve distribution, where the sites with the medium number of links have the highest relevance, with underlinked and overlinked sites falling off the ends.

How are new sites found out about and linked enough to show up in an engine like Google? Advertising. Mostly word of mouth and link ads, and in certain cases print and television advertising (although this is less effective, because it requires the user additional steps to make use of the message (ie: remembering the domain name at a much later time and then typing it in), which is why the .com ad explosion on TV failed to do anything..)

Really, to be effective, you need to have 10-20 contacts online, have each link to your site. Spread the word as much as you can. And save your ad budget until your word of mouth traffic reaches critical mass. Then spend it on bandwidth.

Really, time is the only key. Oh, and having something useful or funny.

Anyway, this quickly turned to the theories of getting a lot of hits, and I apologize, but you can see that the middle is the best place to be, and maybe Google will recognize this. This would do a LOT for online commerce, and the economy in general. Support bell curve relevance.

Cheers.

Webcomics

[BitwizeGHC]

This is true of Webcomics as well. Ask someone what their favorite Webcomic is, and they will almost invariably respond with one of the following: User Friendly, Penny Arcade, PvP, Sluggy, Sinfest, Megatokyo or Exploitation Now. With the exception of Penny Arcade, I have found the total combined quality (art + writing + humor) to be fair at best, and atrocious at worst (guess what the worst is; hint: think of a little dustball with feet). But these sites are linked to from all over, and they often link to each other, creating "flash crowds" from Slashdot, other comic sites, personal home pages, etc.

There is a class of "second tier" comics which have nice little followerships: Little Gamers, Sexy Losers, Polymer City and Cool Cat Studio (really, any Keenspot comic that isn't Sinfest or EN) are among these. Everyone else, myself and my comic included, is "third tier", i.e., tumbleweeds rolling across their allotted server space.

Then there is Pokey, which stands conspicuously on its own. HOORAY.

Re:power law

[soboroff]

The difference between a Pareto distribution and a power law distribution is that in a Pareto distribution, the probability P[X > x] ~ x^-k, (that is, the probability that a observed value is greater than x is proportional to the inverse power of x) whereas a power law is P[X == x].

And a Zipf law is a power law on ranks, rather than values.

Lada Adamic of HP has an excellent how-to on power law distibutions you might find interesting.

Some misunderstandings here

[dipfan]

Researchers have discovered that across the entire web, links are distributed according to a "power law" which leads to "rich get richer" or "winner's take all" behaviour

Ah, no that's not what it shows:
Quote from the research: "In fact, pure power law scaling appears to be the exception, rather than the rule." What the research shows is that "winner takes all" varies across the web between categories.

Also, the researchers have (I suspect) a mistaken view of "competition" and competitiveness. They rate the Amazon category of book-sellers as "more competitive" - when in fact it may be less competitive in economic terms, being dominated by one or two sites/sellers. Whereas the photography caregory may be "more competitive" because of a larger number of rivals of about the same size.

What they mean, I suspect, is that the publications/Amazon category has higher barriers to entry - the amount of adertising etc being a greater sunk cost, and likely to deter any aspiring internet book retailers. In purely economic terms, that makes the category less competitive. As an illustration, ask yourself if the operating system software market is more or less competitive because it is dominated by one large brand in Microsoft?

A good tool to look at web linkage

[SweenyTod]

Is called VisIT. It produces a graphical representation of how sites link together, based around any given query. It was used quite sucessfully to demonstrate how Scientology had spammed Google, by creating multiple domains all linking back to their main web page.

It's a freebie download and you can get it here.

Re:Gee, Name recognition should have some bearing.

[sugrshack]

absolutely. Amazon understood a basic principle of commerce and marketing... BRANDING.

Most people associate even the concept of purchasing something online with Amazon... they were pioneers, and managed (using linking, etc) to embed themselves in the mind of the consumer. I must admit; even though I may like to think of myself as one who resists marketing, when I need to buy a book online, where's the first place I think of going? Amazon.

Insidious, ain't it?

I am the village idiot i don't have anything to do with this pathetic little opera i just felt like passing through... - PDQ Bach