Modeling Linking on the Web

An Anonymous Coward writes "Amazon has a much greater market share among online bookstores compared to the greatest market share for offline stores. How is this possible? Because the web changes how people find information. There are millions of links to Amazon on the web, which makes it more likely for people to find Amazon when surfing the web, or when using search engines which typically use link popularity in ranking. This makes it harder for new businesses to compete. Researchers have discovered that across the entire web, links are distributed according to a "power law" which leads to "rich get richer" or "winner's take all" behaviour where a small number of sites get the vast majority of links and traffic. A new study just released by NEC shows that this behaviour varies in different communities, and shows how to predict competition in different areas. For example, you can see how much tougher competition is among booksellers compared to photographers."

Comments

trust

[selderrr]

the internet is a very young medium and most online buyers just don't trust much... yet. I'm a regular buyer, but wouldn't really consider buying a CD from a company that just showed up and is hosted on some obscure domain.

Wait until online buying settles down a bit, and everyone gets used to buying stuff everywhere. Then wait for VISA to become more secure (read : Hardware cardreader devices at home, which is inevitable) or for banks to do direct transactions between you and the book reseller. THEN we can start to evaluate models.

Re:trust

[GigsVT]

How would a hardware card reader at home help anything? Havn't you even followed the whole satellite-stealing saga? If you give the end user control of the hardware, you can't trust it, period.

Re:trust

[swillden]

How would a hardware card reader at home help anything? Havn't you even followed the whole satellite-stealing saga? If you give the end user control of the hardware, you can't trust it, period.

Geez, I get tired of correcting this misperception on /.

The pay TV systems are the worst thing that has ever happened to smart cards. There are numerous reasons why they're insecure, and those reasons do not apply to other systems, particularly not credit card systems.

First, pay TV engineers have an essentially intractable problem. No security system is unbreakable and the problem is particularly difficult when the attacker has unlimited access to key components of the system (as is the case with all smart card systems). However, the pay TV guys have the additional complication that satellite television is *broadcasted*. That means that every card is doing the same thing with the same keys and the same data, which in turn means that breaking one card essentially breaks all of them (or at least a large number of them).

To try to manage this risk, the pay TV designers use flashable cards and a security-by-obscurity approach. In most environments flashable cards are a very bad idea, because you want to make sure that attackers cannot fiddle with your software. In this situation, it permits the system designers to modify their system frequently and remotely, to make it obscure again and shut down all the existing pirate cards. OTOH, flashable cards are easier for attackers to fiddle with. The result is that every update is broken within a few months, but most breaks are disabled within a few months, and that keeps the whole thing in the news all of the time.

Nearly every other smart card system can take a different approach, one in which (a) the card is expected to be secure for a few years and then replaced and (b) the break of a single card does not compromise the whole system.

Card-unique keys, key rotation, audit mechanisms, hotlists and moderately frequent on-line validations are all tools used by careful designers to ensure that breaking one card does not break the whole system, and that broken cards can quickly be detected and shut down.

When you add those tools to the security measures offered by the card itself (such as S/D PA resistance, environmental monitoring, security-sensitive layering, chemically-similar protective shields, etc.) the result can be a highly secure system.

The job of a smart card system designer is easy to state: Design a system such that the cost of breaking a single card is much higher than the value of breaking that card. A good estimate for the cost in expertise and equipment to break (extract the secrets from) a single, modern, well-designed card is about $250,000, give or take a little. As the card ages and new attacks are devised, that cost goes down. Meanwhile, card manufacturers make changes to defeat the new attacks, raising the bar again for new cards.

For credit cards, this is an easy thing to do. Not only are the cards unique, used on-line frequently (whether at home or at a merchant) and replaced regularly, the possessor of the card is generally motivated to maintain the secrecy of the information in it. Further, the job of the card is identification and authentication, rather than decryption, which is a completely different problem from that of pay TV cards.

Secure smart card credit cards are easy for a variety of other reasons that I won't go into, but perhapes the most important is that there is already a significant and mostly workable security infrastructure in place that doesn't need the cards themselves to provide any security. This model breaks down to some degree for on-line transactions and a chip card is a perfect way to close that loophole.

It's true that nothing is completely secure, but it's also true that nothing has to be completely secure. Success in security is when the attackers decide to find an easier target.

Google?

[cetan]

I don't know if this applies or not, but what about the rise in popularity of Google?

Clearly they came in late, after all the other search engines were established, but they ate up market share and eye balls because they were (and are) better.

So I could imagine that there are some Amazon-killers out there or that at least there could be...

Re:Google?

[dthable]

I think the key is that a bookstore, like Amazon, doesn't have a nice single location for data. All of the book information needs to be input somewhere. Google can just create a robot to go collect information. I think this single difference makes an Amazon-killer more difficult than a Google-killer.

Re:Google?

[selderrr]

I really hope there are a lot off Googles out there to kick Amazons fluffy butt : Google has proven that ad-free online information (and in a way commerce) actually works. Works better even than the ones with Ads.

Amazon gives me headaches with all their fsking commercials and screaming colors. I don't go there unless I really can't find my info anywhere else. The opposite with Google : their layout is so nice I sometimes find myself just googling away for fun... like the old days when the internet was still fun to explore.

SOC

[dave_mcmillen]

I have a horrible feeling that the words "self-organized criticality" are heading inexorably towards this discussion. Please, just say no.

No, seriously: it doesn't look like the article is making that particular connection, thank goodness. Per Bak's theory of self-organized criticality predicts power law distributions of many things under many conditions, so people (or, specifically, Bak fans) often get all excited whenever they see a power law and start saying, "Hey, it's SOC!" But there are many ways to get power laws, and of course the logic doesn't work: "SOC gives power laws; thus if it has a power law, it was caused by SOC."

If you want to know more about SOC, you can check out Bak's modestly-titled book, How Nature Works.

Feedback Loops & Linking

[inKubus]

Well, in "Cybernetics and Society: The Human Use of Humans" by Norbert Wiener, the author talks about messages--communication. Links to a web site are messages given to people using the web that a given site/page/whatever exists. They are easy to make use of, since all you have to do is click it and whatever is on the other end is given.

This is all fairly obvious. The neat thing is how these messages and the messages they point to interact. Dr. Wiener says that the more unique a message, the more "important" it is. This is simply because an overused message (a cliche) quickly becomes filtered by the human mind, and loses its meaning.

Take the Amazon example, for instance: How often do you click a LINK to Amazon? Yes, there are hundreds of thousands of links to Amazon, but I would guess most of them NEVER get clicked. Why? Because there are too many of them. The first time I saw one, I followed it, but now I just ignore them. I almost never click on Amazon links because I know it goes to some bookstore.

When I do go to Amazon, I just type the DOMAIN name into my browser, and go directly there, and do my own searching, follow my own links.

So, basically, there is an upper limit to the number of links before they essentially become useless. Of course, this upper limit is dependent on the total number of users who haven't seen the links, which is increasing every day as new people come on to the web. As the number of links reach this critical mass, more and more people are just typing in the domain name rather than following a link.

This is Google's essential flaw. It does not recognize that a site like Amazon does not need an entry in a search engine. There are enough links out there already for just about anyone to find it. Google should instead group searches around a bell curve distribution, where the sites with the medium number of links have the highest relevance, with underlinked and overlinked sites falling off the ends.

How are new sites found out about and linked enough to show up in an engine like Google? Advertising. Mostly word of mouth and link ads, and in certain cases print and television advertising (although this is less effective, because it requires the user additional steps to make use of the message (ie: remembering the domain name at a much later time and then typing it in), which is why the .com ad explosion on TV failed to do anything..)

Really, to be effective, you need to have 10-20 contacts online, have each link to your site. Spread the word as much as you can. And save your ad budget until your word of mouth traffic reaches critical mass. Then spend it on bandwidth.

Really, time is the only key. Oh, and having something useful or funny.

Anyway, this quickly turned to the theories of getting a lot of hits, and I apologize, but you can see that the middle is the best place to be, and maybe Google will recognize this. This would do a LOT for online commerce, and the economy in general. Support bell curve relevance.

Cheers.

Some misunderstandings here

[dipfan]

Researchers have discovered that across the entire web, links are distributed according to a "power law" which leads to "rich get richer" or "winner's take all" behaviour

Ah, no that's not what it shows:
Quote from the research: "In fact, pure power law scaling appears to be the exception, rather than the rule." What the research shows is that "winner takes all" varies across the web between categories.

Also, the researchers have (I suspect) a mistaken view of "competition" and competitiveness. They rate the Amazon category of book-sellers as "more competitive" - when in fact it may be less competitive in economic terms, being dominated by one or two sites/sellers. Whereas the photography caregory may be "more competitive" because of a larger number of rivals of about the same size.

What they mean, I suspect, is that the publications/Amazon category has higher barriers to entry - the amount of adertising etc being a greater sunk cost, and likely to deter any aspiring internet book retailers. In purely economic terms, that makes the category less competitive. As an illustration, ask yourself if the operating system software market is more or less competitive because it is dominated by one large brand in Microsoft?

Booksellers vs. photographers? Bad example.

[Crash Culligan]

Another thing to consider is that booksellers have a rigid stock they can deal in: books. The one book that a seller gets is the same book that ever seller gets, and so they have fierce competition for eyeballs to sell you their copy of that book.

They may specialize and carry more of certain kinds of books, but you won't see much difference between The Hunchback of Notre Dame as carried by Amazon, Barnes and Noble, or Borders.

Meanwhile, each and every photographer has his or her own unique vision which they commit to film. Two photographers can snap pictures of the same subject, but composition of each shot would be different.

Booksellers usually sell the same goods. Photographers usually sell their own product, which can't be found anywhere else.

The article doesn't match my experience.

[Futurepower(R)]

It seems to me that the article is nonsense. There are a lot of links to Amazon because Amazon is running a very active business in selling books.

CmdrTaco said recently that efficient search engines like Google make the web flatter and more democratic. It has been my experience that this is true.

In search results, my book, What should be the Response to Violence? is often ranked just below stories from large news companies.

If you search for "books", you will find Amazon. If you search for something more specific, you may find a small, specialized bookstore.

The rich still get richer, but those who are not rich can now be heard.

Re:Gee, Name recognition should have some bearing.

[sugrshack]

absolutely. Amazon understood a basic principle of commerce and marketing... BRANDING.

Most people associate even the concept of purchasing something online with Amazon... they were pioneers, and managed (using linking, etc) to embed themselves in the mind of the consumer. I must admit; even though I may like to think of myself as one who resists marketing, when I need to buy a book online, where's the first place I think of going? Amazon.

Insidious, ain't it?

I am the village idiot i don't have anything to do with this pathetic little opera i just felt like passing through... - PDQ Bach