Search Engines Leech Value from Web Sites

bigenchilada writes "Jakob Nielsen, former Sun Distinguished Engineer and now usability guru, proposes "that search engines are sucking out too much of the Web's value, acting as leeches on companies that create the very source materials the search engines index." He says that the value provided by search engines may be tilting too much in favor of the search engines. The web sites that create content are now simply fodder for the search engines' revenue stream."

Comments

more evolving and changing business models

[yagu]

This "tilt" intrigues. It's interesting in that it describes an unfolding and evolving business model to which companies must react.

I like that he doesn't just whine about the problem but offers solutions too, to provide the "stickiness" required to keep customers coming and interacting with companies' sites. This is the way the evollution should work.

Oh, that the RIAA and the music industry would have to abide by the same principle now that their business model has changed, rather than buying legislation to cripple advances in technology (which, btw, will NOT work).

Maybe, maybe, the music industry could learn something from this.

Re:more evolving and changing business models

[DesertBlade]

I agree that copyright laws need to change I don't think they should go away.

People spend time and capitol to invent something, a book or a new type of fuel. It might cost millions to develop a new medicine and if there where no copyright laws someone could copy that product and sell it cheaper (because they do not need to recover the startup costs). Now if every time you spend your money and time to create something just to have someone copy it really kills the incentive to create it in the first place.

Re:more evolving and changing business models

[RingDev]

"If you create content that is mass produceable, don't give out all your answers in that mass produced content. Hold some back, hold the most important parts back, for one-on-one or face-to-face interaction!"

The problem is not mass production. VCR remotes are mass produceable. Toy cars are mass produceable. Silicon chips are mass produceable.

And e-book is infinately producable. For $0 cost I can share an ebook with the entire digital population, whether I wrote it or not. Sure, hook people with a cliff hanger story and sell a seperate episode. But at some point, someone will transcribe the book to a digital format and it will be distributed. Encourage people to come to your shows and presentations, and they will be recorded and distributed.

I mean, we are looking down the barrel of the end of the supply/demand market in the digital media content market. Face-to-Face interactions are one way for content creators to get by, but what about those reclusive authors who write amazing stories but have the public speaking ability of a def-mute.

I'm not saying we should fight to keep the existing system, but I am curious as to how the market will shape up, how content delivery companies (RIAA, MPA, etc) will shake out, and how it will all effect the original authors.

-Rick

Re:more evolving and changing business models

[Syberghost]

I'm working on a deep article about banning copyright (I'm opening a 6 figure music studio in Chicago this spring called No Copyright Studios) for artists who want to succeed without the cartels and their lifetime monopoly granted by government.

Excellent. With no copyright, the big music companies can just copy your recordings and sell them, without sending one dime to the artists. Sure, your artists can fight back by selling their music online, but the big music company will get all the CD sales because they can produce and distribute them cheaper than you can.

Re:more evolving and changing business models

[cherokee158]

A book author typically makes less than five grand for a GOOD selling book, which can take months or even years to produce and get published. Copyright law helps to insure that he makes royalties above and beyond his advance(sometimes), and makes money from resale rights. You business model would see him lose a lot of that money, and spend valuable writing time touring the country signing books in exchange for his pittance. (Which writers rarely get paid for at all right now, beyond expenses...it is normally part of their contract that the publisher has the discretion to request them for special appearances while promoting their book.)

Most bands working the clubs make a few hundred dollars a night, which is split among the members and their crew and rarely adds up to much. Big name acts make millions, but booking their huge venues is not cheap, either, and a LOT of people get their fingers in the pie. They often make less than you would think.

Commerical artists get paid very little for their work, unless they are (once again) among the top tier of a very competitive industry. Stock art and photography have put increasing pressure on their revenue sources, as most publishers look for low cost solutions to their graphic needs. Art has, and always will, take time to produce, and a single sale rarely justifies the time invested in a work of art. Copyright law enables the artist to offer different rights packages to different clients (first North American serial rights, reprint rights, exclusive rights for s particular sales medium, etc), which can help offset what would otherwise be a clear loss.

It's that, or the galleries.

All of these professions exist because copyright law makes them workable.

Copyright law could stand to be loosened (specifically, it's duration needs to be shortened, and it needs to be less retrictive regarding derivative works), but abolishing it altogether is not such a good idea from the standpoint of most of the truly creative people in this country. It's hard enough to make a living as an artist, musician or writer now. We'd have to put them on welfare if we abolished copyright law altogether.

Everyone here seems to asume that copyright law is just the lapdog of large corporations and overpaid celebrities. In some ways, it is. But for every band of thugs like the RIAA, there are dozens of little guys ekeing out a bit of money from the arts who really need that protection...mainly from parasites like the RIAA. (Most people are too decent to rip you off, but corporations will rob you blind in a heartbeat.)

What you propose is intellectual socialism. I think we've all seen just how well things work out when "the people" collectively own all the property.

No thanks.

Re:more evolving and changing business models

[griffjon]

I feel that he's avoiding (not missing, mind you) an underlying problem of conflicting interests:

User: I want information on X
Search Engine: I want to give users information about X and advertise services related to X
Websites: I want users to become involved at my website that contains content about X

Users want a quick answer, Websites want them to spend some time, sign up/login/register/ignore the "subscribe to newsletter" checkbox being pre-checked, whereas Search Engines want to provide things that look like the answer as best/fastest as possible (and also throw some ads around it). If websites don't want so much leeching from search engines, they must become better known, get a solid brand and offer good and complete information.

A good example -- If I want information about a perl script, I know from experience and recommendation that I can go straight to perl.com or perlmonks and probably find the best answer, and more focused than a google search will generally provide. However, if I'm trying to find help about Random Microsoft Bug #8000436531, experience recommends that I avoid Microsoft.com (which you'd think would be the logical choice) as google will generally return more useful answers and **solutions** and viewpoints, whereas MS will provide only the MS-recommended approach, which may or may not take into account other issues (I love it when the help guide tells you to use a menu feature that's just Not There -- very helpful, thanks)

So, I'd recommend websites, if they're complaining about search engines sucking their users away, should ponder if there's a reason why the user would want to stay at their site -- is it comprehensive? Do you expire/charge for content? Do you require annoying registration? Move all that further back! Make content acquisition easier, and users will want to go straight to a known-good source rather than sifting through Search Engine results.

It is a symbiant relationship

[dada21]

This is likely why Google and Yahoo are offering monetization options for content publishers (and creators). Plus, if you don't like search engines "leeching" from you, just set up robots.txt and say no to everything -- they'll go away.

I find that search engines account for nearly 70% of my visitors overall, and account for nearly 60% of my return visitors. I don't believe I can rely on my websites to generate income for me (even if I start selling more products on some sites). As I don't copyright any of my text (I am anti-copyright and put all my creations into the public domain immediately), I use my writings to try to increase my income in my regular life -- speaking engagements, one-on-one consulting, and professional advice to companies and individuals in the markets that I'm valuable in.

Nielsen is nuts if he thinks that the web should scoff at search engines. Search engines are (to me) the biggest reason for the web's overall explosion. Bookmarks help, links from other sites are great, but Google, MSN and Yahoo are the big reasons people can find what we want when we want. If they can't index our sites, how can they send us traffic? Sure, he acknowledges this in his article, but he says that web sites are going from information stores to answer engines. This is completely true, and we all fall victim to our own stupidity when it comes to creating content in an "answer" fashion. I've been working over the past few months to try to create extended interest in my most popular pages (found via search engines) by offering crosslinks to other articles. The longer I can keep the people interested, the more likely I am to see them come back again and again. If you make old "answer" pages, offer links out of those pages that give people MORE information, or give them more questions to find answers to.

Content is worthless without distribution. I prefer face-to-face distribution for profit by using more generic information to "catch" the customer who will hire me. Yet without the search engines, how will I get the word out? Hire a publicist?

Slightly off-topic here:
I think its crazy to put quality profitable information on a website (or even in a book, on a CD or in a movie) that you don't want used by others. Copyright may "protect" you from someone knocking it off in high quantity, but that isn't always where information is the most valuable. Using information in an expert situation is how you can turn quality profitable information into that quality profit -- by selling your advice on a person-to-person level (I call it a performance).

Re:It is a symbiant relationship

[Irish_Samurai]

Offtopic about offtopic.

I think its crazy to put quality profitable information on a website (or even in a book, on a CD or in a movie) that you don't want used by others. Copyright may "protect" you from someone knocking it off in high quantity, but that isn't always where information is the most valuable.

Finally. I have been listening to your anti copyright tirades for some time, even took part in a few, and have never fully understood how you could hold that opinion.

Now I understand. I personally stick to that same practice. My most valuable information never gets published, it is transfered through consultation. Also, if the party I am consulting has any sense, they shouldn't need me to consult on the same subject twice.

After the information I use is no longer of a premium value, I will begin to publish it as a means to prove credibility for my services. Plus, by that time I have already learned something new and would be using that.

Only a few annoying sites...

[Godeke]

I read this article when it went high on the del.icio.us/popular list. Long story short: this guy is complaining about *advertising* links in a search engine. Then he goes and compares a bunch of apples to oranges and concludes the sky is falling (yes, I meant to mix metaphors, as this is what this guy does in his complaint).

If you look at his analysis, he is coming from this from a perspective that most of the Internet can't really related to: a business to business commerce site that uses no advertising revenue and pays a high "click-through" cost for each visitor from a search engine.

After all of those constraints are in place, he further comes up with the idea that by making $4 per visitor (after COGS and conversion rates) "the site can pay $3.99 per click". Well, I guess if you really are hellbent on giving your profits away you could...

He tries to justify this by saying that "if you don't pay this, other sites can outbid you". He justifies this by saying that others will use his sites methods to improve conversion rates and therefore they will outbid you with the increased revenue. Well, maybe, or maybe they will keep some of the profit.

This commentary is not applicable to those with advertising supported models, nor those who are willing to differentiate themselves by more than hyper-competition in search engine optimization. Which means pretty much most web sites are *not* going to see the results that are predicted here. The ones that *will* see this are those that don't have a differentiator and live and die by the converted sale. I think I will cry now... [sniff]... poor toner refill sites.

His solution: #1, spam the user. #2, notification spam. #4, multi level marketing.

Search Engines Are...

[MorderVonAllem]

...Free. I don't pay them to index my sites, and they send me potential customers. Somewhere around 80% of my traffic is related to search engines. Sure, they're getting money from advertising other sites that may compete with you, and they don't produce content of their own. But at least I don't have to pay 400 dollars for a page in the yellow book and reach a whole city of customers when I can have my site indexed and reach a whole world. (unless they don't speak english, then they can't get much value from my site)

Food for thought.

[Concern]

The author's point is, in a nutshell, that web business are reducible to the cost of their traffic and the revenue it can generate.

His example is something like this: 100 users with a 1% conversion rate for a $4 net profit means you pay $3.99 to the Search Engine for traffic to make 1 cent. Since the search engines are effectively a traffic auction, you always pay exactly as much as your competitors are willing to pay plus a small amount...

I find fault with this argument, because search engines are not a traffic auction, exactly. Google sells adwords but it primarily gives users what they ask for, not what others pay for. Still this is the reality and the mindset of many online businesses, if there are 10,000 other companies like yours, you can only be seen by buying traffic.

His concern is that the search engines' position is too strong - they're the bottleneck, and they price like it. They've created a market where they take most of the profits from any online enterprise. If web businesses find a way to increase margins then it instantly translates into increased search engine fees rather than increased profits, and google earns it by sitting back and "doing nothing."

Of course, they do something, but just like Sony and Tower records, their indispensability may have been converted into a disproportionate amount of the profits of global enterprise.

From 20,000 feet, thinking in a way we seldom do anymore, we could consider alternate regulatory regimes that might tinker with the market. For instance, if you accept that this state of affairs may not be optimal (a few megagiants and millions of small businesses beholden to them), you could flatten it by reinterpreting things like copyright, so that the search engine is not entitled to list anything without splitting a cut of the profits of that enterprise with the content creators.

I'm not actually suggesting this, just trying to seed discussion. One thing that this vaguely reminds me of is the Neal Stephenson concept of the free-market encyclopedia, where anyone can write anything and upload it into the system, and then you get paid, more or less directly, for traffic... presumably by redistribution of fees paid into the system to view content. It's appealing in the way it incentivizes creation of content, especially in such an egalitarian way.

We've got an all-you-can eat model where you pay for access and others pay to publish and writers can pay the rent with advertising or subscriptions... and of course, we have a free market for search services... I like it well enough, but I do sympathize with content creators, who still seem to struggle to realize the value of their intellectual works.

Please don't

[jmc]

"He says that the value provided by search engines may be tilting too much in favor of the search engines. The web sites that create content are now simply fodder for the search engines' revenue stream."

Yes, and this is exactly why everyone I know in the e-commerce business spends an exorbitant amount of time trying to figure out how to prevent their site from becoming "fodder" for Google's revenue stream. Because, of course, Google brings absolutely no value what-so-ever and and does nothing to drive traffic.

Riiiiiight....

Lame

[hackstraw]

No wonder Sun has so many problems. They used to be the "dot in dot com"
until IBM took over that in 2000 or so*.

The value of the web is priceless (and free!), how could a search engine
to find stuff on it decrease that value?

This is the one of the most silly things I have read since Taco said the
iPod was lame.

TFA says, "We've known since AltaVista's launch in 1995 that search is
one of the Web's most important services."

Then, "There's no doubt that search engines provide a valuable
service to users. The issue here is what search engines do to
the companies they feed on -- the companies that fund the creation
of original information. Search engines mainly build their business on
other websites' content. The traditional analysis has been that search
engines amply return the favor by directing traffic to these sites."

I use Google for everything. I never type in a blind url, because I make
mistakes from time to time and get some typosquatter or other troll.
What I do is go to the Google box next to the url box and type something
like "barns an noble". Notice I mispelled the name, but even with my
error, the first linke was "Barnes & Noble", which is what I was
looking for. (DNS is already dead because of this, Google is the dot in
dot eveything).

I would have no idea that the url would have been barnesandnoble.com.
Many users don't know that a & is not a valid url character, and
they would mistakently put it in the url if they were to blindly type
it. Most web browsers would give a worthless error message like "The
specified server could not be found." Thanks. I used to run a web proxy,
I've seen everything in the world typed into the URL spot.

What Google and other search engines have done is flood the market with
worthless, fly by night companies. Stable ones have no issues. Search
for Apple computers, or Oracle database, you get useful links. Search
for a commodity item you can get anywhere for the same price, and you
get every sleezeball in the world trying to get your $25, when you could
also just have walked to the store and got it for $30, and played with
it that day.

Niche items are different. I can find them via Google. I have a nice
whip cream dispenser that had the rubber grommet freeze because I was
making so much whipped cream (right). I paid something like $40 for it.
I wanted a new grommet for it, and I used Google to find a store in
Pennsylvania that sold me the grommet for something like $2. I ordered 2
boxes of whipped cream to offset the shipping, and in less than a week I
was back to making whipped cream!

I could have never, ever have found a $2 part without Google.

Google is a monopoly for a reason. Why would you need more than one
place to find all of the questions you have?

* Sun boxes used to run the root DNS servers, and that is were they got
the idea that they were the "dot in dot com". IBM took over that role in
2000 or so, I'm not sure if they still have it or not.

Search engines ADD value

[jridley]

I find that google blows away most commercial sites for finding content on those sites. I use google when looking for products on web store sites. In that respect at least, google is doing them a favor. I'm finding products I want and buying them because Google is there; many of the sites have crappy or nonexistant searching capabilities. Heck, I've tested some; I can be looking at a product I found on their site, and I can type in the words that are in the name of the product into the site's search, and it will tell me there's no such product. Ridiculous.

You need to keep working to earn money. So what?

[zaydana]

After I RTFA, I basically got one thing out of it. The writer was complaining that unless you keep working at a website you made, you are going to not earn as much money.

I've got news for him - you can't expect to earn money out of nothing. I know some people manage to do it, and good for them. But its not something you can expect to do. If you don't work on improving your site, and others do, its not the search engine companies' fault that people will be more interested in their work and so they can afford more on advertising.

I do see his point - the search engines will get paid more because your work improves, and other's work improves. But this is not something that is unique to search engines. It is part of advertising in general. The larger a company gets, the more it can put into advertising, which means the competing companies need to keep up with them and put more into advertising themselves. It works in a bit of a different way, but its the same concept.

It doesn't matter what advertising it is - TV, Radio, Newspaper, Search Engine - the way the companies make their money is the same. Google, microsoft, yahoo, etc. are not doing anything new here, they are just brining proven concepts to a new medium. Why do we critisize them for it? If anything we should be critisizing the people who have drummed it into many a programmer that once you've written something, you shuoldn't need to maintain it.

Simple economics

[nonlnear]

The author of TFA is more than a little ignorant of simple economics.

In the long run, every time companies increase the value of their online businesses, they end up handing over all that added value to the search engines. Any gain is temporary; once competing sites improve their profit-per-visitor enough to increase their search bids, they'll drive up everybody's cost of traffic.

This is a simple fact of economics: There is no profit in a competitive market. (That is the economics definition of competitive - not the pedestrian definition.) The point is that you have to differentiate yourself from the competition in order to (successfully) charge a premium for your product - either through website improvement, or having a different product that you're selling.

The fact that the proliferation of auction models has made many markets more competitive is a fabulous thing. If you draw the conclusion that the author should have drawn, it becomes ainfully clear: search engines make it harder to be a retail "squatter" and make money. (That is, to run a site that doesn't have any innovation in either site design or product.)

There is one valid economic objection that the author could have made (but didn't). That is that the web advertising market is asymmetric. Google has a near monopoly (AFAIK), which allows it to extract (close to) the full consumer surplus for the ads it sells (the site makes $4 per visitor. Given these assumptions, the site can pay up to $3.99 for each click on its search engine ads...) This wouldn't be the case if there were substantial competition for Google (and I might honestly be wrong about the lack of competition - I just haven't heard of any).

My point is that there might have been a substantial argument to be made about the search market, but if there was, the author failed to make it.

Indexes versus content

[smbarbour]

I don't know about anyone else, but I only use search engines to find the site that has what I am looking for. I don't go around typing random URLs in to try and find a site. I type what I am looking for into the search engine, and it responds with a list of sites that match my criteria and the context (usually) of the site. If the context I'm looking for isn't on the first page, I refine my search.
You wouldn't do research in books looking only at the index (or even a library's card catalog system) would you?

Without search engines..

[_LORAX_]

I will probably never find your site... so go ahead and block search crawlers if you feel you are getting screwed.

The plain fact of the matter is there is SO MUCH data on the itnernet as to make it nearly impossible for me to find your site by chance without search engines indexing and suggesting the content on a related search. I don't have time to go and independantly discover your site when looking for a topic or product, if it does not come up with a few google searches it effectivly does not exist and you get 0 revenue from me AT ALL.

Re:hahahahaha

[aztracker1]

"What about all the filthy, filthy porn?" -- RvB

Re:Search Engines = SuperMarket Tactics

[Zphbeeblbrox]

You also have to be aware that those Search engines only stay on top by being the best. If google starts doing what you suggest then MSNSearch or Yahoo will be more than happy to take their market share. The Internet is the closest thing to a pure "free market" economy we have. The product is information and the currency is our attention. If you forget what kind of economy you are in you'll fail, whether you're Google, or Microsoft.

Google doesn't operate in a vacuum. No matter what you might think.

I'm failing to see...

[RingDev]

Where you turn a profit. You mention Thousands of dollars over years. $2,000 over 5 years isn't a profession, it isn't even worth reporting on your taxes.

From your post it sounds like you are saying content is worthless, but the author has value. But to me as a consumer, I couldn't give to shakes of a monkey's ass over who wrote my SQL Bible, but it's content has saved me hours of head banging frustration.

Why would I pay $20 to meet the author of the book when I already have the book? Why would I pay $20 for the book when I can get it online for free? Why bother with the ebook when I can use Google to get the same information faster?

-Rick

What about people who don't think like you?

[Mister_IQ]

very thing we record in the studio will instantly be public domain. We'll be watching for others to take the content and redo it, and then we'll be able to use that content as well for our own gain.

What if they copyright their work?

This idea sounds great in a world where everyone who hears or uses copyright free music contributes back to the cause, but that's not the case.

By "public domain" do you mean some sort of creative commons-like license, or are you really going to produce it and then let it be used for whatever?