Slashdot Mirror
Web Caching: Google vs. The New York Times
An anonymous reader writes "The Google cache is a popular feature among karma fetishists. Many stories with links to the NY Times attract comments pointing to Google's copy of the article. This gives readers access to the content without registering. C|Net reports that Google is in talks with the NY Times to close this backdoor. The article raises some general concerns regarding the caching of webcontent. Shouldn't the NY Times simply tell Google not to cache their site?"
I'd love to see their user database, just to count the number of Mickey Mice and Elmer Fudds on there. Apart from giving the NYT your e-mail addy for spam purposes, what real point is there to free registration?
When I am king, you will be first against the wall.
Now we can't karma whore by linking to the google cache?
Shouldn't the NY Times simply tell Google not to cache their site?
You do realize that this is probably the basis of the "talks" that are going on, right? C|Net (as per usual for them and every news agency) is making a big deal of it to get themselves and their advertisers that tiny wee bit more of attention. Every little bit helps i guess.
Check http://nytimes.com/robots.txt in a week.
The reason they're trying to stop this is because with NYT reputation, they keep retracting stories all the time. With Google cache this could be problematic and the management/editors/authors could get into trouble again.
I do however dislike Google cache for many reasons. It's bad for privacy.
- When will slashdot stop linking to articles that require a registration?
- When will slashdot consider implementing caching for pages that, by linking to, they manage to take off the internet?
Sure, the 2nd question has been answered in the FAQ. Except it was written three years ago and Google manages this just fine. Maybe time for a second look?On the topic of site updates, has anyone noticed that 90% of the links on http://slashdot.org/code.shtml don't work any more?
Hell the link to an Avantgo version of Slashdot points to a website which has been broken for over 2 years.
Avantslash - View Slashdot cleanly on your mobile phone.
As the poster mentioned, Google already has a way to opt out of caching, so "talks" sounds like this is something different. My guess is that Google will become an affiliate of the NYT (in other words, if you hit a NYT link from Google, you're exempted from registering), and will then drop the caching.
The article talks about Google's caching of articles that have expired to the NYT archives (which you have to pay to access). What most /. folks use to link to current NYT articles are the Google partner links, which simply bypass the free registration. I'd assume these links only work as long as an article hasn't been archived yet, so the karma whores are safe; I doubt the NYT's Google partner links will be going away any time soon... ;)
DennyK
It's worth remembering that newspapers sometimes edit/remove articles they publish on their homepage. Without a Google cache it may be much harder to verify that a story has indeed been modified.
Don't you just hate it when promising new technology is curbed by outdated laws?
Here in Denmark we had a service similar to news.google.com for danish newspapers. The newspaper organisation sued the service for parasiting on their databases (which is prohibited in Denmark). The service was shut down half a year ago and we now don't have that kind of service anymore.
Of course newspapers should be allowed to publish their stuff without others copying it but they refused to even use a "robots.txt" (which the news service respected) to stop indexing.
If you publish your stuff on the internet and don't tell people that they should not index it, cache it or what do I know - then you better expect them to do that. Let us put those lawyers back where they belong.
The reason that the NYT just doesn't tell google not to cache them is visitors. Let's face it, even though the registration is a bitch the content on the NYT website is fairly decent. They have good articles often enough that geeks went through the effort of finding out how to read without registering. If they have google not cache them, and they close the google news loophole, then they wont appear on google news any longer. And google news is used by many more people than you think.
Hey, we get quite a few visitors from this google news. Let's change it so we get 0 visitors from it.
Duh.
The GeekNights podcast is going strong. Listen!
the nytimes website needs google for the traffic google brings into their pages, so they can't turn away their spiders. but then, they don't want the spiders either because of copyright violations. why should this be google's problem anyway?
Actually, free reg requires a valid email id. It thus filters most bogus registrations. Secondly, news sites are planning to go the 'pay' way in about a couple of years. Getting readers to register would give more accurate estimates of readership.
And lastly, once a site requires registration, even if free, Copyright ptohibits quoting entire articles on the web. This indeed could be the prime reason for this.
If you keep throwing chairs, one day you'll break windows....
The Internet Archive, which I just used minutes ago to find a handy page removed years ago, is an interesting corollary to the Google cache. I often wonder how it has survived thus long without a major lawsuit. It also reminds how crappy the web looked 5 years ago.
At any rate, cache-ing is an important force on the internet, and isn't one that should be limited in any legal way, including litigation.
You are the new editor of the New York Times, the "Newspaper of Record" for the United States, if not the world. You are, of course, the new editor because the previous editor had to resign, taking the blame for an individual reporter's flagrant disregard for the awe-inspiring credibility of your institution. In the process of rebuilding your credibility, should you:
A) Insist that unaffiliated digital libraries restrict access to or simply eliminate all records of your "Newspaper of Record", or
B) Realize that maybe right about now is not particularly the best time to be saying to the world, "Please forget what we published last week."
"Google News is highly unusual in that it offers a news service compiled solely by computer algorithms without human intervention. While the sources of the news vary in perspective and editorial approach, their selection for inclusion is done without regard to political viewpoint or ideology. While this may lead to some occasionally unusual and contradictory groupings, it is exactly this variety that makes Google News a valuable source of information on the important issues of the day." source
Remove your tinfoil hat please. There is no conspiracy. Google News features articles from Newsmax, Electronic Intifadah, Islam Online, Al Jazeera, World Net Daily, etc. If there was any filtering going on, these sites would have been off the radar long time ago.
Also, Slashdot is not a professional journalistic site. It's a News-based comment board where people come to share their opinion. In a perfect world Slashdot doesn't even belong on Google News.
Actually the NYT has already begun using google's NOARCHIVE option to prevent content caching. Here's an excerpt from the this morning's front page story's source:
i al/14IRAQ.html positions: Top5,Middle1,Right3,Middle5,Right,Travel7,Travel11 ,Bottom1A,Bottom3A,Right5,Right6,Right7,Right8,Bot tom8,Bottom7,Inv1,Inv2,Inv3,Frame4,Right4 kwds: politics+and+government;international+relations;ir aq;suggested%5ftopnews;suggested%5finternational;s uggested%5fworldspecial;suggested%5fmiddleeast --
!-- ADX SETUP: page: www.nytimes.com/yr/mo/day/international/worldspec
meta name="ROBOTS" content="NOARCHIVE"
Kind of makes me wonder what's the point of the story, since it even says there's an easy way for concerned parties to opt out of the cache.
In case the cnet is /.'tted, here's link to Google cached page.
That's the thing - it's not free depending on your definition. By my own definition, you're giving them valuable information, and they get to keep it and use it as they will, including spamming if they feel like it (or spam from any company which buys them out, they sell it to if they're feeling bankrupt, etc). It's practically misadvertising of a service, but it's accepted now, so everyone gets away with it.
If it really were free, why would you need to register in the first place?
Trying to sell web pages is like attempting to sell mp3s on a p2p services where all mp3s are free.
IT wont work. Instead you should use your websites to market and sell your magazine subscriptions.
Like Wired.
If you use Linux, please help development of Autopac
Ecspect? Ecspect? ECSPECT? Jesus H mother of god, it's like just when I think slashdot couldn't possibly get any worse... it does.
First off, Google News is still in Beta at the moment.
Second, the Google News database only goes back a month or so, probably by design.
Third, I was able to search for 'site:slashdot.org microsoft oregon' on Google just fine this morning. Got 243 results, and the Google Cache has copies of the first three pages returned, which relate directly to the Oregon bill you use as your example.
So, where is the problem?
-MT.
In case google gets /.ed here is it's google cache
w ww.google.com/webmasters/3.html+%22I+need+my+site+ information+changed.%22&hl=de&ie=UTF-8
http://www.google.at/search?q=cache:qw2H0d95VkEJ:
--
Karma 50, and all I got was this lousy T-Shirt.
Brand recognition is not always a good thing. When I think NY times I think "that annoying registration website". They are free to do what they want, but it leaves me cold.
My Karma: ran over your Dogma
StrawberryFrog
Here we have the NYT, one of the premier news organizations in the world, offering its articles for free on the same day that they are published. Yet a large number of people, of this online community at least, refuses to provide even a minimal amount of information (and no money) so that the newspaper can try to make its online presence profitable.
I think the spam fears are a red herring, I've been registered with the times for over 2 years. I've never gotten spam that I think is traceable from them. I get a daily email of the day's headlines (and with the click of a box I could discontinue this).
Why should the RIAA change its business model to a pennies per song method when there is such a blatant example of the online community refusing to go directly to the source for even free material?
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
Actually, registration is not required to protect a work. Creating a work automatically protects it under copyright law -- no need for registration, user fees, or that little (c) thingy. At least in countries respecting the Berne Convention.
The Mongrel Dogs Who Teach
the facts are a commercial company A (google) are making a profit from unauthorised copying of other peoples content without permission , meaning company B (you) has to spend money (webmaster) or take proactive steps to remove your content from their databases, google are not an ISP or a goverment agency so really they have no buisness in taking without asking other peoples content.
I don't know what planet you're on, but I profit when my site is listed in Google. People spend an inordinate amount of time and money to make sure their site is listed in the best way possible. Are you going to exclude what could possibly be a huge source of revenue for you? But maybe you have some obscure site you don't want anybody to be able to search for. So, given the amount of time it takes to build even the simplest site, is it really that much trouble to upload a robots.txt file with noindex, noarchive, nofollow in it?
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
Just FYI, this behavior is due to the fact that Googlebot has a sort of "built-in" mechanism to ignore (or at least rank lower) forum-type sites. Since /. is primarily a "news headline and discussion" site, Google will not rank it as highly as one that seems to be more "on-topic". This is because there is no guarantee that any URLs or email addresses within the page have anything to do with the actual page content itself.
Outside of user posts, /. has little genuine unique content. It summarizes a lot of headlines; this content is not unique.
Other (large) factors determine the way Google ranks pages, including the "PageRank" feature. There are lots of documents about the way Google ranks sites, I suggest to check them out. The best way is probably to Google for it :).
Anyway, this is a bit more on-topic:
I highly appreciate Google's caching feature, and don't see how it can be taken as "bad".
This is what's "bad" about Google and what I expect that, at some point, will come to haunt them. For instance, if I want to get serial numbers without porn popups, I can usually search for something like "Office XP Serial Number Serialz Warez" or something similar. Within the first couple of pages, I will probably find my serial number in the text of the page description.
If not, it's on the page, oftentimes without a popup, since the serial/crack page itself is the one linked.
Want to find X-Win32? How about doing "* * * * * * xwin32*.exe" - lets get some directory listings containing this filename.
No doubt this proves that Google is more than just a search machine... but I think that their superior techniques will definitely come back to haunt them in the future. NYT is way off target with bitching about their caching features... you can turn this off easily, and there are a plethora of scripts one can use to break out of Google's cache and send someone to the main site (or, perhaps, login area in the case of NYT).
But, in other news, Google might need to watch out...
Kind regards, Devon H. O'Dell
www.majcher.com/nytview.html
Use it frequently and often!
Currently I see "Welcome, paohjjkmtpfd."
At Washingtonpost.com, they only want gender, year of birth, Zip or Country. Pick most randomly, but always use 1984 for birth year.
More and more archives are already going pay per view.Hardcopy newspapers can't be erased or amended to suit whatever powerful interests might be embarassed by the truth.
Web-based publications may not be immune to such protection if they are archived by one source.
To not allow independant caching of news is just another step closer to historical revisions and distortions.
I'm not trying to say that such a thing is inevitable, but it would make things a great deal easier for those who would be inclined to manipulate the public.
I've never been sent a single spam from the NYT. The reason they want this is for demograpics. A) it tells them who their web readers are, and B) it tells their advertizers who their web readers are. And it also allows them to show ads for products people would be most intrested in.
autopr0n is like, down and stuff.
Anyone else see the irony that big buisness feels that "Opt-Out" is a fair policy when advertising to thier customers by phone and Spam. But when google gives them an easy and accesable way to opt-out of thier caching system by use of robot.txt and the NOARCHIVE meta-tag that isn't enough for them and they feel opt-in is the only way to go.
iRepairIT - iPhone, Mac, & PC Repair
I like google's caching quite a lot. I use it almost exclusively these days before visitting the actual page (if I even get that far). Using Google's cached link has the advantage of:
:)
1) Speed... Google's cache is fast. If there's one thing that annoys the heck out of me, then its websites that take more than 5 seconds to load. This is quite annoying when its caused by javascripts, slow servers or popup ads when Google can serve me effectively the same page in under a second -- especially when I'm not even sure if it is the right page, the one I'm looking for.
2) Nice highlighting so I can quickly page down to whatever I was looking for (now if only Google blocked those Tripod background pictures which makes their cached pages unreadable..) Sometimes I wish Google made their highlight examples at the top clickable so it jumped to the first appearance of the keyword immediately.
3) Using Google's cached links usually blocks silly popups and other annoying stuff too many websites seem to incorporate these days.
Perhaps I'll make a proxy server which browses the web exlusively using Google's caching... word highlighting on all pages, fast browsing everywhere and working links to more cached pages... should work fine for any webpages below 100kB
As for the NY Times being annoyed with Google's cache, they can easily fix that themselves. Either that or Google's spiders are a lot smarter than I thought to automatically register themselves for the NY times. Furthermore, as far as I'm concerned everything that's publicly accessible on the web without some form of password protection (which would of course also block robots) should be cachable and archivable in whatever form you see fit. Respecting robots.txt is no more than a courtesy as far as I'm concerned. If you don't want your pages to be archived or cached or whatever, then by all means protect your page, or donot put up a webpage in the first place (I'm sure a thousand others will leap at the chance to fill the void).
--Swilver
well cant they just use meta tags to prevent archving of their pages
<META NAME="robots" CONTENT="noarchive">
from
http://www.google.co m/bot.html"
I wonder where this will stop. NYTimes might get google to stop caching the direct link for a certain article. That is fine. But it is just one more step to do a search in google for the article with a few keywords from the article. If any person has been good enough to save it in a personal page, discussion board (like traditionally done for articles likely to be slashdotted) or any other place, the google results will show it. Would NYTimes now want to restrict google from showing thses pages because of the copyrighted stuff. You will be amazed as to how many articles I find this way. Many of them are just excerpts but others are complete.
Another thing on a tangent was that I really do hate the fact that information is restricted for just one fundamental reason - if it is not commonly available then it cannot be linked to in most of my writings for they are going to be unavailable to the party that I am writing to. This is especially true if the writing is not immediate but is meant to be read a month or two later. This is also relevant to Bloggers who might make comments and refer to a link, only to have the links go dead because the content is space,time, or space-time restricted. I am willing to pay for reading the articles, but before I can write about them I need to ensure that they are going to be available to my common readers. And as in the Blogging or P2P scenario I am not sure if one person is going to read my writing or thousands so buying a license for them is illogical. And then, if they need to send it further, are they also supposed to pay ??? Basically, for me to be able to write, to build upon existing work, to look ahead standing on the shoulder's of the giants, I need to be able to pass on the information. I am adding value because I am couching that content in a context, but until I can freely share the underlying articles too, my product is stunted. I can reach narrow audience but can't reach the common All this is very good in developing software where you might negotiate a deal once in a while to include someone's underlying code, but not writing where you might be writing 10-15 articles a week ...
To see a world in a grain of sand, and then to step back and see the beach where the sand lies
nope Sir, you are wrong. Wired Magazine is indeed commercialized on Wired Website. Nobody talked about company relations, well, before you did. And I still see the Lycos bar when I am on Wired Magazine's Homepage.
(yes this can be compared with sex)
The New York Times wants Google to continue ranking their stories but they want Google to do them the special favor of only pointing to their registration page:
"We are working with Google to fix that problem--we're going to close it so when you click on a link it will take you to a registration page," said Christine Mohan, a spokeswoman at New York Times Digital,
If I were Google, I'd tell them such advertising services would cost them a great deal of money. That or simply drop the New York Times right into the bit bucket. It will cost Google programing time to make it happen and computing time to keep it going. If every site on the web required this kind of custom treatment, Google's task would be much more difficult and it might be easier for them to drop it.
Droping the NYT from Google is fine by me. People who don't understand the implications of digital publishing don't deserve readership. If they won't let librarians make digital coppies, libraries should drop them too. What's next, the New York Times sends cease and dissist orders to everyone who runs a proxy? It's like the NYT is trying to make their digital publication harder to share than their paper one was. A paper copy can be shared by an entire office and that's what a proxy does. A paper copy can be indexed and archived by a librarian, and Google did not even do that much. One day the paper version won't be available. If librarians can't keep their own coppies of the digital version for verification, the publication will have no credibility. If the New York Times wants to continue charging advertisers for eyballs, they had better remember that their credibility is bassed in part on widespread availability.
Friends don't help friends install M$ junk.
Is there any easy (spam isn't such a problem for me - touch wood - that I'm willing to spend ages looking into where it comes from) way of telling where this stuff originates from?
The technology has changed the way that things work but the law has not kept up with it. To start with, we continue to talk about "copyright". Controlling copying of information makes sense when the distribution mechanism is trucks moving bales of paper around. Once you start sending bits around, everything is copied. From the article:
And technically, any time a Web surfer visits a site, that visit could be interpreted as a copyright violation, because the page is temporarily cached in the user's computer memory.
When you have the newspaper delivered to your door, the content basically comes for free (the cost of a newspaper doesn't pay for much more than printing and handling). However, you get to keep the content as long as you like, chop it into bits and what not. Libraries have archives of newspapers going back years and you get to see them for free. What's the right mechanism as we move forward? The "pay per view" model that content providers want to shove down our throats courtesy of the DMCA is not pretty and when it starts to affect the average Joe I suspect it will be booed out of favor pretty quickly. But what is the right mechanism to make sure content providers get paid something and that we, the citizens, get something for our money?
Actually, free reg requires a valid email id. It thus filters most bogus registrations
I know what you mean. For a while, my 'valid' email ID was 'root@nytimes.com,' but they eventually caught up to me. Now it's 'sales@nytimes.com.' And if you think there is any legitimate information in my registration, then you would be in error.
What those who want activist courts fear is rule by the people.
As many others have emphasised, it is easy to turn of the Google cache for whatever pages you wish. But, in the case of the NYT, there is a further factor. They must have special code within their system to recognise the google spider and allow it access without registration. Either that, or there is some other prior agreement allowing access. Given that, they can scarcely claim extra work to support Google. I believe the whole thing is mainly to get some free publicity for their site. I suppose the other possibility is that they want the page accessible from Google News but not the regular search engine cache.
The NYT needs to call off the lawyers and seriously think about how they brought this on themselves.
There are so many models for running a news site that avoid this problem (Salon) that calling out the lawyers is just childish and inapropriate. If a site wants to be indexed by a search engine, then they should be aware of what that means, and if they don't like how a particular search engine functions, then they should take measures to change thier own site to prevent what they don't want indexed, or cached, from being accessed.
I know that finding pages on google that I cannot access would be infuriating, and I hope that Google realizes that many of thier users would agree.
Read, L
[Set Cain on fire and steal his lute.]
And if they require a phone number, use 867-5309 # ask for Jenny
Need a Linux consultant in New Orleans?
Every time a cached link is clicked, pay sites like the New York Times can receive notice from Google (easy to automate this) that one of their pages (which is cached in Google) has been accessed, and all advertisements in the cache have been displayed (Google caches Ads in the page as well as the contents). This allows the website to "offload" traffic and at the same time keeping the books on the number of times their Ads have been viewed so that they can send the accounting record to their paid Advertisers.
Google would find this very simple to implement, and paid sites would find this very beneficial (borrowing Google's enormous bandwidth and server capabilities for free) and at the same time should solve most of their concerns. After all, Google's cache isn't sufficient for proper access to ALL the paid-content at the New York Times as the cache is temporary in nature. Also, its too spotty in coverage to be considered reliable enough for really digging into a paid-sites entire content.
Using Google like this is akin to using Google as a window into the pay-site's house of content. You can part of a room, but not the whole interior. Now, every time someone peeks, the House gets notified and can get paid for it. The more windows Google adds to the House, the more chances the House gets paid.
Why doesn't Slashodt cache news articles and stories before running a story? It would make a lot of sense for text based news items.
I use Google news pretty regularly, and I've noticed that some of their links are to paid subscription sites. These are clearly marked as such ("subscription").
;=)
I don't generally click on those links, but I think it's a good idea, since I'm not actually going to Google for the news, rather for links to the news. The reason I personally don't click on the subscription links is that I have my favorite set of real newspaper sites (some registration, some free, some not) and that's not what I'm using Google News to find. Someone else, however, probably is using it that way.
I would guess that Google gets something back from that sort of link, since the site owner is getting more from the link than Google is from the listing. (Maybe I'm wrong, of course.)
It makes perfect sense to have something like that for the regular search engine, and to charge for it, as long as it doesn't affect the link's rank in the search results.
For example they could have a special command for robots.txt (or google.txt maybe) that would allow Google to access and cache the page, but the regular link would go to some registration page (easy to do) *and* the cache link would also go to some kind of registration page, defined in the google.txt file.
The NYT would promise that the cached page is really the cached page, and pay Google something for redirecting to NYT's cache (with registration). Or even better, there would be some kind of redirect where I actually get the cache from Google after I've registered with NYT.
They're probably thinking of something like that, because otherwise the solution would be to simply disallow caching, and that wouldn't be news, would it?
This Like That - fun with words!
The question is framed very narrowly by Slashdot, so this discussion misses the larger issues. The cache copy is an issue in Google's main index for many webmasters. The Google News situation is a subset of a larger problem; the cached link doesn't exist in Google News. Google News is a much narrower issue. I'd like to bring up the issue of full-text caching done by Google in their main index.
.txt files, there's no place to insert a "noarchive" and Google goes ahead and caches it anyway.
My problem with the cache is that it gives Google a competitive advantage that is unfair, and furthers their monopoly. This is especially unfair since it is most likely illegal -- assuming that you could ever get a good test case into court, or get a class action lawsuit going by some webmasters, publishers, or search engines.
To add to the attractiveness of the cache copy, consider what Google has done:
1) The cache copy makes it possible to highlight the search terms, whether or not you have the toolbar installed.
2) The download time for the cache copy from Google's servers is always faster than from the original website.
3) You never get a 404 "not found" or a DNS lookup failure for the cache copy.
4) The link to the page recommended by Google for bookmarking at the top of the cache copy is a link to Google's copy, not to the original page.
5) How about all that Google branding on the top of the cache copy? Priceless. I feel the cache should be opt-in, not opt-out. The only way you can avoid it right now is to place a "noarchive" meta on every page in your site. On some file types, such as
The cache copy tends to keep eyeballs on google.com, and increases their searches. You may have noticed that many major news sites won't link to other websites in their stories anymore, but rather just mention the relevant site without putting a link behind it. That's because they don't want eyeballs wandering off of their page. A wandering eyeball may not come back and look at more ads. That's basically one of the big reasons behind the cache copy as well -- it keeps eyeballs from wandering as much as they would without the cache.
All the Google partners -- AOL, Earthlink, Yahoo, Netscape -- don't include the cache links, and I assume that this is the reason. They don't want people wandering off to Google and staying there.
As new competition is organizing to challenge Google's monopoly, from places such as Overture (Alltheweb and AltaVista), Yahoo (Inktomi), AskJeeves/Teoma and Microsoft, these engines have to consider whether to fight Google on the cache copy, or offer their own cache copy even if they think it is illegal. There isn't really any middle ground on this.
Many observers with legal expertise feel that while the snippets are "fair use" of a website's content, offering the full text in a cache version is not. Copyright law requires "express permission," but Google only offers an incomplete and inconvenient opt-out. I suspect that the legal departments of these other engines are more inclined to challenge Google rather than launch into their own violations of copyright law.
Your comment was confusing to me until I realized that you are talking about giving NYT an actual email address. Why would you do that? Isn't that why we have hotmail.com? Give an address that does not exist or a throw-away address.
Last week I was registering at a web site and I put in xx@xx.com for the address. The system responded, "This address has already been registered." So then I put in xxx@xxx.com. The system responded, "This address has already been registered." So I entered xxxx@xxxx.com. Same response. Finally I awoke fully and entered some Ds, xxxxdd@xxxx.com, and the system accepted my "registration".
How about if the Times got over their registration fetish?
From the Times Subscriber Agreement:
What is meant by "exploit"???From the "Forums and Discussions" section:
What is meant by "abusive"???And how about this>
Interpretation: The user/poster is entirely responsible for the content of their post, which the Times may alter in any way. Yikes!!! Granted, this applies only to content submitted to the Times, but the wording seems pretty scary.
The Russians have won. They have made the world a cesspool of distrust, greed, fear and hate.