Judge Bars eBay Crawler

matty writes: "A judge has said that Bidder's Edge could no longer use its crawler to gather information from eBay. 'Even if its searches use only a small amount of eBay's computer system capacity, Bidder's Edge has nonetheless deprived eBay of the ability to use that portion of its personal property for its own purposes.' So what about Yahoo! and all the other search engines? Don't they use similar technology? Read the article and see for yourself." Or maybe it's not such a bad precedent; it'd be interesting if such a ruling helped discourage hard-drive searching by software which searches for "undesirable" content without your consent or knowledge.

Comments

Re:robots.txt

[breser]

Not really... Take a look at some of the other robots.txt files they have on various other machines. For example listings.ebay.com

# go away
User-agent: *
Disallow: /

I'd say that's everything.

Seems Like a Really Dumb Thing but ....

[The Code Hog]

At first blush, it seems like this is a stupid ruling, mainly for the reasons the judge gave for making it. He claims that they are essentially stealing cycles from eBay's servers and this could slow down ebay's service and have a negative impact on their customers' experience.

This is just plain stupid; if you have a page on your website which is viewable by the public then it is available for the public to download. That's the point of having a public website. Hey, I'm a customer of eBay's, am I guilty of using server cycles and slowing down the eBay website for other customers? You bet. eBay should secure the entire site and require authentication if they really want to pick and choose who can view.

On the other hand, I think what Bidder's Edge does is really indefensible from an ethical standpoint and I am rooting for them to lose because they are in effect *competing* with eBay for advertising dollars by *using* eBay's content. If you view content from ebay through Bidder's Edge, that's advertising revenue eBay doosn't get which BE does. Seems really lousy.

So it seems like the right ruling, but for totally the wrong reasons. The way the judge worded things it sounds like you could make a case for suing Yahoo, AltaVista, Google etc., if they dare to spider your site.

Whot crap!

robots.txt

[Citrix]

I would have some sympathy with Bidders Edge but they don't follow the robots.txt file on eBay.

here is http://search.ebay.com/robots.txt:

# robots.txt for eBay

User-agent: *
Disallow: /aw/listings/
Disallow: /aw-cgi/
Disallow: /aw-secure/
Disallow: /cgi-bin/

It isn't like eBay is disallowing access to everything, crawlers are allowed to index anything on www.ebay.com (no robots.txt) and whatever is not excluded search.ebay.com. IMO whether the judge knows it he is upholding a standard and that is a good thing.
Citrix