Slashdot Mirror
Is Data Mining for Product Pricing, Illegal?
wessman asks: "I started to read Orin S. Kerr's 80-page paper looking for how his proposal would pertain to: ripping music/movies, P2P, corporate espionage, and lastly, the use of web scraper robots. Little did I know just how relevant his paper would be in regards to that last item! Kerr makes note of EF Cultural Travel v. Explorica in which Explorica is caught hiring a consultant to program a scraping robot to gather pricing information from a competitor, EF Cultural Travel. Well, I do consulting on the side from home and am currently working a project whereby I gather pricing information from all the major travel conglomerates (Orbitz, Expedia, Lodging.com, WorldRes, Sabre, etc.) so that the travel booking business that hired me can meet or beat all their prices. Granted, the circumstances of the Explorica case are different and the case was an example of an extreme ruling, but my questions to the Slashdot community are: Do I notify the company that hired me of the Explorica case? Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually? Should I continue with this project and the similar projects I do in this area of programming?" Now, add in the text in the "deliverables" section of this press release and it seems we may have some contradictory information. Who is right, and under what circumstances is price harvesting off of the internet not allowed?
Pricewatch doesnt mine. Companies PAY for the privelege of listing their prices on pricewatch.
Filing lawsuits to protect your price information is just dumb, not to mention waste (if not abuse) of the legal system.
Personal feelings about freedom of information aside, and just from a coder's POV, here's my solution.
If they really want to avoid getting scraped, they should just get their existing, underpaid web developers to create a backend setup that generates the prices as gif's that give OCR hell (such as those used to prevent automated registration of say Yahoo! email accounts).
Coders are cheaper than lawyers (at least those needed to write such code as this).
Sure, the compition could pay more money to get somebody to develop better OCR to read each and every dynamically generated GIF, but most people require proof reading of OCR data, which leads to even more cost.
Something I learned from my Uncle who works with the DOD is this: Any lock can be picked; Any encryption can be broken. It's just a matter of if it's worth the time and money to get what's inside.
In short, with a little one time cost, the company that doesn't want it's prices scraped can just make it so hard to scrape their prices that it's not worth it. The price of scraping the graphically displayed price tags would also be an ongoing cost of software and proofreaders that would dip into profit margins, which management at the company that desires the scraping won't like.
It's not perfect, but it's better (and more bankable) than going whining to the legal system. (Especially since coders are generally cheaper than lawyers).
DONT PANIC
Wrong. Read Sam Walton: Made in America: My Story. Sam Walton says that was a story put out by his competitors to disparage his name and he never did anything of the sort.
Read the case...EF Cultural Travel BV v. Explorica hinges on the fact that the defendant company hired an ex-programer from the plaintif company. The programmer had special knowledge of codes used in the pricing (which he had signed a confidentiality agreement not to disclose). When he made the scrapper program he violated the confidentiality agreement.
:) Depending on how the contract is written you could be jointly liable.
It was the violation of the confidentiality agreement that the court held was illegal.
As for whether you should tell your employer, it depends on your employment agreement!
While this is a 1st Circuit case, it has been followed by the 5th Circuit (Ingenix, Inc. v. Lagalante) and cited in cases in the 7th and 9th Circuit.
Hope this helps.
--me
I am guessing that the prohibition on "visit[ing] for any commercial purpose" precludes me from actually purchasing their wares.
Here is a related incident:
http://news.com.com/2110-1017-944258.html
Bargain Network spidered real estate prices on homestore.com/realtor.com and posted them on the bargain.com website. Homestore sued and the case was settled out of court. I wish it was not settled out of court because that would set up a precident.
In my opinion you are asking for the problems. Taking a case like this to court and winning would be difficult. At the very least it would be a serious legal expense.
The last time I checked the rules for Froogle you had to be the actual merchant that ships the product in order to show up in their index. If you are spidering a merchant then you are an affiliate, the products do not originate from you so you would be exluced from Froogle. Froogle does not allow you to sort products by price - so obviously what you plan on doing is different. Froogle also gives merchants the option to be excluded from their index.
My advice is this - get a lawyer because one will surely be contacting you. Familiarize yourself with these phrases: false advertising, breach of contract, and unfair competition.