Slashdot Mirror
Is Data Mining for Product Pricing, Illegal?
wessman asks: "I started to read Orin S. Kerr's 80-page paper looking for how his proposal would pertain to: ripping music/movies, P2P, corporate espionage, and lastly, the use of web scraper robots. Little did I know just how relevant his paper would be in regards to that last item! Kerr makes note of EF Cultural Travel v. Explorica in which Explorica is caught hiring a consultant to program a scraping robot to gather pricing information from a competitor, EF Cultural Travel. Well, I do consulting on the side from home and am currently working a project whereby I gather pricing information from all the major travel conglomerates (Orbitz, Expedia, Lodging.com, WorldRes, Sabre, etc.) so that the travel booking business that hired me can meet or beat all their prices. Granted, the circumstances of the Explorica case are different and the case was an example of an extreme ruling, but my questions to the Slashdot community are: Do I notify the company that hired me of the Explorica case? Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually? Should I continue with this project and the similar projects I do in this area of programming?" Now, add in the text in the "deliverables" section of this press release and it seems we may have some contradictory information. Who is right, and under what circumstances is price harvesting off of the internet not allowed?
In what sane land would PRICES be protected under law? You can't really keep them secret, so "trade secret" is right out. It's not a identifying mark (unless you're a dollar store), so much for trademark. There's nothing useful that hasn't been done before, fuck patenting them. Copyright? It's a simple derivation of what the supplier charges you.
There is nothing creative about pricing stuff. Good lord.
-- Bill "Houdini" Weiss
Of course it is. Let's dissect this sentence:
Not having the comma would completely distort the meaning of the sentence.
My grandmother calls this "shopping around." The only difference is that someone else is doing all the work.
Smeghead every day of the week.
As long as you get paid, let them worry about the lawsuit. They're the ones who are going to actually use it. Keep your mouth shut.
If powerful people get screwed, it's illegal.
If it forces large corporations have to work harder to earn a profit, it's illegal.
If it give the little guy a leg up or levels the playing field in any way, it's illegal.
If it's illegal and you're big and powerful, don't worry about it, you can probably get away with it with little damage to your business or career and keep almost all of you cash minus legal fees.
<a href="http://www.joblessjimmy.com">Work is dumb and so is Jobless Jimmy.</a>
Who is this "Illegal" person and why are we asking him questions about Star Trek characters?
I'm not "Illegal," but I'll answer. Seeing as how he was killed off in the last movie, I think it's safe to say that no, Data is not mining for product pricing.
(In other words, you illiterate clods need to be more careful with your commas.)
Look, I I can visit a web site and the business (Let's say Amazon) publicly posts their prices for anybody to see then you sure as hell can use them! If suddently using bots to do work are illegal then I'd wadger that every shell script that I write is an affront to US Laws. Rotating log files and all sorts of other "make my job easier so that I can play Quake" scripts are perfectly legal, so how the hell can it be questionable just to go to a site and record prices???
Jebus, please help the Unites States Gub'ment!
How does one receive authorization to access a web server? Hmm, maybe with a simple html GET? The basic fact here is that of judicial cluelessness. If I put information on a public web server, pretend to "protect" it with a disclaimer (of everything) at the bottom of the page, and then get pissed off because somebody browsed that information, I'm an idiot. In addition, I am legless in court. Web servers make information available to the world. If I had wanted to make information available to certain parties that I trust not to compete with me, I should have set up a secure server with some provision for authentication and authorization.
It really is that simple
later,
Jess
I am programmed for etiquette, not destruction!
Once their prices hit the Internet, they're in the public domain. It would be like posting your prices in the window, and complaining that a car driving past could photograph them.
We all know that bots crawl the web - Google, Altavista, spam-bots... they're all common knowledge. You put information on a website, and it's going to be viewed by an automated process. Surely with that knowledge, it's ridiculous to think you can ban people for using the information you've posted publicly in whatever way they desire.
Perhaps these companies (airlines, computer stores, whatever) need to start offering their services at the price they really mean to sell it for, rather than this stupid haggling they expect from us. Or maybe it's time they focused on quality of service, value-add, etc rather than price wars (which never help anybody in the long term).
Bottom line? If you don't want your competitors seeing your prices, don't make them available to them - this means no junkmail, no spam, no website, no prices in the store window, no prices inside the store, nothing.
Also, Pricewatch, Pricegrabber and Froogle scour the web for prices and create search engines out of them so consumers can find the best price.
I'm not saying just because everyone else is doing it means you can too (and you might have a slightly different objective causing these examples to be weightless) but it's being done all over the place.
Hope that helps.
seems like it's the using confidential information part that got the scrapper capped.
I don't see why accessing *public* information be problematic.
the only thing that may be of trouble is the website EULA, but then the EULA would be saying the same thing as "don't visit my store unless you intend to buy," which would be rediculous in brick-and-mortar world (and should be similarly in cyberspace).
last question, though - why the heck would you ask this kind of stuff HERE? wouldn't a law-forum be a better choice?
My life in the land of the rising sun.
I am not a lawyer.
Slashdot is not a lawyer.
Slashdot is not a replacement for a lawyer.
Individual posters on slashdot may be lawyers, but are you really willing to trust your future to what some random person online says, when they could be a lawyer, but could also be some 14 year old kid who thinks it's amusing to screw with people?
Repeat after me:
I will seek proper legal advice.
Seriously, this comes up time and time again. If you're in a situation where you need actual concrete legal advice, SLASHDOT IS NOT THE PLACE TO GO. Sending in an Ask Slashdot is fine for theoretical questions, but when your ass is at stake if a lawsuit comes around, do you really want to trust your future to the legal advice given to you by Anonymous Cowards and karma whores?
Be the Ultimate Ninja! Play Billy Vs. SNAKEMAN today!
Filing lawsuits to protect your price information is just dumb, not to mention waste (if not abuse) of the legal system.
Personal feelings about freedom of information aside, and just from a coder's POV, here's my solution.
If they really want to avoid getting scraped, they should just get their existing, underpaid web developers to create a backend setup that generates the prices as gif's that give OCR hell (such as those used to prevent automated registration of say Yahoo! email accounts).
Coders are cheaper than lawyers (at least those needed to write such code as this).
Sure, the compition could pay more money to get somebody to develop better OCR to read each and every dynamically generated GIF, but most people require proof reading of OCR data, which leads to even more cost.
Something I learned from my Uncle who works with the DOD is this: Any lock can be picked; Any encryption can be broken. It's just a matter of if it's worth the time and money to get what's inside.
In short, with a little one time cost, the company that doesn't want it's prices scraped can just make it so hard to scrape their prices that it's not worth it. The price of scraping the graphically displayed price tags would also be an ongoing cost of software and proofreaders that would dip into profit margins, which management at the company that desires the scraping won't like.
It's not perfect, but it's better (and more bankable) than going whining to the legal system. (Especially since coders are generally cheaper than lawyers).
DONT PANIC
I think what you have to look at is the media context in which the prices are displayed.
It's quite true that many stores will try to prevent you from making recordings of any kinds on their physical premises. I've been reprimanded by store managers many times for taking photos in the store. But their right to prevent me from creating media on their premises is based on their property rights, not any some legally backed authority to censror the media.
The web is a totally different story. I use web scrapers all the time and a site that doesn't like it can kindly take its ass off the web. Once you place material on the web, it is published. If you don't want to publish your prices, you don't have to. That's like publishing a book and complaining the readers read it too fast.
The people who compain about such things are the idiots who create unworkable business plans based on their own assumptions about how people are going to use the resource. This is an interesting issue with news media that want to sell access to their archives. There's no way they can both publish to the web and prevent me from caching old copies. If that's the business plan then web publishing is an inappropriate business decision and guess who should pay for bad business decisions: the consumer, or the fool who pursued an ignorant business plan?
Wrong. Read Sam Walton: Made in America: My Story. Sam Walton says that was a story put out by his competitors to disparage his name and he never did anything of the sort.
Read the case...EF Cultural Travel BV v. Explorica hinges on the fact that the defendant company hired an ex-programer from the plaintif company. The programmer had special knowledge of codes used in the pricing (which he had signed a confidentiality agreement not to disclose). When he made the scrapper program he violated the confidentiality agreement.
:) Depending on how the contract is written you could be jointly liable.
It was the violation of the confidentiality agreement that the court held was illegal.
As for whether you should tell your employer, it depends on your employment agreement!
While this is a 1st Circuit case, it has been followed by the 5th Circuit (Ingenix, Inc. v. Lagalante) and cited in cases in the 7th and 9th Circuit.
Hope this helps.
--me
Here is a related incident:
http://news.com.com/2110-1017-944258.html
Bargain Network spidered real estate prices on homestore.com/realtor.com and posted them on the bargain.com website. Homestore sued and the case was settled out of court. I wish it was not settled out of court because that would set up a precident.
In my opinion you are asking for the problems. Taking a case like this to court and winning would be difficult. At the very least it would be a serious legal expense.
The last time I checked the rules for Froogle you had to be the actual merchant that ships the product in order to show up in their index. If you are spidering a merchant then you are an affiliate, the products do not originate from you so you would be exluced from Froogle. Froogle does not allow you to sort products by price - so obviously what you plan on doing is different. Froogle also gives merchants the option to be excluded from their index.
My advice is this - get a lawyer because one will surely be contacting you. Familiarize yourself with these phrases: false advertising, breach of contract, and unfair competition.