Slashdot Mirror
Is Data Mining for Product Pricing, Illegal?
wessman asks: "I started to read Orin S. Kerr's 80-page paper looking for how his proposal would pertain to: ripping music/movies, P2P, corporate espionage, and lastly, the use of web scraper robots. Little did I know just how relevant his paper would be in regards to that last item! Kerr makes note of EF Cultural Travel v. Explorica in which Explorica is caught hiring a consultant to program a scraping robot to gather pricing information from a competitor, EF Cultural Travel. Well, I do consulting on the side from home and am currently working a project whereby I gather pricing information from all the major travel conglomerates (Orbitz, Expedia, Lodging.com, WorldRes, Sabre, etc.) so that the travel booking business that hired me can meet or beat all their prices. Granted, the circumstances of the Explorica case are different and the case was an example of an extreme ruling, but my questions to the Slashdot community are: Do I notify the company that hired me of the Explorica case? Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually? Should I continue with this project and the similar projects I do in this area of programming?" Now, add in the text in the "deliverables" section of this press release and it seems we may have some contradictory information. Who is right, and under what circumstances is price harvesting off of the internet not allowed?
The problem is that when you sell a commodity like a TV or a vcr and the only difference between them is price you can't exactly maintain a high profit margin. What they need to do is obfuscate the prices so that its next to impossible to compair products. Thats how cell phones work.
This phone has 500 any time minutes for 3 cents a minute from your calling area roaming is 10 cents a minute, unlimited text messaging, 800 night and weekend minutes is free for the first 6 months and has rollover.
This other phone has 1000 minutes for 2 cents a minute but with out rollover and text messaging costs 1 cent per message, night and weekends are free but don't start till 9pm.
See its not exactly easy to just look at the plans and see which one is the better plan.
He who knows not and knows he knows not is a wise man. He who knows not and knows not he knows not is a fool.
Also, Pricewatch, Pricegrabber and Froogle scour the web for prices and create search engines out of them so consumers can find the best price.
I'm not saying just because everyone else is doing it means you can too (and you might have a slightly different objective causing these examples to be weightless) but it's being done all over the place.
Hope that helps.
The amount of beauty required to launch 1 ship: 1 Millihelen
I think what you have to look at is the media context in which the prices are displayed.
It's quite true that many stores will try to prevent you from making recordings of any kinds on their physical premises. I've been reprimanded by store managers many times for taking photos in the store. But their right to prevent me from creating media on their premises is based on their property rights, not any some legally backed authority to censror the media.
The web is a totally different story. I use web scrapers all the time and a site that doesn't like it can kindly take its ass off the web. Once you place material on the web, it is published. If you don't want to publish your prices, you don't have to. That's like publishing a book and complaining the readers read it too fast.
The people who compain about such things are the idiots who create unworkable business plans based on their own assumptions about how people are going to use the resource. This is an interesting issue with news media that want to sell access to their archives. There's no way they can both publish to the web and prevent me from caching old copies. If that's the business plan then web publishing is an inappropriate business decision and guess who should pay for bad business decisions: the consumer, or the fool who pursued an ignorant business plan?
Anybody who publishes information about their business runs the risk that a competitor will get hold of the information and use it in some way. This has always been a fact of life in the physical world. As computers came online in recent decades many companies have maintained databases of information about competitors' products. The Internet doesn't change any of this.
Output the price in a un-OCRable jpeg image.
You see signs everywhere you go:
-Shirt & shoes required.
-No loitering.
-No soliciting.
-Check all bags at counter.
-No more than two students allowed in store at one time.
-Parking lot, bathroom, etc. for customer use only.
Just because a building (or a web site) is in a public place doesn't mean that everyone is free to do whatever they want. Business owners are free to create house rules that everyone needs to follow.
Similarly, web sites can legally restrict what you are allowed to do when you visit them without having to build security measures to force compliance. If web retailers don't mind robots harvesting their inventory and prices, great. If, however, they want to place restrictions on who can access their site and how, that's entirely their prerogative.
Think about it. Leaving the door to your home unlocked would make it easier for people to steal your stuff, but it still wouldn't make it legal for them to do so unless you put up a sign saying something like, "Free for the taking."
Web scrapers are legal to develop and they're legal to use on sites with acceptable use policies that allow them. However, your customers should be prepared for the possibility that some or all of their competitors could make them stop using it at some point. And, in the interests of maintaining your own professional ethics, you should probably call their attention to the issues surrounding the job they're asking you to do.
"Instead, the Court concluded that the mere fact that Register.com had decided to sue Verio meant that Verio's use of the search robot was without authorization: 'because Register.com objects to Verio's use of search robots,' the Court held, 'they represent an unauthorized access to the [Register.com] WHOIS database.'"
[Ironically, the pdf for the paper apparently uses some feature of Acrobat which disallows copying text from it. I guess they don't want robots scraping text out of it or something. First time in quite a while I've had to type a quote from the net by hand!]
There are 0x40000000 types of people: those who understand 32-bit IEEE 754 floating point, and those who don't.
If a bot activates a click-through agreement, does anyone hear it fall?
"A modicum of creativity" is part of the standard test of copyrightability. Telephone books are not copyrightable on this basis (see Feist Publications v. Rural Telephone Service, 499 US 340 (1991)).
But airline prices probably are.
Calculating the price of an airline ticket is a difficult process, part of the reason travel agents exist and make use of sophisticated computer programs to calculate prices. Compare 5 different web sites' prices for a complex ticket and you'll get 5 different numbers, often differing by thousands of dollars -- for exactly the same flights. So it would seem that creativity is involved. Certainly one could argue that the selection of what tickets to display on a web site like Orbitz is a creative process.
On this basis, I'd say airline prices are copyrightable. And all the major travel web sites specifically prohibit robots in their terms-of-use legalese.
So I think that screen-scraping travel web sites is a very iffy proposition, legally. Courts seem to agree: American Airlines was granted an injunction prohibiting FareChase from scraping them and selling their prices to competitors.
Yes, because god knows self-biographies always tell the truth.
You don't understand the fundamental use of law under the Corporate Administration. Data mining is legal when it involves Admiral Poindexter, your grocery store, your viewing habits or your medical data. Data minig is illegal when it benefits you the consumer at all, much less at the expense of a needy company's profits.
I spent a year in Iraq looking for WMD and all I found was this lousy sig.
The corporation will continue to hump the internet like a dog in heat until it becomes as regulated, watered down, and crapped out like the NBC news.
Both Best Buy and the judge were off their rocker. Best Buy can ask him to leave. The store is private property. If he is a customer, they should have allowed him to continue to calculate prices. If Best Buy asked him to put his laptop away, he should have complied. Further, the article mentions other stores that claim to not have a policy against Mr. Kahlow's behavior. I know for a fact that one of them actually is okay with Mr Kahlow's behavior, but does have a policy (and not just an "unwritten" one)against competitor's doing what Mr. Kahlow was doing. (Which is a different thing - and more on topic - one means a potential sale for them, the other means potential sales for their competition) Their policy is to ask the suspected competitor what they are doing, and then if confirmed as a competitor, they are to make a specific statement essentially revoking that individual's right to visit the store, and then, they are to ask them to leave. When that store shops a competitor, policy compels them to wear normal street clothes, to not use any automated devices, and to admit their affilliation and leave if asked.
However, if they don't want to be listed then they should have right not to be. No?
.htacces?
robots.txt or
There was an incident of a man being arrested in a Best Buy (for trespassing, I believe) after being asked to stop writing down prices for large screen TVs. The irony of it, and I explained this in a post a long time ago, is that I used to work at Best Buy, and on weekends, we were asked to bring a non-blue shirt with us to work so that we could go incognito to the local electronics store (H.H. Greggs, before we had a Circuit City locally) and use a micro-cassette recorder to 'steal' all their prices so we could mark down the items in store to compete. Now they're telling people that they can't do what they themselves do (or did). Reminds me of a local story about a guy who was wearing one of those fancy NASCAR leather jackets with either Home Depot or Menards as the sponsor of the team going into the store that wasn't the sponsor and who was celebrating their grand opening (i.e., wore a Menards jacket into a Home Depot, or maybe it was the other way around) and was asked by management to leave because they thought he was a spy from the other company. Made the local paper when it happened...
A computer once beat me at chess, but it was no match for me at kick boxing -- Emo Phillips
But does that entitle their work to special protection? Let's go back and look at why we have "IP laws" at all, lest we forget the purpose and perversely misapply the law. In the United States, the Constitution empowers congress on this basis:
It is effort for you to write a book, and if you didn't have a monopoly on the sales of that book, your effort would be wasted from a commercial perspective and you might not choose to expend that effort. So we give you a copyright. You get what you want -- compensation for your effort -- and we get what we want -- promotion of the useful arts. Perhaps that promotion is somewhat delayed (until after the copyright expires) but we'll eventually get it.Now let's look at your sophisticated pricing example, in that light. You expend effort in figuring out just the right price for your widget, and you would like a monopoly on the fruits of that work. If we grant that to you, then you get what you want -- compensation for your effort -- and we get ... hey, what do we get? Does the creative act of figuring out that $30 is the most profitable price for your underwear, help us in some way? Does it promote the progress of the sciences or useful arts? I don't see how.
I see that there's something in it for me, to grant copyright on creative works such as books. I don't think there's anything in it for me at all, to grant copyright on prices, even prices that take effort to optimize.
And without a monopoly, you won't have sufficient incentive to do the work of determining the most profitable price for your satin underwear or truck? As if!
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
I think the point here is being missed. Read the brief. There were confidentiality agreements involved which ultimately decided the direction of the original injunction. As for the second injunction, a third party cannot aid in the circumventure of an injunction. Period.
You or I could gather price data from sites and use it as a guidelines without legal worries. Compiling and selling that data outright would probably get you in trouble because of its time sesitive nature.
Evolution doesn't mvoe fast enough for cheap beer, mobile homes, and the KKK to be genetic.
They aren't evil because they charge money. They're evil because they control who may compete with them (so they can charge vastly more money than needed to do the job while preventing most disruptive innovation) and now they want to control how their own customers use the only services they grudgingly provide.
No, I don't want to go to jail. ILECs (along with cable TV providers) have a monopoly on laying wire for communication. Did you think CLECs wanted to rely on adequate service from their most ruthless competitors?
I'm sure there are a few products that assist disabled persons to "surf" websites by disecting the web page (through essentially screen scraping techniques) and performing one or more the following:
1. Adjusting text size.
2. Dictation of content.
3. Numbering of links.
4. Numerous other alternate presentation of the same data (changing colors for the color blind for example).
An outright ban of automated scraping techniques would eliminate these uses. (While I am at it: What is a web-browser but a form of screen scrapper?).
If the basic technique is allowed, all that can be debated is the use of such data and I think that is a much more dubious area. Facts are public domain.
Maybe they can use the "my bot is blind defense".
Check out Largest recipe database on the web.