Is Data Mining for Product Pricing, Illegal?

wessman asks: "I started to read Orin S. Kerr's 80-page paper looking for how his proposal would pertain to: ripping music/movies, P2P, corporate espionage, and lastly, the use of web scraper robots. Little did I know just how relevant his paper would be in regards to that last item! Kerr makes note of EF Cultural Travel v. Explorica in which Explorica is caught hiring a consultant to program a scraping robot to gather pricing information from a competitor, EF Cultural Travel. Well, I do consulting on the side from home and am currently working a project whereby I gather pricing information from all the major travel conglomerates (Orbitz, Expedia, Lodging.com, WorldRes, Sabre, etc.) so that the travel booking business that hired me can meet or beat all their prices. Granted, the circumstances of the Explorica case are different and the case was an example of an extreme ruling, but my questions to the Slashdot community are: Do I notify the company that hired me of the Explorica case? Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually? Should I continue with this project and the similar projects I do in this area of programming?" Now, add in the text in the "deliverables" section of this press release and it seems we may have some contradictory information. Who is right, and under what circumstances is price harvesting off of the internet not allowed?

Comments

Examples...

[oaf357]

Buy.com was built around a huge server farm that scoured the web and found the best prices for products it sells and then beat those prices (to the best of its ability). That has changed a little now but buy.com was built from that.

Also, Pricewatch, Pricegrabber and Froogle scour the web for prices and create search engines out of them so consumers can find the best price.

I'm not saying just because everyone else is doing it means you can too (and you might have a slightly different objective causing these examples to be weightless) but it's being done all over the place.

Hope that helps.

Ah, just the comment I was looking for.

[ahfoo]

I think what you have to look at is the media context in which the prices are displayed.
It's quite true that many stores will try to prevent you from making recordings of any kinds on their physical premises. I've been reprimanded by store managers many times for taking photos in the store. But their right to prevent me from creating media on their premises is based on their property rights, not any some legally backed authority to censror the media.
The web is a totally different story. I use web scrapers all the time and a site that doesn't like it can kindly take its ass off the web. Once you place material on the web, it is published. If you don't want to publish your prices, you don't have to. That's like publishing a book and complaining the readers read it too fast.
The people who compain about such things are the idiots who create unworkable business plans based on their own assumptions about how people are going to use the resource. This is an interesting issue with news media that want to sell access to their archives. There's no way they can both publish to the web and prevent me from caching old copies. If that's the business plan then web publishing is an inappropriate business decision and guess who should pay for bad business decisions: the consumer, or the fool who pursued an ignorant business plan?

Public servers are really private

[mypalmike]

While I agree wholeheartedly with your argument, the courts do not. Another example from the same paper is even scarier. In a similar case (Register.com vs. Verio), the court said the use of search robots to gather information may be illegal regardless of whether it broke the site's terms of service:

"Instead, the Court concluded that the mere fact that Register.com had decided to sue Verio meant that Verio's use of the search robot was without authorization: 'because Register.com objects to Verio's use of search robots,' the Court held, 'they represent an unauthorized access to the [Register.com] WHOIS database.'"

[Ironically, the pdf for the paper apparently uses some feature of Acrobat which disallows copying text from it. I guess they don't want robots scraping text out of it or something. First time in quite a while I've had to type a quote from the net by hand!]

Re:I'm reminded

[shumacher]

Both Best Buy and the judge were off their rocker. Best Buy can ask him to leave. The store is private property. If he is a customer, they should have allowed him to continue to calculate prices. If Best Buy asked him to put his laptop away, he should have complied. Further, the article mentions other stores that claim to not have a policy against Mr. Kahlow's behavior. I know for a fact that one of them actually is okay with Mr Kahlow's behavior, but does have a policy (and not just an "unwritten" one)against competitor's doing what Mr. Kahlow was doing. (Which is a different thing - and more on topic - one means a potential sale for them, the other means potential sales for their competition) Their policy is to ask the suspected competitor what they are doing, and then if confirmed as a competitor, they are to make a specific statement essentially revoking that individual's right to visit the store, and then, they are to ask them to leave. When that store shops a competitor, policy compels them to wear normal street clothes, to not use any automated devices, and to admit their affilliation and leave if asked.

Re:I swear

[Sloppy]

Actually, there's a lot of creativity that goes into pricing things.

Ever see this: $39.95 ? The .95 cuts .05 cents off the price but greatly increases the chances you'll buy it.

How about 2/6.00? Unless they say otherwise, that means 1's only $3, but again, you're more likely to buy the two. A supermarket by me commonly sells things like limes and small bags of snacks at 4/$1.00, rather than just $.25, which i guess seems cheap.

A lot of the delis around me have sandwiches for 4.63, which with tax is exactly a finski. No numeric change means they can ring you out faster.

Perhaps the first time someone came up with these techniques, it was an act of creativity. But they are now well-known. When someone prices something as 39.95, they are not being creative; they are merely "using the .95 trick."

Besides this, figuring out what things are worth is a very exact science when you live, as we do, in a vastly service based economy. Prices have nothing to do with the actual cost of physical production, and everything to do with the public's perceived value of the product in question. This is how GM can make two identical trucks, label one GMC, and get another $3000 for it because GMCs are "professional grade and therefore more reliable." This is how Victoria's Secret can get $30+ for maybe a quarter square foot cloth and a pair of wires, and in fact get more money when they use less material or "exotic" materials like satin (which often cost less than cotton).

Ok, these are much better examples of creativity. The person doing the pricing is actually doing work and making real decisions.

But does that entitle their work to special protection? Let's go back and look at why we have "IP laws" at all, lest we forget the purpose and perversely misapply the law. In the United States, the Constitution empowers congress on this basis:

To promote the progress of science and the useful arts by securing for a limited time to authors and inventors the exclusive right to their respective rights and discoveries...

It is effort for you to write a book, and if you didn't have a monopoly on the sales of that book, your effort would be wasted from a commercial perspective and you might not choose to expend that effort. So we give you a copyright. You get what you want -- compensation for your effort -- and we get what we want -- promotion of the useful arts. Perhaps that promotion is somewhat delayed (until after the copyright expires) but we'll eventually get it.

Now let's look at your sophisticated pricing example, in that light. You expend effort in figuring out just the right price for your widget, and you would like a monopoly on the fruits of that work. If we grant that to you, then you get what you want -- compensation for your effort -- and we get ... hey, what do we get? Does the creative act of figuring out that $30 is the most profitable price for your underwear, help us in some way? Does it promote the progress of the sciences or useful arts? I don't see how.

I see that there's something in it for me, to grant copyright on creative works such as books. I don't think there's anything in it for me at all, to grant copyright on prices, even prices that take effort to optimize.

And without a monopoly, you won't have sufficient incentive to do the work of determining the most profitable price for your satin underwear or truck? As if!