Is Data Mining for Product Pricing, Illegal?

wessman asks: "I started to read Orin S. Kerr's 80-page paper looking for how his proposal would pertain to: ripping music/movies, P2P, corporate espionage, and lastly, the use of web scraper robots. Little did I know just how relevant his paper would be in regards to that last item! Kerr makes note of EF Cultural Travel v. Explorica in which Explorica is caught hiring a consultant to program a scraping robot to gather pricing information from a competitor, EF Cultural Travel. Well, I do consulting on the side from home and am currently working a project whereby I gather pricing information from all the major travel conglomerates (Orbitz, Expedia, Lodging.com, WorldRes, Sabre, etc.) so that the travel booking business that hired me can meet or beat all their prices. Granted, the circumstances of the Explorica case are different and the case was an example of an extreme ruling, but my questions to the Slashdot community are: Do I notify the company that hired me of the Explorica case? Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually? Should I continue with this project and the similar projects I do in this area of programming?" Now, add in the text in the "deliverables" section of this press release and it seems we may have some contradictory information. Who is right, and under what circumstances is price harvesting off of the internet not allowed?

Comments

I swear

[cultobill]

In what sane land would PRICES be protected under law? You can't really keep them secret, so "trade secret" is right out. It's not a identifying mark (unless you're a dollar store), so much for trademark. There's nothing useful that hasn't been done before, fuck patenting them. Copyright? It's a simple derivation of what the supplier charges you.

There is nothing creative about pricing stuff. Good lord.

Re:I swear

[Gortbusters.org]

Look at pricewatch.com, it already goes around collecting price data from many online stores.

I don't see why this is such a big problem... one site creates competitive prices based upon other sites' prices. In reality if a consumer reaches your site on the internet for your product they probably didn't do it by accident. They will evaluate all aspects of the business (licensing, service/support, upgrade cost, security of the site, etc) before they jump ship to another site to save a few bucks.

I'm sure Dell, Sony, and HP already keep an eye on eachother's prices.

Re:I swear

[whereiswaldo]

Here's a niche for P2P software... obtaining data from a website in a distributed way so as not to stick out in the website logs the way a one or a few download clients would. The collected data could be processed remotely or uploaded to a central server.

Re:I swear

[anonymous loser]

Pricewatch doesnt mine.

No, but froogle.google.com does.

Re:I swear

[smack_attack]

We don't live a capitalist society... we live in what's called a "mixed economy".

Essentially it goes something like this: government supports business rights over individual rights; if two business' butt heads, then the government supports whichever one has more "pull" with the government.

Pretty slick market we have in place eh?

TOS and more ideal markets

[weston]

I think any large business where the pricing structure isn't directly related to costs is probably deeply afraid of agents that aggregate their data with competitors. You end up with a more ideal market, a more frictionless market, if you will, and they'll be forced to compete on narrower and narrower margins of profit. Of course they'll want to throw up barriers to that.

But I'll bet this issue comes down to Terms of Service and what a company can reasonably expect to be able to legally require/forbid about the use of data provided via an automated means...

You have a contract?

[gpinzone]

As long as you get paid, let them worry about the lawsuit. They're the ones who are going to actually use it. Keep your mouth shut.

Re:You have a contract?

[vandan]

Would you apply this line of reasoning to ALL areas?
What if the job were researching Bush's all-feared biological weapons?
Or GM products?

The problem is that if everybody decides to look the other way (and everyone can find a reason why they should take their money and shut up), then some pretty fucked up things get done, and people are left wondering, "How did it come to this?"

Now I'm not saying that the world is going to end because someone's harvesting prices off the net. I'm just questioning your "Me first, no-one else 2nd" argument.

Re:You have a contract?

[stiller]

I do hope that every /. reader at least can discriminate between building life-threatening devices and products that may or may not just fall under some crazy copyright law. It's the user's responsibility not his. They assigned him, they carry the consequences.
It's your way of extreme reasoning that gets us nowhere. We must be reasonable, and it's reasonable to expect that no judge in his right mind would convict anyone for this.

Rules of Thumb to Live By

[release7]

I'm not going to pretend to know what the laws are in this case. It seems that the only law these days is "He who can afford the best, most impressive lawyer wins." Here's some other cynical words of wisdom I've come to believe:

If powerful people get screwed, it's illegal.
If it forces large corporations have to work harder to earn a profit, it's illegal.
If it give the little guy a leg up or levels the playing field in any way, it's illegal.
If it's illegal and you're big and powerful, don't worry about it, you can probably get away with it with little damage to your business or career and keep almost all of you cash minus legal fees.

Re:Rules of Thumb to Live By

[urbanRealist]

It seem like you're right. In no way shape or form did anyone break either the spirit or the letter of the law here. Yet,

First Circuit holds that prohibitions found in a website's Terms of Use can be used to establish that a visitor to that site exceeded his authorized use thereof for the purposes of establishing a violation of the Computer Fraud and Abuse Act

Which says to me I can say, "oh, you can't read my website while wearing sunglasses." I could be wrong here, but according to this ruling, I can just put up any restriction I can dream up and you're bound by it when reading my website. That ruling truly must have cost a fortune. The other down side is that data mining is probably one of the most useful developments in recent times. Look at Google News.

Re:Rules of Thumb to Live By

[poot_rootbeer]

It seems that the only law these days is "He who can afford the best, most impressive lawyer wins."

I can see how someone could be fooled into believing this, if the only news they get about what's happening in the courts is the fearmongering they read in "Your Rights Online."

I suggest you obtain a copy of the verdict records from a court that deals with a wide variety of cases. Ten bucks says you'll come to the conclusion that regardless of who the legal counsel is on both sides, justice is truly served far more often than not.

well....duh

[mrpuffypants]

Look, I I can visit a web site and the business (Let's say Amazon) publicly posts their prices for anybody to see then you sure as hell can use them! If suddently using bots to do work are illegal then I'd wadger that every shell script that I write is an affront to US Laws. Rotating log files and all sorts of other "make my job easier so that I can play Quake" scripts are perfectly legal, so how the hell can it be questionable just to go to a site and record prices???

Jebus, please help the Unites States Gub'ment!

Re:well....duh

[Some Dumbass...]

so how the hell can it be questionable just to go to a site and record prices???

Let's not forget that this was a legal ruling, and some of judges don't know squat about technology. The lawyer for the defense probably showed the judge a web page with prices on it and the judge assumed it was "hacking" or somesuch.

Re:well....duh

[wierdling]

Unfortunately, due to hazy and unclear laws on what is considered to be unauthorized access in regards to computers and all things computer related, it is very likely that this technology (screen scrapers) can be considered "hacking" and the person/company who does it can be prosecuted. Is what we need is a much better definition of what is considered unauthorized access (if you spend 23 hours trying to guess my username and password and do it, that is unauthorized in my opinion, if I put a click through on my site that says only people who love science fiction can access the site, yet you hate science fiction but are doing a report on it so you click through to see what I have to say,that is not).

Re:well....duh

[Anonymous Coward]

Amazon says you can't.

Then again, Amazon can say whatever they want. They are more than welcome to put in their license a statement that says that by visiting their site you are willing to give up your first-born son to them.

Of course that doesn't mean it'll hold up in court.

Actually, sometimes they do.

[Anonymous Coward]

Two specific cases in point.

1) At many of the deal sites (i.e. slickdeals.net, etc) once in a while this offer appears where after getting back your rebate, you have more money than you spent for the product.

2) Grocery coupons - in some cases, a store will run one of those "triple coupon Thurdays" promotions, and if you have the right coupon, the money-off total will exceed the price of the product. Depending on the the store, money is returned, or a credit is.

web servers are not protected

[Provincialist]

accesses a protected computer without authorization, or exceeds authorized access [18 U.S.C. 1030(a)(4) of the CFAA]

How does one receive authorization to access a web server? Hmm, maybe with a simple html GET? The basic fact here is that of judicial cluelessness. If I put information on a public web server, pretend to "protect" it with a disclaimer (of everything) at the bottom of the page, and then get pissed off because somebody browsed that information, I'm an idiot. In addition, I am legless in court. Web servers make information available to the world. If I had wanted to make information available to certain parties that I trust not to compete with me, I should have set up a secure server with some provision for authentication and authorization.

It really is that simple

later,
Jess

Easy answer

[Lurgen]

Once their prices hit the Internet, they're in the public domain. It would be like posting your prices in the window, and complaining that a car driving past could photograph them.

We all know that bots crawl the web - Google, Altavista, spam-bots... they're all common knowledge. You put information on a website, and it's going to be viewed by an automated process. Surely with that knowledge, it's ridiculous to think you can ban people for using the information you've posted publicly in whatever way they desire.

Perhaps these companies (airlines, computer stores, whatever) need to start offering their services at the price they really mean to sell it for, rather than this stupid haggling they expect from us. Or maybe it's time they focused on quality of service, value-add, etc rather than price wars (which never help anybody in the long term).

Bottom line? If you don't want your competitors seeing your prices, don't make them available to them - this means no junkmail, no spam, no website, no prices in the store window, no prices inside the store, nothing.

"Automated" can mean a browser too.

[Anonymous Coward]

Just the act of clicking on a link sets in motion a series of automated software tasks that will deliver me a price. Millions of shoppers do this every day. Are they in violation as well?

hmm, anybody rfta?

[lingqi]

on the ground that Explorica had used confidential information obtained from EF to assist in obtaining this pricing information in violation of confidentially agreements executed by former EF employees now working for Explorica.

seems like it's the using confidential information part that got the scrapper capped.

I don't see why accessing *public* information be problematic.

the only thing that may be of trouble is the website EULA, but then the EULA would be saying the same thing as "don't visit my store unless you intend to buy," which would be rediculous in brick-and-mortar world (and should be similarly in cyberspace).

last question, though - why the heck would you ask this kind of stuff HERE? wouldn't a law-forum be a better choice?

Seek real legal advice.

[Gerad]

I am not a lawyer.

Slashdot is not a lawyer.

Slashdot is not a replacement for a lawyer.

Individual posters on slashdot may be lawyers, but are you really willing to trust your future to what some random person online says, when they could be a lawyer, but could also be some 14 year old kid who thinks it's amusing to screw with people?

Repeat after me:

I will seek proper legal advice.

Seriously, this comes up time and time again. If you're in a situation where you need actual concrete legal advice, SLASHDOT IS NOT THE PLACE TO GO. Sending in an Ask Slashdot is fine for theoretical questions, but when your ass is at stake if a lawsuit comes around, do you really want to trust your future to the legal advice given to you by Anonymous Cowards and karma whores?

Re:Just like Popeye

[tupps]

How much was postage?

Have you actually received your rebate?

click-through considered non-binding

[Provincialist]

If you have to agree to some sort of "terms of use" to get onto a website, you are bound by what those "terms of use" say. By clicking through, you have agreed to a contract and you have to abide by it.

Aside from the well-known problems with any click-through agreement (contract between unknown parties, software circumvention, lack of notarization, etc.), the additional flaw in this case is provided by web archives. If you don't want to have to look at a click-through page before reading your competitor's deep dark secrets, just download what you want from a public web cache. Are these jokers going to turn around and sue Google, as well?

Actually, that brings up an interesting point. When Google gets sued for forwarding information to competitors without click-throughing them, they will probably deny that such was not their "intent" in providing the web archive. Of course, the competitors do have an "intent" that the original site doesn't condone. But there is not a technical means of determining intent over the current version of HTTP. If the original site wants to do this, it is using the wrong technology. Of course someday if the ebXML folks get off their collective butts, we might have some sort of contract-negotiation protocol. I doubt a consumer e-commerce site would be interested in erecting such barriers to entry, but this would probably be useful in certain B2B contexts. Until then, honoring click-through pages in the breach will only harm the internet. Any court case that declares that particular intents make a party ineligible to download particular material served over the web (that's my understanding of the agreement that we're clicking through here) will only harm the web and all open systems.

later,
Jess

Re:If they don't want the price stolen.

[Theatetus]

Output the price in a un-OCRable jpeg image.

What did blind people ever do to you?

Adam Smith and perfect information

[viniciusxp]

People who studies economics faces some irreal hipothesis in text books. The first topic most students have to deal with when taking the microeconomics course is when you have a big group of firms selling the very same product. If the buyer has perfect information about prices hi will choose the lowest price. The buyer's choice will influence the behaviors of all other firms that will tend to get their prices down to beat the one choosed by the buyer. We will have a dynamics that will make the price go down until the item will cost to the user the same it costs to be produced. In the real world it is very unrealistc to believe someone could have information about all the sellers prices. But with Data Minig we can have MORE information about sellers than in the real world, and we can access this infrmation with a smaller cost. We should then be nearest to perfect competition books theorize than in the real world. There is although some problems to solve before jumping to this conclusion: There is not that big number of firms competing, delivery fees, warranty and time of arrival of the product can be very different from seller to seller. Could a "perfect bot" could handle all this information. If the answer is positive firms can folow two paths : cartelization or dumping. The first one happens when firms pacts prices together and force buyers to py more, because competition is "freeze". The second one hapens when the firm artificially gets down the price to a lower level than the costs to force the competition to bankrupcy. Both behaviors are dangerous to consumers and are forbiden in most countries. IMHO a site's EULA can't go agains market law. I presume that, at least inside the same democratic country, it is legal to data mine in that way. And I can't see why a competitor can't use it as tool to build it's price strategy. It's the invisible hand Adam Smith's intuished about. The WWW is evolving, maybe in a way some people can dislike, and is using the same rules we use in the real world to make money. And I'm sure competitor will soon find solutions to prevent data mining from their sites, at least information they don't want to share. IT solutions. That do not require lawyers but intelligence and insight.

you are stating the obvious

[g4dget]

Every time someone asks a legal question on Slashdot, people like you come out of the woodwork and point out that Slashdot is the same as going to a lawyer. You don't say. Do you really think everybody else is stupid?

Discussing legal issues is not just a business for lawyers. Non-lawyers can give each other useful pointers. And non-lawyers actually have an obligation to determine whether their legislators are doing a good job with the laws they enact and judges they appoint, and a healthy discussion is a good start.

Secondhand legal advice

[yerricde]

Slashdot is not a replacement for a lawyer.

Slashdot is useful to get a sense of what the legal landscape is like. Some comments are to the effect: "I am not a lawyer, but my lawyer told me this." Or "I am not a lawyer, but here is the statute [cornell.edu], and here is how a court has interpreted it [eff.org]." When you do see an attorney after reading the comments, you don't have to wait for the attorney to explain the basics. This saves time, and time is money, especially at the typical copyright and trade secret specialist's rate.

That said, you're right about one thing: anything you read on Slashdot is not legal advice.

Other Rule of Thumb to Live By

[Rick.C]

If you fear that it might be the wrong thing to do, it probably is.