Is Data Mining for Product Pricing, Illegal?

wessman asks: "I started to read Orin S. Kerr's 80-page paper looking for how his proposal would pertain to: ripping music/movies, P2P, corporate espionage, and lastly, the use of web scraper robots. Little did I know just how relevant his paper would be in regards to that last item! Kerr makes note of EF Cultural Travel v. Explorica in which Explorica is caught hiring a consultant to program a scraping robot to gather pricing information from a competitor, EF Cultural Travel. Well, I do consulting on the side from home and am currently working a project whereby I gather pricing information from all the major travel conglomerates (Orbitz, Expedia, Lodging.com, WorldRes, Sabre, etc.) so that the travel booking business that hired me can meet or beat all their prices. Granted, the circumstances of the Explorica case are different and the case was an example of an extreme ruling, but my questions to the Slashdot community are: Do I notify the company that hired me of the Explorica case? Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually? Should I continue with this project and the similar projects I do in this area of programming?" Now, add in the text in the "deliverables" section of this press release and it seems we may have some contradictory information. Who is right, and under what circumstances is price harvesting off of the internet not allowed?

Comments

I swear

[cultobill]

In what sane land would PRICES be protected under law? You can't really keep them secret, so "trade secret" is right out. It's not a identifying mark (unless you're a dollar store), so much for trademark. There's nothing useful that hasn't been done before, fuck patenting them. Copyright? It's a simple derivation of what the supplier charges you.

There is nothing creative about pricing stuff. Good lord.

Re:I swear

[Gortbusters.org]

Look at pricewatch.com, it already goes around collecting price data from many online stores.

I don't see why this is such a big problem... one site creates competitive prices based upon other sites' prices. In reality if a consumer reaches your site on the internet for your product they probably didn't do it by accident. They will evaluate all aspects of the business (licensing, service/support, upgrade cost, security of the site, etc) before they jump ship to another site to save a few bucks.

I'm sure Dell, Sony, and HP already keep an eye on eachother's prices.

Re:I swear

[Sparr0]

Pricewatch doesnt mine. Companies PAY for the privelege of listing their prices on pricewatch.

Re:I swear

[whereiswaldo]

Here's a niche for P2P software... obtaining data from a website in a distributed way so as not to stick out in the website logs the way a one or a few download clients would. The collected data could be processed remotely or uploaded to a central server.

Re:I swear

[anonymous loser]

Pricewatch doesnt mine.

No, but froogle.google.com does.

Re:I swear

[Fat Casper]

I don't see why this is such a big problem... one site creates competitive prices based upon other sites' prices.

You don't understand the fundamental use of law under the Corporate Administration. Data mining is legal when it involves Admiral Poindexter, your grocery store, your viewing habits or your medical data. Data minig is illegal when it benefits you the consumer at all, much less at the expense of a needy company's profits.

Re:I swear

[smack_attack]

We don't live a capitalist society... we live in what's called a "mixed economy".

Essentially it goes something like this: government supports business rights over individual rights; if two business' butt heads, then the government supports whichever one has more "pull" with the government.

Pretty slick market we have in place eh?

Re:I swear

[CTho9305]

WARNING: You can see I write crappy perl / shell script. I make no guarantees about this code.

NOTE: replace all instances of "ABC" with "|".

parse.sh:
#!/bin/bash
#Copyright CTho9305 2003. You are given permission to redistribute this file provided this copyright notice is left intact. You may modify it as you want. Please share any modifications (you are not required to).
#barton
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i xp ABC grep 333 ABC cut -f1,4 -d ' ' > XP333.dat
#XP
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i xp ABC grep -v 333 ABC cut -f1,4 -d ' ' > XP.dat
#Apparently I have too many junk characters
#MP
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i mp ABC perl -pe 's/\.//' ABC perl -pe 's/GHz/00/' ABC cut -f1,4 -d ' ' > MP.dat
# Celeron
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i celeron ABC cut -f 1,3 -d ' ' ABC perl -pe 's/ 1GHz/ 1.0GHz/;s/\.//;s/GHz/00/' > celeron.dat
#P4
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//;s/\s+/ /' ABC grep -i 'pentium 4' ABC egrep -i "sockABC2\.ABC3\." ABC grep -v -i 400MHz ABC perl -pe 's/ 533MHz//;s/GHz/00/;s/ Sock 478//;s/\.//' ABC cut -f1,4 -d ' ' ABC cut -b -8 > pentium4.dat
gnuplot gnuplot.script > ~/www/out.png

Anyway, I wrote this because I was bored and wanted to see what a good price point was for current Athlons. If you examine the graphs carefully you might note that the XP's are not properly differentiated. Some are 333s and marked as that, others aren't marked properly, etc. With the new 400s, it gets worse. For the P4s, I got a little luckier because the speed ranges don't overlap as much. I think I'm going to not differnetiate between the various FSBs of Athlon XPs because the prices are close enough anyway.

Anyway, it has served its purpose by helping me find a point where the processors are reasonably fast, and the bang for the buck is decent.

gnuplot.script

#Copyright CTho9305 2003. You are given permission to redistribute this file provided this copyright notice is left intact. You may modify it as you want. Please share any modifications (you are not required to).set terminal png color
set xlabel "Speed (MHz or rating)"
set ylabel "Cost ($USD)"
set title "Speed vs. Cost"
set grid
set time
set linestyle 1 lw 3
plot "XP.dat" using 2:1 title "XP" with linespoints, \
"XP333.dat" using 2:1 title "XP333" with linespoints, \
"celeron.dat" using 2:1 title "celeron" with linespoints, \
"MP.dat" using 2:1 title "MP" with linespoints, \
"pentium4.dat" using 2:1 title "P4" with linespoints

Anyone know how to change the text font, or the thickness of the lines?

Sample output

Re:I swear

[Sloppy]

Actually, there's a lot of creativity that goes into pricing things.

Ever see this: $39.95 ? The .95 cuts .05 cents off the price but greatly increases the chances you'll buy it.

How about 2/6.00? Unless they say otherwise, that means 1's only $3, but again, you're more likely to buy the two. A supermarket by me commonly sells things like limes and small bags of snacks at 4/$1.00, rather than just $.25, which i guess seems cheap.

A lot of the delis around me have sandwiches for 4.63, which with tax is exactly a finski. No numeric change means they can ring you out faster.

Perhaps the first time someone came up with these techniques, it was an act of creativity. But they are now well-known. When someone prices something as 39.95, they are not being creative; they are merely "using the .95 trick."

Besides this, figuring out what things are worth is a very exact science when you live, as we do, in a vastly service based economy. Prices have nothing to do with the actual cost of physical production, and everything to do with the public's perceived value of the product in question. This is how GM can make two identical trucks, label one GMC, and get another $3000 for it because GMCs are "professional grade and therefore more reliable." This is how Victoria's Secret can get $30+ for maybe a quarter square foot cloth and a pair of wires, and in fact get more money when they use less material or "exotic" materials like satin (which often cost less than cotton).

Ok, these are much better examples of creativity. The person doing the pricing is actually doing work and making real decisions.

But does that entitle their work to special protection? Let's go back and look at why we have "IP laws" at all, lest we forget the purpose and perversely misapply the law. In the United States, the Constitution empowers congress on this basis:

To promote the progress of science and the useful arts by securing for a limited time to authors and inventors the exclusive right to their respective rights and discoveries...

It is effort for you to write a book, and if you didn't have a monopoly on the sales of that book, your effort would be wasted from a commercial perspective and you might not choose to expend that effort. So we give you a copyright. You get what you want -- compensation for your effort -- and we get what we want -- promotion of the useful arts. Perhaps that promotion is somewhat delayed (until after the copyright expires) but we'll eventually get it.

Now let's look at your sophisticated pricing example, in that light. You expend effort in figuring out just the right price for your widget, and you would like a monopoly on the fruits of that work. If we grant that to you, then you get what you want -- compensation for your effort -- and we get ... hey, what do we get? Does the creative act of figuring out that $30 is the most profitable price for your underwear, help us in some way? Does it promote the progress of the sciences or useful arts? I don't see how.

I see that there's something in it for me, to grant copyright on creative works such as books. I don't think there's anything in it for me at all, to grant copyright on prices, even prices that take effort to optimize.

And without a monopoly, you won't have sufficient incentive to do the work of determining the most profitable price for your satin underwear or truck? As if!

Re:I swear

[Pharmboy]

We sell two products that are the exact same item, except we photographed one item with an "optional" part missing, even though we ship it with it. One is priced $1599, the other is priced $1899. The one for $1599 has an uglier picture, and is described simpler. It just has a different label on it. We sell about 3x more of the item at $1899. The idea is to average 1799 for each one sold.

We use the 1599 to get more people to the site, then promote the 1899 unit more. All descriptions are 100% accurate, we just make the 1899 unit look better, and point out more benefits. This was my idea. Sad thing, I haven't gotten a bonus for it yet.

Just like Popeye

[Anonymous Coward]

Anybody remember the Popeye cartoon where Popeye opens a car wash, and then Bluto opens a car wash right across the street?

When Popeye posts his price, Bluto beats it by five cents. Then Popeye beats Bluto's by five cents.

It goes back and forth until Popeye is washing cars for free.

I was tempted to call this an infinite loop, but I doubt a retailer would pay you to take their products.

Re:Just like Popeye

[87C751]

I was tempted to call this an infinite loop, but I doubt a retailer would pay you to take their products.

Last month, I got a $40 rebate on a $39.95 router.

Re:Just like Popeye

[tupps]

How much was postage?

Have you actually received your rebate?

Re:let me ask you this.

[slamb]

An Anonymous Coward asked: Is the comma near the end of this question, necessary?

Of course it is. Let's dissect this sentence:

Is Data Mining for Product Pricing, Illegal?

• Subject: Data (from Star Trek, I suppose)
• Verb: Mining
• Object: Product pricing
• Listener: Illegal

Not having the comma would completely distort the meaning of the sentence.

TITLE HAS POOR GRAMMAR

[Anonymous Coward]

EDITORS: A better way to phrase that headline is as follows:

Is It Illegal to Data-Mine for Product Pricing?

Thank you.

Re:TITLE HAS POOR GRAMMAR

[hazem]

Or if you're really married to using that comma:

"Data Mining for Product Pricing, Is It Illegal?"

Commas just don't get used enough. Maybe he just wants to get more even wear on his keyboard. I'm sur we can expect more tildes soon!

Re:TITLE HAS POOR GRAMMAR

[dipipanone]

Commas just don't get used enough.

Perhaps not, and I don't wanna get all grammar -nazi prescriptive here, but if I was writing that strapline, I would have used a colon.

Data Mining for Product Pricing: Is It Illegal?

Re:TITLE HAS POOR GRAMMAR

[Anomalous Cowturd]

Like the previous poster said, you would more properly use a colon. The better way to write the original subject would be

"Is Data Mining, For the Purposes of Product Pricing, Illegal?"

which does not make for a snappy headline. I believe

"Is Data Mining, For Product Pricing, Illegal?"

would also be correct.

I'd like to see more semicolons, myself, but nobody really knows how to use them.

Ummm..

[deadgoon42]

My grandmother calls this "shopping around." The only difference is that someone else is doing all the work.

Re:Ummm..

[Codex The Sloth]

My grandmother calls this "shopping around."

Damn! I was going to patent this but it sounds like there is some "prior art"...

Price Scraping?

[Anonymous Coward]

Actually, price scraping was done in a very low-tech way a reasonably long time ago by a pretty well-known businessman: Mr. Sam Wal-Mart. Early in his career, he would dumpster-dive his competitors to find out the prices his competitors paid for their goods, the contracts they had with their suppliers, etc. This provided him with "insider information" so he'd know how to prepare his pricing in a more strategic fashion and obviously out-compete them where it counts: financially.

Re:Price Scraping?

[Anonymous Coward]

Wrong. Read Sam Walton: Made in America: My Story. Sam Walton says that was a story put out by his competitors to disparage his name and he never did anything of the sort.

TOS and more ideal markets

[weston]

I think any large business where the pricing structure isn't directly related to costs is probably deeply afraid of agents that aggregate their data with competitors. You end up with a more ideal market, a more frictionless market, if you will, and they'll be forced to compete on narrower and narrower margins of profit. Of course they'll want to throw up barriers to that.

But I'll bet this issue comes down to Terms of Service and what a company can reasonably expect to be able to legally require/forbid about the use of data provided via an automated means...

You have a contract?

[gpinzone]

As long as you get paid, let them worry about the lawsuit. They're the ones who are going to actually use it. Keep your mouth shut.

Re:You have a contract?

[vandan]

Would you apply this line of reasoning to ALL areas?
What if the job were researching Bush's all-feared biological weapons?
Or GM products?

The problem is that if everybody decides to look the other way (and everyone can find a reason why they should take their money and shut up), then some pretty fucked up things get done, and people are left wondering, "How did it come to this?"

Now I'm not saying that the world is going to end because someone's harvesting prices off the net. I'm just questioning your "Me first, no-one else 2nd" argument.

Re:You have a contract?

[stiller]

I do hope that every /. reader at least can discriminate between building life-threatening devices and products that may or may not just fall under some crazy copyright law. It's the user's responsibility not his. They assigned him, they carry the consequences.
It's your way of extreme reasoning that gets us nowhere. We must be reasonable, and it's reasonable to expect that no judge in his right mind would convict anyone for this.

Rules of Thumb to Live By

[release7]

I'm not going to pretend to know what the laws are in this case. It seems that the only law these days is "He who can afford the best, most impressive lawyer wins." Here's some other cynical words of wisdom I've come to believe:

If powerful people get screwed, it's illegal.
If it forces large corporations have to work harder to earn a profit, it's illegal.
If it give the little guy a leg up or levels the playing field in any way, it's illegal.
If it's illegal and you're big and powerful, don't worry about it, you can probably get away with it with little damage to your business or career and keep almost all of you cash minus legal fees.

Re:Rules of Thumb to Live By

[urbanRealist]

It seem like you're right. In no way shape or form did anyone break either the spirit or the letter of the law here. Yet,

First Circuit holds that prohibitions found in a website's Terms of Use can be used to establish that a visitor to that site exceeded his authorized use thereof for the purposes of establishing a violation of the Computer Fraud and Abuse Act

Which says to me I can say, "oh, you can't read my website while wearing sunglasses." I could be wrong here, but according to this ruling, I can just put up any restriction I can dream up and you're bound by it when reading my website. That ruling truly must have cost a fortune. The other down side is that data mining is probably one of the most useful developments in recent times. Look at Google News.

Re:Rules of Thumb to Live By

[poot_rootbeer]

It seems that the only law these days is "He who can afford the best, most impressive lawyer wins."

I can see how someone could be fooled into believing this, if the only news they get about what's happening in the courts is the fearmongering they read in "Your Rights Online."

I suggest you obtain a copy of the verdict records from a court that deals with a wide variety of cases. Ten bucks says you'll come to the conclusion that regardless of who the legal counsel is on both sides, justice is truly served far more often than not.

huh?

[Anonymous Coward]

Who is this "Illegal" person and why are we asking him questions about Star Trek characters?

I'm not "Illegal," but I'll answer. Seeing as how he was killed off in the last movie, I think it's safe to say that no, Data is not mining for product pricing.

(In other words, you illiterate clods need to be more careful with your commas.)

well....duh

[mrpuffypants]

Look, I I can visit a web site and the business (Let's say Amazon) publicly posts their prices for anybody to see then you sure as hell can use them! If suddently using bots to do work are illegal then I'd wadger that every shell script that I write is an affront to US Laws. Rotating log files and all sorts of other "make my job easier so that I can play Quake" scripts are perfectly legal, so how the hell can it be questionable just to go to a site and record prices???

Jebus, please help the Unites States Gub'ment!

Re:well....duh

[Some Dumbass...]

so how the hell can it be questionable just to go to a site and record prices???

Let's not forget that this was a legal ruling, and some of judges don't know squat about technology. The lawyer for the defense probably showed the judge a web page with prices on it and the judge assumed it was "hacking" or somesuch.

Re:well....duh

[Jeremy Erwin]

Amazon says you can't.

Amazon.com grants you a limited license to access and make personal use of this site and not to download (other than page caching) or modify it, or any portion of it, except with express written consent of Amazon.com. This license does not include any resale or commercial use of this site or its contents; any collection and use of any product listings, descriptions, or prices; any derivative use of this site or its contents; any downloading or copying of account information for the benefit of another merchant; or any use of data mining, robots, or similar data gathering and extraction tools. This site or any portion of this site may not be reproduced, duplicated, copied, sold, resold, visited, or otherwise exploited for any commercial purpose without express written consent of Amazon.com.

I am guessing that the prohibition on "visit[ing] for any commercial purpose" precludes me from actually purchasing their wares.

Re:well....duh

[wierdling]

Unfortunately, due to hazy and unclear laws on what is considered to be unauthorized access in regards to computers and all things computer related, it is very likely that this technology (screen scrapers) can be considered "hacking" and the person/company who does it can be prosecuted. Is what we need is a much better definition of what is considered unauthorized access (if you spend 23 hours trying to guess my username and password and do it, that is unauthorized in my opinion, if I put a click through on my site that says only people who love science fiction can access the site, yet you hate science fiction but are doing a report on it so you click through to see what I have to say,that is not).

Actually, sometimes they do.

[Anonymous Coward]

Two specific cases in point.

1) At many of the deal sites (i.e. slickdeals.net, etc) once in a while this offer appears where after getting back your rebate, you have more money than you spent for the product.

2) Grocery coupons - in some cases, a store will run one of those "triple coupon Thurdays" promotions, and if you have the right coupon, the money-off total will exceed the price of the product. Depending on the the store, money is returned, or a credit is.

web servers are not protected

[Provincialist]

accesses a protected computer without authorization, or exceeds authorized access [18 U.S.C. 1030(a)(4) of the CFAA]

How does one receive authorization to access a web server? Hmm, maybe with a simple html GET? The basic fact here is that of judicial cluelessness. If I put information on a public web server, pretend to "protect" it with a disclaimer (of everything) at the bottom of the page, and then get pissed off because somebody browsed that information, I'm an idiot. In addition, I am legless in court. Web servers make information available to the world. If I had wanted to make information available to certain parties that I trust not to compete with me, I should have set up a secure server with some provision for authentication and authorization.

It really is that simple

later,
Jess

Easy fix.

[Photar]

The problem is that when you sell a commodity like a TV or a vcr and the only difference between them is price you can't exactly maintain a high profit margin. What they need to do is obfuscate the prices so that its next to impossible to compair products. Thats how cell phones work.

This phone has 500 any time minutes for 3 cents a minute from your calling area roaming is 10 cents a minute, unlimited text messaging, 800 night and weekend minutes is free for the first 6 months and has rollover.

This other phone has 1000 minutes for 2 cents a minute but with out rollover and text messaging costs 1 cent per message, night and weekends are free but don't start till 9pm.

See its not exactly easy to just look at the plans and see which one is the better plan.

Re:Easy fix.

[canajin56]

Scott Adams refers to this as a "confusopoly" Telephone companies and airlines use them. Since you can never tell who has the best price, they can all remain in business.

Easy answer

[Lurgen]

Once their prices hit the Internet, they're in the public domain. It would be like posting your prices in the window, and complaining that a car driving past could photograph them.

We all know that bots crawl the web - Google, Altavista, spam-bots... they're all common knowledge. You put information on a website, and it's going to be viewed by an automated process. Surely with that knowledge, it's ridiculous to think you can ban people for using the information you've posted publicly in whatever way they desire.

Perhaps these companies (airlines, computer stores, whatever) need to start offering their services at the price they really mean to sell it for, rather than this stupid haggling they expect from us. Or maybe it's time they focused on quality of service, value-add, etc rather than price wars (which never help anybody in the long term).

Bottom line? If you don't want your competitors seeing your prices, don't make them available to them - this means no junkmail, no spam, no website, no prices in the store window, no prices inside the store, nothing.

Examples...

[oaf357]

Buy.com was built around a huge server farm that scoured the web and found the best prices for products it sells and then beat those prices (to the best of its ability). That has changed a little now but buy.com was built from that.

Also, Pricewatch, Pricegrabber and Froogle scour the web for prices and create search engines out of them so consumers can find the best price.

I'm not saying just because everyone else is doing it means you can too (and you might have a slightly different objective causing these examples to be weightless) but it's being done all over the place.

Hope that helps.

The US Code...

[Pettifogger]

It seems that the US Code and the First Circuit make this one pretty clear. If you have to agree to some sort of "terms of use" to get onto a website, you are bound by what those "terms of use" say. By clicking through, you have agreed to a contract and you have to abide by it. If your competitor's site requires you to agree to such terms, and those terms prohibit data mining, then you can't data mine there. Simple.

As for the ethical part of telling your employer about this... well, first, remember, this is just a decision of the First Circuit. If you live in a different Circuit, then it may or may not be binding on you. I know this jurisdictional stuff can be a little confusing, but a decision by a Circuit only affects the jurisdictions within it. Only the US Supreme Court (generally, I know there are federal tax, patent, admiralty, etc. courts, too) can make decisions that are binding on the entire country. If you're not sure, check with your corporate counsel. And it might be a good idea to forward the case to him anyway, you might be able to pick up some "bonus points" from your boss for being an especially conscientious employee.

hmm, anybody rfta?

[lingqi]

on the ground that Explorica had used confidential information obtained from EF to assist in obtaining this pricing information in violation of confidentially agreements executed by former EF employees now working for Explorica.

seems like it's the using confidential information part that got the scrapper capped.

I don't see why accessing *public* information be problematic.

the only thing that may be of trouble is the website EULA, but then the EULA would be saying the same thing as "don't visit my store unless you intend to buy," which would be rediculous in brick-and-mortar world (and should be similarly in cyberspace).

last question, though - why the heck would you ask this kind of stuff HERE? wouldn't a law-forum be a better choice?

Re:hmm, anybody rfta?

[Anonymous Coward]

I work for one of the major online travel sites.

If you screen scrape us, and we notice it (and we very often do), your IP address will be blocked by our firewalls.

What most people don't realize is that very often any search performed on these sites costs the company money. In many cases, if you search for, say, a hotel on Expedia or Orbitz or Travelocity, those companies are paying one of the major Hotel reservation systems for their results from that search.

So, if someone is screenscraping our site, each search they perform to grab prices for a hotel for a day requires us to send some money to Pegasus or Travelweb or one of the other biggies. In hotels, for example, Pegasus is the big CRS. In order to get enough bookable hotels to make thier site useful, Expedia and Orbitz and Travelocity all need to buy search results from Pegasus.

So it's not just a matter of taking their bandwidth and CPU time.

We do allow some companies to screen scrape our site for some various reasons, but they all pay us for the privilege. We don't tend to take legal action against unallowed screen scrapers, but we will do what we can to make their life harder.

froogle seems to do exactly this

[dakainivanua]

Doesn't Google's Froogle service do exactly this?

Seek real legal advice.

[Gerad]

I am not a lawyer.

Slashdot is not a lawyer.

Slashdot is not a replacement for a lawyer.

Individual posters on slashdot may be lawyers, but are you really willing to trust your future to what some random person online says, when they could be a lawyer, but could also be some 14 year old kid who thinks it's amusing to screw with people?

Repeat after me:

I will seek proper legal advice.

Seriously, this comes up time and time again. If you're in a situation where you need actual concrete legal advice, SLASHDOT IS NOT THE PLACE TO GO. Sending in an Ask Slashdot is fine for theoretical questions, but when your ass is at stake if a lawsuit comes around, do you really want to trust your future to the legal advice given to you by Anonymous Cowards and karma whores?

Why waste time in the legal system?

[x00101010x]

Filing lawsuits to protect your price information is just dumb, not to mention waste (if not abuse) of the legal system.

Personal feelings about freedom of information aside, and just from a coder's POV, here's my solution.

If they really want to avoid getting scraped, they should just get their existing, underpaid web developers to create a backend setup that generates the prices as gif's that give OCR hell (such as those used to prevent automated registration of say Yahoo! email accounts).

Coders are cheaper than lawyers (at least those needed to write such code as this).

Sure, the compition could pay more money to get somebody to develop better OCR to read each and every dynamically generated GIF, but most people require proof reading of OCR data, which leads to even more cost.

Something I learned from my Uncle who works with the DOD is this: Any lock can be picked; Any encryption can be broken. It's just a matter of if it's worth the time and money to get what's inside.

In short, with a little one time cost, the company that doesn't want it's prices scraped can just make it so hard to scrape their prices that it's not worth it. The price of scraping the graphically displayed price tags would also be an ongoing cost of software and proofreaders that would dip into profit margins, which management at the company that desires the scraping won't like.

It's not perfect, but it's better (and more bankable) than going whining to the legal system. (Especially since coders are generally cheaper than lawyers).

Ah, just the comment I was looking for.

[ahfoo]

I think what you have to look at is the media context in which the prices are displayed.
It's quite true that many stores will try to prevent you from making recordings of any kinds on their physical premises. I've been reprimanded by store managers many times for taking photos in the store. But their right to prevent me from creating media on their premises is based on their property rights, not any some legally backed authority to censror the media.
The web is a totally different story. I use web scrapers all the time and a site that doesn't like it can kindly take its ass off the web. Once you place material on the web, it is published. If you don't want to publish your prices, you don't have to. That's like publishing a book and complaining the readers read it too fast.
The people who compain about such things are the idiots who create unworkable business plans based on their own assumptions about how people are going to use the resource. This is an interesting issue with news media that want to sell access to their archives. There's no way they can both publish to the web and prevent me from caching old copies. If that's the business plan then web publishing is an inappropriate business decision and guess who should pay for bad business decisions: the consumer, or the fool who pursued an ignorant business plan?

Same as in the Physical World

[serutan]

Anybody who publishes information about their business runs the risk that a competitor will get hold of the information and use it in some way. This has always been a fact of life in the physical world. As computers came online in recent decades many companies have maintained databases of information about competitors' products. The Internet doesn't change any of this.

click-through considered non-binding

[Provincialist]

If you have to agree to some sort of "terms of use" to get onto a website, you are bound by what those "terms of use" say. By clicking through, you have agreed to a contract and you have to abide by it.

Aside from the well-known problems with any click-through agreement (contract between unknown parties, software circumvention, lack of notarization, etc.), the additional flaw in this case is provided by web archives. If you don't want to have to look at a click-through page before reading your competitor's deep dark secrets, just download what you want from a public web cache. Are these jokers going to turn around and sue Google, as well?

Actually, that brings up an interesting point. When Google gets sued for forwarding information to competitors without click-throughing them, they will probably deny that such was not their "intent" in providing the web archive. Of course, the competitors do have an "intent" that the original site doesn't condone. But there is not a technical means of determining intent over the current version of HTTP. If the original site wants to do this, it is using the wrong technology. Of course someday if the ebXML folks get off their collective butts, we might have some sort of contract-negotiation protocol. I doubt a consumer e-commerce site would be interested in erecting such barriers to entry, but this would probably be useful in certain B2B contexts. Until then, honoring click-through pages in the breach will only harm the internet. Any court case that declares that particular intents make a party ineligible to download particular material served over the web (that's my understanding of the agreement that we're clicking through here) will only harm the web and all open systems.

later,
Jess

That Reminds Me...

[Esion Modnar]

of the old joke about this fella who was selling everything in his store at a loss.

When somebody asked him how he can make a living like that, he replied:

"Volume!"

Are Extraneous Commas, Annoying?

[reconn]

I'd say yes. Second only to apostrophe's.

Read the case...

[anubis]

Read the case...EF Cultural Travel BV v. Explorica hinges on the fact that the defendant company hired an ex-programer from the plaintif company. The programmer had special knowledge of codes used in the pricing (which he had signed a confidentiality agreement not to disclose). When he made the scrapper program he violated the confidentiality agreement.

It was the violation of the confidentiality agreement that the court held was illegal.

As for whether you should tell your employer, it depends on your employment agreement! :) Depending on how the contract is written you could be jointly liable.

While this is a 1st Circuit case, it has been followed by the 5th Circuit (Ingenix, Inc. v. Lagalante) and cited in cases in the 7th and 9th Circuit.

Hope this helps.

--me

I'm reminded

[Nexzus]

of a man named Ronald Kahlow and his troubles with Best Buy back in 1997.

Re:I'm reminded

[shumacher]

Both Best Buy and the judge were off their rocker. Best Buy can ask him to leave. The store is private property. If he is a customer, they should have allowed him to continue to calculate prices. If Best Buy asked him to put his laptop away, he should have complied. Further, the article mentions other stores that claim to not have a policy against Mr. Kahlow's behavior. I know for a fact that one of them actually is okay with Mr Kahlow's behavior, but does have a policy (and not just an "unwritten" one)against competitor's doing what Mr. Kahlow was doing. (Which is a different thing - and more on topic - one means a potential sale for them, the other means potential sales for their competition) Their policy is to ask the suspected competitor what they are doing, and then if confirmed as a competitor, they are to make a specific statement essentially revoking that individual's right to visit the store, and then, they are to ask them to leave. When that store shops a competitor, policy compels them to wear normal street clothes, to not use any automated devices, and to admit their affilliation and leave if asked.

Re:If they don't want the price stolen.

[Theatetus]

Output the price in a un-OCRable jpeg image.

What did blind people ever do to you?

Not the really data mining?

[fatwreckfan]

What he's doing isn't really data mining. Data mining is the process of discovering patterns in data which are not known ahead of time, such as the infamous "beer and diapers" correlation.

That said, I don't understand why the author is worried. I can't see how looking at publicly posted prices could be considered illegal.

Adam Smith and perfect information

[viniciusxp]

People who studies economics faces some irreal hipothesis in text books. The first topic most students have to deal with when taking the microeconomics course is when you have a big group of firms selling the very same product. If the buyer has perfect information about prices hi will choose the lowest price. The buyer's choice will influence the behaviors of all other firms that will tend to get their prices down to beat the one choosed by the buyer. We will have a dynamics that will make the price go down until the item will cost to the user the same it costs to be produced. In the real world it is very unrealistc to believe someone could have information about all the sellers prices. But with Data Minig we can have MORE information about sellers than in the real world, and we can access this infrmation with a smaller cost. We should then be nearest to perfect competition books theorize than in the real world. There is although some problems to solve before jumping to this conclusion: There is not that big number of firms competing, delivery fees, warranty and time of arrival of the product can be very different from seller to seller. Could a "perfect bot" could handle all this information. If the answer is positive firms can folow two paths : cartelization or dumping. The first one happens when firms pacts prices together and force buyers to py more, because competition is "freeze". The second one hapens when the firm artificially gets down the price to a lower level than the costs to force the competition to bankrupcy. Both behaviors are dangerous to consumers and are forbiden in most countries. IMHO a site's EULA can't go agains market law. I presume that, at least inside the same democratic country, it is legal to data mine in that way. And I can't see why a competitor can't use it as tool to build it's price strategy. It's the invisible hand Adam Smith's intuished about. The WWW is evolving, maybe in a way some people can dislike, and is using the same rules we use in the real world to make money. And I'm sure competitor will soon find solutions to prevent data mining from their sites, at least information they don't want to share. IT solutions. That do not require lawyers but intelligence and insight.

House rules

[scgops]

You see signs everywhere you go:

-Shirt & shoes required.
-No loitering.
-No soliciting.
-Check all bags at counter.
-No more than two students allowed in store at one time.
-Parking lot, bathroom, etc. for customer use only.

Just because a building (or a web site) is in a public place doesn't mean that everyone is free to do whatever they want. Business owners are free to create house rules that everyone needs to follow.

Similarly, web sites can legally restrict what you are allowed to do when you visit them without having to build security measures to force compliance. If web retailers don't mind robots harvesting their inventory and prices, great. If, however, they want to place restrictions on who can access their site and how, that's entirely their prerogative.

Think about it. Leaving the door to your home unlocked would make it easier for people to steal your stuff, but it still wouldn't make it legal for them to do so unless you put up a sign saying something like, "Free for the taking."

Web scrapers are legal to develop and they're legal to use on sites with acceptable use policies that allow them. However, your customers should be prepared for the possibility that some or all of their competitors could make them stop using it at some point. And, in the interests of maintaining your own professional ethics, you should probably call their attention to the issues surrounding the job they're asking you to do.

you are stating the obvious

[g4dget]

Every time someone asks a legal question on Slashdot, people like you come out of the woodwork and point out that Slashdot is the same as going to a lawyer. You don't say. Do you really think everybody else is stupid?

Discussing legal issues is not just a business for lawyers. Non-lawyers can give each other useful pointers. And non-lawyers actually have an obligation to determine whether their legislators are doing a good job with the laws they enact and judges they appoint, and a healthy discussion is a good start.

Re:you are stating the obvious

[Jerf]

You don't say. Do you really think everybody else is stupid?

Stupid? No. But the number of people who seem to think they are lawyers is very large, and not just on Slashdot either. I can't count the number of times in my real life I've discussed intellectual property issues and not only has the other person been very, very wrong, but I was not even able to get them to listen to me.

I'm not a lawyer, but I've taken a close interest in that sort of thing and I know the basics very well.

As a Slashdot example, basically, if someone is insisting that they have a "fair use" right to something, unless they are jusitifying it with reference to the four criteria used to determine if something is fair use, they're wrong.

People seem to need periodic reminding that they aren't lawyers, and other non-lawyers aren't lawyers.

For computer examples, how many times have you heard someone around you give an incredibly wrong reason for a crash... and stick by it, even after you fix it, because of course they're right?

Discussion of issues is one thing. Talking about something that could make or break a career, that's a time for a real lawyer, not hundreds of people who think they are lawyers.

Please read Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments. I don't know if that's risen to "classic paper" status but as far as I'm concerned it has.

er, BS:Re:Best buy is a really really bad example.

[randyest]

Sorry, I call bullshit.

Over the past 6 or 7 years I've used a palm (handspring visor, to be more precise) hundreds of times, in every Best Buy (and Circuit City, MicroCenter, etc.) in the Boston area to record prices. I've never had anyone even look at me funny.

Maybe it's related to how guilty (or difficult to remove) you look, but I really doubt that happened to anyone ever (note the once-removed story -- it's always a 'friend of mine' in these types of stories.)

In any case, what kind of wuss would leave without making a fuss and forcing them to call the police over something so ridiculous? I could be using my palm to look up my friend's number to call and ask which video card to buy. Fsck them if they don't like it.

Or, maybe this particular Best Buy was located in an airplane and the event happened during takeoff or landing. Or your friend lied to you. One or the other.

Public servers are really private

[mypalmike]

While I agree wholeheartedly with your argument, the courts do not. Another example from the same paper is even scarier. In a similar case (Register.com vs. Verio), the court said the use of search robots to gather information may be illegal regardless of whether it broke the site's terms of service:

"Instead, the Court concluded that the mere fact that Register.com had decided to sue Verio meant that Verio's use of the search robot was without authorization: 'because Register.com objects to Verio's use of search robots,' the Court held, 'they represent an unauthorized access to the [Register.com] WHOIS database.'"

[Ironically, the pdf for the paper apparently uses some feature of Acrobat which disallows copying text from it. I guess they don't want robots scraping text out of it or something. First time in quite a while I've had to type a quote from the net by hand!]

Secondhand legal advice

[yerricde]

Slashdot is not a replacement for a lawyer.

Slashdot is useful to get a sense of what the legal landscape is like. Some comments are to the effect: "I am not a lawyer, but my lawyer told me this." Or "I am not a lawyer, but here is the statute [cornell.edu], and here is how a court has interpreted it [eff.org]." When you do see an attorney after reading the comments, you don't have to wait for the attorney to explain the basics. This saves time, and time is money, especially at the typical copyright and trade secret specialist's rate.

That said, you're right about one thing: anything you read on Slashdot is not legal advice.

AA.com v FareChase

[FredEFF]

Since you mention that you may be building a screen scraper that gathers airline fares, you may be interested to know that American Airlines has already sued (and won a preliminary injunction against) a software company that built a tool that does much the same thing. The case is American Airlines v. FareChase, and was discussed on LawMeme:

American ... sued FareChase in a Texas court (American is based in Dallas, so that's its home turf) and got a preliminary injunction against FareChase's screen-scraping practices. The court decided that the screen-scraping constituted an "interfer[ence] with American's personal property," also known these days as a trespass to chattels. The court also noted that FareChase's actions might be a criminal violation of Texas Law, which states, "A person commits an offense if the person knowingly accesses a computer, computer network, or computer system without the effective consent of the owner." Tex. Penal Code 33.02(a).

The injunction order is posted on EFF's site, and the briefs are posted on Bag & Baggage.

Other Rule of Thumb to Live By

[Rick.C]

If you fear that it might be the wrong thing to do, it probably is.

Screen scraping is not data mining

[Call Me Black Cloud]

Screen scraping is data gathering. Data mining is looking for trends or patterns in data you already have. Getting the nuggets out of your data to continue with the mining analogy. From this presentation titled "An Introduction to Data Mining Technology" data mining is defined as "The automated extraction of hidden predictive information from (large) databases".

The bottom line is this: when you put this work experience down on your resume don't say you were data mining. Companies looking for that experience will ask you hard questions you don't know the answers to and you will be embarrassed.

Not the pricing -- the timing

[RalphSlate]

There may be precedent for this. eBay was able to convince a judge to bar spidering of their site.

There is another legal concept called "Unfair Competition" which links copyright and facts.

Normally, facts cannot be copyrighted. However, this law seems to kick in when one company compiles and publishes time-sesitive information that it has taken from a direct competitor in a way which "free-rides" on the efforts of the competitor. It is usually applied to news organizations, when one newspaper sends a reporter to Iraq and a second newspaper (perhaps an evening edition) uses the "facts" in the first newspaper's article to publish the very same news.

I could see the instantaneous publishing of all competitors' prices as a violation of this legal theory.

Re:Not the pricing -- the timing

[jkabbe]

Keep in mind that eBay lost their first case against Bidder's Edge's because it was solely a copyright case. Then they brought a conditions of use case that they won.

Re:Yeesh.. give it a rest. (OT)

[Lt Razak]

Are you kidding me? I'd love the internet to "only" be the green screen hooked up to a MUD and IRC that I had back in college.

The corporation will continue to hump the internet like a dog in heat until it becomes as regulated, watered down, and crapped out like the NBC news.

Protecting against scrapers

[nalfeshnee]

Two things that come to mind (depends on the site and what they currently support technically browser- and serverside):

-- use front-end technologies that prohibit or at least inhibit the workings of server-side scraperbots. examples: flash, javascript.

-- use sessions to control how often a given client can access prices, e.g. 'a 10 and you're out' rule: most 'ordinary' users have no need to view a certain page of prices more than X times in a browser session. here, cookies provide even more protection since some scrapers won't be set up to handle them.

both systems may have their drawbacks (no flash allowed), weaknesses (against sessions i can simply make multiple logins), but i've incorporated similar systems on sites for clients with prices that need a sensible level of protection (i.e. one shouldn't be able to grab the whole damn price list with a one-page GET e.g.). guarding against SQL injection is also something which is often forgotten.

Cheers,

Nalfy

Can They Stop Disabled From 'Scraping'

[roboneal]

I'm sure there are a few products that assist disabled persons to "surf" websites by disecting the web page (through essentially screen scraping techniques) and performing one or more the following:

1. Adjusting text size.
2. Dictation of content.
3. Numbering of links.
4. Numerous other alternate presentation of the same data (changing colors for the color blind for example).

An outright ban of automated scraping techniques would eliminate these uses. (While I am at it: What is a web-browser but a form of screen scrapper?).

If the basic technique is allowed, all that can be debated is the use of such data and I think that is a much more dubious area. Facts are public domain.

Maybe they can use the "my bot is blind defense".

here is a similar case

[mcguyver]

Here is a related incident:

http://news.com.com/2110-1017-944258.html

Bargain Network spidered real estate prices on homestore.com/realtor.com and posted them on the bargain.com website. Homestore sued and the case was settled out of court. I wish it was not settled out of court because that would set up a precident.

In my opinion you are asking for the problems. Taking a case like this to court and winning would be difficult. At the very least it would be a serious legal expense.

The last time I checked the rules for Froogle you had to be the actual merchant that ships the product in order to show up in their index. If you are spidering a merchant then you are an affiliate, the products do not originate from you so you would be exluced from Froogle. Froogle does not allow you to sort products by price - so obviously what you plan on doing is different. Froogle also gives merchants the option to be excluded from their index.

My advice is this - get a lawyer because one will surely be contacting you. Familiarize yourself with these phrases: false advertising, breach of contract, and unfair competition.

It was not price mining that the court prohibited!

[theNAM666]

AARRRGGH. I hate ./ sometimes.

From the court summary of the decision:

The court affirmed a preliminary injunction enjoining defendant Zefer Corporation ("Zefer") from utilizing a "scrapper" tool it designed to obtain pricing information from plaintiff's website on the ground that Zefer was doing so to assist defendant Explorica, Inc. ("Explorica"), which was itself enjoined from such activity by virtue of its improper use of confidential information obtained from plaintiff to aid it in gathering this information.

This means that Zefer was prohibited from mining the data because their client (Explorica) was prohibited, because Explorica and Zefer had gained access to the data by exploiting confidential information. Which is another issue entirely from data mining...