Is Data Mining for Product Pricing, Illegal?

wessman asks: "I started to read Orin S. Kerr's 80-page paper looking for how his proposal would pertain to: ripping music/movies, P2P, corporate espionage, and lastly, the use of web scraper robots. Little did I know just how relevant his paper would be in regards to that last item! Kerr makes note of EF Cultural Travel v. Explorica in which Explorica is caught hiring a consultant to program a scraping robot to gather pricing information from a competitor, EF Cultural Travel. Well, I do consulting on the side from home and am currently working a project whereby I gather pricing information from all the major travel conglomerates (Orbitz, Expedia, Lodging.com, WorldRes, Sabre, etc.) so that the travel booking business that hired me can meet or beat all their prices. Granted, the circumstances of the Explorica case are different and the case was an example of an extreme ruling, but my questions to the Slashdot community are: Do I notify the company that hired me of the Explorica case? Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually? Should I continue with this project and the similar projects I do in this area of programming?" Now, add in the text in the "deliverables" section of this press release and it seems we may have some contradictory information. Who is right, and under what circumstances is price harvesting off of the internet not allowed?

Comments

I swear

[cultobill]

In what sane land would PRICES be protected under law? You can't really keep them secret, so "trade secret" is right out. It's not a identifying mark (unless you're a dollar store), so much for trademark. There's nothing useful that hasn't been done before, fuck patenting them. Copyright? It's a simple derivation of what the supplier charges you.

There is nothing creative about pricing stuff. Good lord.

Re:I swear

[Gortbusters.org]

Look at pricewatch.com, it already goes around collecting price data from many online stores.

I don't see why this is such a big problem... one site creates competitive prices based upon other sites' prices. In reality if a consumer reaches your site on the internet for your product they probably didn't do it by accident. They will evaluate all aspects of the business (licensing, service/support, upgrade cost, security of the site, etc) before they jump ship to another site to save a few bucks.

I'm sure Dell, Sony, and HP already keep an eye on eachother's prices.

Re:I swear

[Anonymous Coward]

uhh, the vendors add themselves to pricewatch bub.

Re:I swear

[Gortbusters.org]

Oh.

Re:I swear

[vt0asta]

What about Crazy Eddie! Just giving stuff away at these prices! Walmart "rolls-back" it's prices. K-Mart has blue light specials. I'd say that is getting creative about pricing. Other than *mark'ing the pricings' vehicle of deliver you are right.

I think protecting the actual numbers that correlate to the value of a product is ridiculous.

It seems to me from this article(?), someone is upset that someone is systematically inventoring (screen scraping) their system into a master database and making money off of listing it. Kind of like pricewatch.com but more flamboyant and annoying.

However, if they don't want to be listed then they should have right not to be. No? Companies spend millions of dollars making sure products sold under them are of such a quality that they deserve to be sold under their brand name and in the manner that they see fit.

Big companies may not like some nobody selling stuff along side their name that they worked hard to create and market regardless of their notariety in the publics mind.

They worked the deals, and the people, and the contacts to be able to sell the products they offer. Some screen scraping hack, didn't. Having a customer think big company is in cohoots with someone they deem of lesser quality can be unnerving and unsavory. There is more to this than just the consumer angle.

Re:I swear

[Sparr0]

Pricewatch doesnt mine. Companies PAY for the privelege of listing their prices on pricewatch.

Re:I swear

[whereiswaldo]

Here's a niche for P2P software... obtaining data from a website in a distributed way so as not to stick out in the website logs the way a one or a few download clients would. The collected data could be processed remotely or uploaded to a central server.

Re:I swear

[anonymous loser]

Pricewatch doesnt mine.

No, but froogle.google.com does.

Re:I swear

[Josuah]

In what sane land would PRICES be protected under law? You can't really keep them secret, so "trade secret" is right out.

I suppose you could consider pricing to be a secret depending on how you sold your product or service to people. OEM deals might be a good example, since a vendor works out a certain price structure with one OEM but no other OEMs can know about this, otherwise the vendor loses its bargaining power.

Re:I swear

[KjetilK]

More importantly, the driving force behind capitalism is supposed to be the ability for custumers to compare products, and compare the prizes of products.

The free market is only really free if customers have perfect information. That can never happen, but you can strive to get something that works good enough.

If biznisses are allowed to keep information from the market, it is the end of the free market, and in fact, the end of capitalism.

It is fairly straightforward, really...

Re:I swear

[Fat Casper]

I don't see why this is such a big problem... one site creates competitive prices based upon other sites' prices.

You don't understand the fundamental use of law under the Corporate Administration. Data mining is legal when it involves Admiral Poindexter, your grocery store, your viewing habits or your medical data. Data minig is illegal when it benefits you the consumer at all, much less at the expense of a needy company's profits.

Re:I swear

[Anonymous Coward]

However, if they don't want to be listed then they should have right not to be. No?

robots.txt or .htacces?

Re:I swear

[dasmegabyte]

Actually, there's a lot of creativity that goes into pricing things.

Ever see this: $39.95 ? The .95 cuts .05 cents off the price but greatly increases the chances you'll buy it.

How about 2/6.00? Unless they say otherwise, that means 1's only $3, but again, you're more likely to buy the two. A supermarket by me commonly sells things like limes and small bags of snacks at 4/$1.00, rather than just $.25, which i guess seems cheap.

A lot of the delis around me have sandwiches for 4.63, which with tax is exactly a finski. No numeric change means they can ring you out faster.

Besides this, figuring out what things are worth is a very exact science when you live, as we do, in a vastly service based economy. Prices have nothing to do with the actual cost of physical production, and everything to do with the public's perceived value of the product in question. This is how GM can make two identical trucks, label one GMC, and get another $3000 for it because GMCs are "professional grade and therefore more reliable." This is how Victoria's Secret can get $30+ for maybe a quarter square foot cloth and a pair of wires, and in fact get more money when they use less material or "exotic" materials like satin (which often cost less than cotton). It's all perception and advertising, baby. The luxuries that seperate us from the communists.

Hell, where I used to work, I got paid $15/hr for my time, when the company would charge $125 for it. At one point, they had a massive sliding scale, decreasing the hourly charge based on the amount of work requested and how locked into our system they became, thus guaranteeing future fees. "Partnered" companies got my time for $75, and any company I did work for within our organization estimated me at $50. It was the same freakin' work (though I did really try to wow the full price folks).

Now, if you spidered our rate system, you'd take the $50 and assume we always charged that much. And we'd probably have gone under without the extra $75 to pay for the 85% of the company who weren't directly related to producing software.

Finally: you know, not all vacations are the same. If you pay a cutrate price for travel, you generally get screwed. Cramped flight, rooms filled with cockroaches, free drinks watered down, free meal vouchers for peanut butter sandwiches and generic chips, and a geo metro rental. It is very difficult to compare travel packages objectively because every travel agent has different alliances and partnerships. If you, as a travel agent, offered me a service that would spider all of the other sites and offer me a lower price, it wouldnt' save me any time because I wouldn't trust it. In the hands of the enemy, and all.

Re:I swear

[smack_attack]

We don't live a capitalist society... we live in what's called a "mixed economy".

Essentially it goes something like this: government supports business rights over individual rights; if two business' butt heads, then the government supports whichever one has more "pull" with the government.

Pretty slick market we have in place eh?

Re:I swear

[CTho9305]

WARNING: You can see I write crappy perl / shell script. I make no guarantees about this code.

NOTE: replace all instances of "ABC" with "|".

parse.sh:
#!/bin/bash
#Copyright CTho9305 2003. You are given permission to redistribute this file provided this copyright notice is left intact. You may modify it as you want. Please share any modifications (you are not required to).
#barton
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i xp ABC grep 333 ABC cut -f1,4 -d ' ' > XP333.dat
#XP
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i xp ABC grep -v 333 ABC cut -f1,4 -d ' ' > XP.dat
#Apparently I have too many junk characters
#MP
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i mp ABC perl -pe 's/\.//' ABC perl -pe 's/GHz/00/' ABC cut -f1,4 -d ' ' > MP.dat
# Celeron
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i celeron ABC cut -f 1,3 -d ' ' ABC perl -pe 's/ 1GHz/ 1.0GHz/;s/\.//;s/GHz/00/' > celeron.dat
#P4
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//;s/\s+/ /' ABC grep -i 'pentium 4' ABC egrep -i "sockABC2\.ABC3\." ABC grep -v -i 400MHz ABC perl -pe 's/ 533MHz//;s/GHz/00/;s/ Sock 478//;s/\.//' ABC cut -f1,4 -d ' ' ABC cut -b -8 > pentium4.dat
gnuplot gnuplot.script > ~/www/out.png

Anyway, I wrote this because I was bored and wanted to see what a good price point was for current Athlons. If you examine the graphs carefully you might note that the XP's are not properly differentiated. Some are 333s and marked as that, others aren't marked properly, etc. With the new 400s, it gets worse. For the P4s, I got a little luckier because the speed ranges don't overlap as much. I think I'm going to not differnetiate between the various FSBs of Athlon XPs because the prices are close enough anyway.

Anyway, it has served its purpose by helping me find a point where the processors are reasonably fast, and the bang for the buck is decent.

gnuplot.script

#Copyright CTho9305 2003. You are given permission to redistribute this file provided this copyright notice is left intact. You may modify it as you want. Please share any modifications (you are not required to).set terminal png color
set xlabel "Speed (MHz or rating)"
set ylabel "Cost ($USD)"
set title "Speed vs. Cost"
set grid
set time
set linestyle 1 lw 3
plot "XP.dat" using 2:1 title "XP" with linespoints, \
"XP333.dat" using 2:1 title "XP333" with linespoints, \
"celeron.dat" using 2:1 title "celeron" with linespoints, \
"MP.dat" using 2:1 title "MP" with linespoints, \
"pentium4.dat" using 2:1 title "P4" with linespoints

Anyone know how to change the text font, or the thickness of the lines?

Sample output

Re:I swear

[Sloppy]

Actually, there's a lot of creativity that goes into pricing things.

Ever see this: $39.95 ? The .95 cuts .05 cents off the price but greatly increases the chances you'll buy it.

How about 2/6.00? Unless they say otherwise, that means 1's only $3, but again, you're more likely to buy the two. A supermarket by me commonly sells things like limes and small bags of snacks at 4/$1.00, rather than just $.25, which i guess seems cheap.

A lot of the delis around me have sandwiches for 4.63, which with tax is exactly a finski. No numeric change means they can ring you out faster.

Perhaps the first time someone came up with these techniques, it was an act of creativity. But they are now well-known. When someone prices something as 39.95, they are not being creative; they are merely "using the .95 trick."

Besides this, figuring out what things are worth is a very exact science when you live, as we do, in a vastly service based economy. Prices have nothing to do with the actual cost of physical production, and everything to do with the public's perceived value of the product in question. This is how GM can make two identical trucks, label one GMC, and get another $3000 for it because GMCs are "professional grade and therefore more reliable." This is how Victoria's Secret can get $30+ for maybe a quarter square foot cloth and a pair of wires, and in fact get more money when they use less material or "exotic" materials like satin (which often cost less than cotton).

Ok, these are much better examples of creativity. The person doing the pricing is actually doing work and making real decisions.

But does that entitle their work to special protection? Let's go back and look at why we have "IP laws" at all, lest we forget the purpose and perversely misapply the law. In the United States, the Constitution empowers congress on this basis:

To promote the progress of science and the useful arts by securing for a limited time to authors and inventors the exclusive right to their respective rights and discoveries...

It is effort for you to write a book, and if you didn't have a monopoly on the sales of that book, your effort would be wasted from a commercial perspective and you might not choose to expend that effort. So we give you a copyright. You get what you want -- compensation for your effort -- and we get what we want -- promotion of the useful arts. Perhaps that promotion is somewhat delayed (until after the copyright expires) but we'll eventually get it.

Now let's look at your sophisticated pricing example, in that light. You expend effort in figuring out just the right price for your widget, and you would like a monopoly on the fruits of that work. If we grant that to you, then you get what you want -- compensation for your effort -- and we get ... hey, what do we get? Does the creative act of figuring out that $30 is the most profitable price for your underwear, help us in some way? Does it promote the progress of the sciences or useful arts? I don't see how.

I see that there's something in it for me, to grant copyright on creative works such as books. I don't think there's anything in it for me at all, to grant copyright on prices, even prices that take effort to optimize.

And without a monopoly, you won't have sufficient incentive to do the work of determining the most profitable price for your satin underwear or truck? As if!

Re:I swear

[Anonymous Coward]

If you walk into a brick and mortar and start writing down prices, the store management and security will escort you out. Guaranteed.

Re:I swear

[jcronen]

The free market is only really free if customers have perfect information. That can never happen, but you can strive to get something that works good enough.

Close... a free market is one where people are free to trade with whomever they want, giving preference to none.

An efficient market is one that is predicated on perfect information. Companies don't have incentive to work in an efficient market -- assuming that what you have for sale is just that, your product*, there's nothing to differentiate one company from another.

Therefore, all that hard-spent advertising revenue is worthless, all that market share and word-of-mouth is useless. Someone will go to bogus-retailer.com rather than a real retailer to get the same item for 3 cents less.

*In reality, no two products are ever really identical, due to the service that comes with a purchase, warranty, shipping speed, customer experience, etc. But as big companies become more "parts vendors" than "solution providers", this becomes moot as time goes on.

Re:I swear

[KjetilK]

Close... a free market is one where people are free to trade with whomever they want, giving preference to none.

You're right, I should have said "a free market is only efficient if".

Companies don't have incentive to work in an efficient market -- assuming that what you have for sale is just that, your product*, there's nothing to differentiate one company from another.

Sure there is, if you're product is better suited to meet the needs of your customers than the other company's product, you'll win.

Someone will go to bogus-retailer.com rather than a real retailer to get the same item for 3 cents less.

So? If bogus-retailer.com's costs are 0.03 less per shipped item, everything else equal, then who cares if bogus-retailer.com is not sanctioned by Big Manufacturer(R)? Retailers should also cut costs in an efficient market.

I'd love to see a world where all that marketing is worthless, and rather, I'll go to a database where I can read objective information, third-party reviews, and stuff like that.

Basically, I think that free market can be extremely efficient in many situations, but not always, particulary, the free market is not very good at staying free.

Re:I swear

[TheMadReaper]

I thought that capitalism was supposed to work best under the assumption of perfect information (in particular price information). Wouldn't making price collecting illegal be un-capitalistic?

Re:I swear

[Anonymous Coward]

In an efficient market, companies with specialized competitive advantages can always take the difference between their production cost and their competitors' as profit. In fact the point of having a market is to allocate resources to the most efficient producers of needed goods, and anyone that interferes with that is stealing resources from our civilization that could have been better used. Ad revenue should be worthless--if a company has no obvious built-in differentiator it's a complete waste of bricks and mortar.

Re:I swear

[Pharmboy]

Tobacco lawsuits. Oreo lawsuit (dropped today). McDonnalds Hot Coffee lawsuit. Fast food makes people fat lawsuit. etc etc etc.

are you SURE businesses have more rights than individuals? I would disagree.

Re:I swear

[Pharmboy]

We sell two products that are the exact same item, except we photographed one item with an "optional" part missing, even though we ship it with it. One is priced $1599, the other is priced $1899. The one for $1599 has an uglier picture, and is described simpler. It just has a different label on it. We sell about 3x more of the item at $1899. The idea is to average 1799 for each one sold.

We use the 1599 to get more people to the site, then promote the 1899 unit more. All descriptions are 100% accurate, we just make the 1899 unit look better, and point out more benefits. This was my idea. Sad thing, I haven't gotten a bonus for it yet.

Re:I swear

[smack_attack]

It's a wildly swinging pendulum.

Re:I swear

[Pharmboy]

It's a wildly swinging pendulum.

Now THAT I would agree with. My point is that on average, taken over a longer period of time, its about even. And should be.

Re:I swear

[smack_attack]

You're forgetting who wins in the long term when government acts not as a mediator but as an instigator. The government officials who get the kickbacks and soft money from lobbyists who press harder to pass more restricting laws.

The ultimate course of action is for the pendulum to stop because the hands that allowed it to swing are now choking it.

Re:I swear

[mrmeval]

Note: Not legal advice, not leegull advice, not even advice but a converted, transmitted, reassebled representation of not legal advice. Call an amulance chaser for more clarification.

First Circuit holds that prohibitions found in a website's Terms of Use can be used to establish that a visitor to that site exceeded his authorized use thereof for the purposes of establishing a violation of the Computer Fraud and Abuse Act ("CFAA").

You can be charged for the crime of illegal access if you fail to follow a TOS.

While prohibitions on authorized usage can also be established by reliance on such things as password protected access, the First Circuit rejected the notion that courts should look to the "reasonable expectations" of the parties to determine if a particular usage was in fact authorized.

The door can be unlocked and you can enter, but a you must follow a posted sign that states what you can do.

The court affirmed a preliminary injunction enjoining defendant Zefer Corporation ("Zefer") from utilizing a "scrapper" tool it designed to obtain pricing information from plaintiff's website on the ground that Zefer was doing so to assist defendant Explorica, Inc. ("Explorica"), which was itself enjoined from such activity by virtue of its improper use of confidential information obtained from plaintiff to aid it in gathering this information.

I need my legal deobfuscator for this, my brain just fried.

Just like Popeye

[Anonymous Coward]

Anybody remember the Popeye cartoon where Popeye opens a car wash, and then Bluto opens a car wash right across the street?

When Popeye posts his price, Bluto beats it by five cents. Then Popeye beats Bluto's by five cents.

It goes back and forth until Popeye is washing cars for free.

I was tempted to call this an infinite loop, but I doubt a retailer would pay you to take their products.

Re:Just like Popeye

[Gortbusters.org]

Unless you're MS and you want to keep Linux out of a large organization! MWahahaha

Re:Just like Popeye

[87C751]

I was tempted to call this an infinite loop, but I doubt a retailer would pay you to take their products.

Last month, I got a $40 rebate on a $39.95 router.

Re:Just like Popeye

[Sparr0]

Heh, youre a rebate newbie. Peachtree Accounting. Costs $200. You get $40 off the purchase price at the store with a coupon. And there is a $160 rebate. And there is another $30 rebate if you own a previous version, or competitors software. This same deal exists every year, I have been "upgrading" for the last 3 years with it. And this is only a mediocre deal. Deal sites regularly list deals where you spend $20 and get $40 back.

Re:Just like Popeye

[tupps]

How much was postage?

Have you actually received your rebate?

Re:Just like Popeye

[Anonymous Coward]

Not only Popeye, but I Love Lucy as well (when they set up a diner).

As far as a larger refund than the original purchase price, the TI99/4a could be purchased for $49.50 at KMart and had a $50 rebate (during its[1] waning days).

[1] dweebs who think "it's" is possessive can pretend it's spelled that way up there.

Re:Just like Popeye

[io333]

I was tempted to call this an infinite loop, but I doubt a retailer would pay you to take their products

I take it you don't hang out over at FatWallet?

Re:Just like Popeye

[87C751]

I "borrowed" a stamp. Had I not, the total out-of-pocket cost would have been $0.32, which is still a good deal. Surely anything under $1 is in the noise. But no, it hasn't arrived yet, so I may have been retroactively ripped off.

Re:Just like Popeye

[Ben Hutchings]

You don't think rebates are for real, do you? Bet you paid sales tax on the $39.95, too, and you're not getting that back (though neither is the company that sold it).

Re:Just like Popeye

[tupps]

Would you let me 'borrow' some money :-) But if you 'borrowed' it from work that doesn't count. If you where at work I hope you sent it express/first class ;-)

Re:let me ask you this.

[slamb]

An Anonymous Coward asked: Is the comma near the end of this question, necessary?

Of course it is. Let's dissect this sentence:

Is Data Mining for Product Pricing, Illegal?

• Subject: Data (from Star Trek, I suppose)
• Verb: Mining
• Object: Product pricing
• Listener: Illegal

Not having the comma would completely distort the meaning of the sentence.

TITLE HAS POOR GRAMMAR

[Anonymous Coward]

EDITORS: A better way to phrase that headline is as follows:

Is It Illegal to Data-Mine for Product Pricing?

Thank you.

Re:TITLE HAS POOR GRAMMAR

[Anomalous Cowturd]

You're welcome.

Another way to put it would be "Is Data Mining for Product Pricing Illegal?".
Or even "Is it Legal To Data-Mine Pricing Info?"

Re:TITLE HAS POOR GRAMMAR

[Anonymous Coward]

It should read:
The grammer of the title is poor.

Re:TITLE HAS POOR GRAMMAR

[hazem]

Or if you're really married to using that comma:

"Data Mining for Product Pricing, Is It Illegal?"

Commas just don't get used enough. Maybe he just wants to get more even wear on his keyboard. I'm sur we can expect more tildes soon!

Re:TITLE HAS POOR GRAMMAR

[dipipanone]

Commas just don't get used enough.

Perhaps not, and I don't wanna get all grammar -nazi prescriptive here, but if I was writing that strapline, I would have used a colon.

Data Mining for Product Pricing: Is It Illegal?

Re:TITLE HAS POOR GRAMMAR

[Potor]

There is nothing wrong with the grammar of the title; your complaint concerns rather a question of style. The writer made his sentence more memorable by fracturing normal syntax, IMO.

Re:TITLE HAS POOR GRAMMAR

[Anonymous Coward]

And this should read:
The spelling of the parent is poor.

Re:TITLE HAS POOR GRAMMAR

[Anomalous Cowturd]

Like the previous poster said, you would more properly use a colon. The better way to write the original subject would be

"Is Data Mining, For the Purposes of Product Pricing, Illegal?"

which does not make for a snappy headline. I believe

"Is Data Mining, For Product Pricing, Illegal?"

would also be correct.

I'd like to see more semicolons, myself, but nobody really knows how to use them.

Re:TITLE HAS POOR GRAMMAR

[radish]

That comma is in an incorrect place - the sentence has no meaning in correct english. Take the comma out and it's correct, if unexciting. The suggestions to use a colon or rearrange it to use the comma elsewhere make the strapline more interesting, but as it stands it's wrong, plain and simple.

Re:TITLE HAS POOR GRAMMAR

[Anonymous Coward]

Neither of these would actually be correct: it implies that the article is about the legality of data mining, rather than data mining for product placement.

There is a difference between saying "Is Data Mining, For Product Pricing, Illegal" and "Is Data Mining for Product Pricing Illegal." The first is generally about data mining, while the second is more specifically about data mining for product pricing.

Re:TITLE HAS POOR GRAMMAR

[Kakurenbo Shogun]

First of all, who moderated the original comment up in the first place? Give me a break.

Second, the post I'm replying to is correct--"Is Data Mining for Product Pricing Illegal"--no commas--is the correct grammer. (I assume comeone's going to jump on my use of hyphens--fortunately I don't care). "Is it illegal to..." is poor grammer, because you're using a pronoun, "it", before you've mentioned what it refers to. Pronouns should always refer back to something.

On the other hand, who really cares? As long as the point is communicated, little else matters in a situation like this. Were we discussing someone's resume, or sending a business letter, that would matter, because grammer mistakes suggest a lack of professionalism and attention to detail.

Re:TITLE HAS POOR GRAMMAR

[tmasman]

Hi!
You must be new to slashdot...
Welcome!

~ tmasman

Re:TITLE HAS POOR GRAMMAR

[FurryFeet]

...if I was writing that strapline, I would have used a colon.

They probably did. That's why the strapline is shite.

Re:TITLE HAS POOR GRAMMAR

[jafiwam]

I'd like to see more semicolons, myself, but nobody really knows how to use them.

Shhhh!! Don't tell! That's exactly why semicolons are good; nobody knows how to use them and yet they allow stilted and incomplete thoughts to be written down; and still look sophisitcated.

Re:TITLE HAS POOR GRAMMAR

[Anonymous Coward]

Your grammar is fine, but your spelling is atrocious. There is no "E" in grammar. Thing is, it DOES matter how the point is communicated. If I ran up to a bar counter, slammed it with my fist, and bellowed "ME WANT ALCOHOL", I'd look like an idiot, even though it's rather obvious that I'm trying to get a drink. Now, just imagine that on a smaller scale...

Re:TITLE HAS POOR GRAMMAR

[Potor]

Once again I see a complaint about style and not grammar. It is not a grammar error to separate parts of one sentence with a comma. Un-natural perhaps, but therein lies the emphasis; the only people he has to fear are the style police. The comma plays no role in grammar, in this instance. Put differently, it is not a logical operater here, but merely a device to slow the reader down. That makes it, rhetorical.

The suggestions to use a colon or rearrange it to use the comma elsewhere make the strapline more interesting, but as it stands it's wrong, plain and simple

perhaps you could parse that first clause for me, btw. cheers.

Re:TITLE HAS POOR GRAMMAR

[Potor]

ignore the second part of my comment; i found the suggestions, and understand what you are saying. cheers.

Re:TITLE HAS POOR GRAMMAR

[radish]

I refer you to some people who know a lot more about English than either of us. Punctuation is part of the language. Like other areas of the language it is open to artistic and stylistic variation, but it can also be just plain wrong. I don't pretend to be a language expert (although both my parents are) - but that headline is wrong, I could tell it was, and the lined document agrees with me.

That makes it, rhetorical.

That sentence is wrongly punctuated. Or maybe you can tell me which of the 4 uses of the comma it fits into? It is not seperating items of a list, it is not part of a pair of commas distinguishing items in a list, it is not joining two complete sentences and it is not indicating an absent repeated section of the sentence.

Ummm..

[deadgoon42]

My grandmother calls this "shopping around." The only difference is that someone else is doing all the work.

Re:Ummm..

[Codex The Sloth]

My grandmother calls this "shopping around."

Damn! I was going to patent this but it sounds like there is some "prior art"...

Re:Ummm..

[tuffy]

Damn! I was going to patent this but it sounds like there is some "prior art"...

That's not much of an obstacle lately. I'm planning to patent a process known as "respiration" that involves oxygen and human lungs. You can all expect letters from my attorney shortly...

Re:Ummm..

[GQuon]

My grandmother calls this "shopping around."

Damn! I was going to patent this but it sounds like there is some "prior art"...

Try "shopping around on the Internet"!

Or "shopping around on a network of interconnected artificial artifacts" would be more like it. You know, for when they make patent terms infinite. (I.e. extend them retro-actively whenever they are bout to expire.)

Re:let me ask you this.

[Anonymous Coward]

I agree. It makes the delivery somewhat, stilted.

Price Scraping?

[Anonymous Coward]

Actually, price scraping was done in a very low-tech way a reasonably long time ago by a pretty well-known businessman: Mr. Sam Wal-Mart. Early in his career, he would dumpster-dive his competitors to find out the prices his competitors paid for their goods, the contracts they had with their suppliers, etc. This provided him with "insider information" so he'd know how to prepare his pricing in a more strategic fashion and obviously out-compete them where it counts: financially.

Re:Price Scraping?

[Anonymous Coward]

Wrong. Read Sam Walton: Made in America: My Story. Sam Walton says that was a story put out by his competitors to disparage his name and he never did anything of the sort.

Re:Price Scraping?

[chunkwhite86]

well-known businessman: Mr. Sam Wal-Mart.

FYI His last name is Walton. And only rednecks shop there anyways.

Re:Price Scraping?

[MrPengy]

FYI His last name is Walton. And only rednecks shop there anyways.

At sales of over $240 BILLION (http://www.fortune.com/fortune/mostadmired/articl es/0,15114,423053,00.html) I'm guessing that more than just "rednecks" shop there. Besides, his last name WAS Walton ... he's dead, Jim!

If all you want to do is browse the web and read email get a Mac. If all you want is for other people to browse your computer and read your email get Windows. For everything else get Linux.

Re:Price Scraping?

[Anonymous Coward]

I think the guy was trying to be cute by saying Mr Wal-Mart. It's pretty clear Wal-Mart is a combination of Walton + Mart. And if he was a dumpster diver, would you expect him to tell the truth in his book? It's kind of like someone claiming they never committed an act they've been accused of. They'd say "no" whether they did it or not. If we asked you if you've stopped fucking your grandmother, what would you say?

Re:Price Scraping?

[Anonymous Coward]

Yes, because god knows self-biographies always tell the truth.

Re:Price Scraping?

[gpinzone]

Yeah and the fact that he probably tried to glom off the name "KMart" since they were around before Walmart.

Re:Price Scraping?

[BernardMarx]

You're kidding, right? I worked for WalMart for a year in high school, and the managers would actually tell us to go into neighboring ShopKo's and KMart's and record prices so that theirs would be lower. Sometimes they'd even give us hand-held scanners to make it even easier. Granted, we didn't have to do this, and we wouldn't get paid for it, but that is beside the point. If you ever worked in a WalMart, you'd know this.

TOS and more ideal markets

[weston]

I think any large business where the pricing structure isn't directly related to costs is probably deeply afraid of agents that aggregate their data with competitors. You end up with a more ideal market, a more frictionless market, if you will, and they'll be forced to compete on narrower and narrower margins of profit. Of course they'll want to throw up barriers to that.

But I'll bet this issue comes down to Terms of Service and what a company can reasonably expect to be able to legally require/forbid about the use of data provided via an automated means...

You have a contract?

[gpinzone]

As long as you get paid, let them worry about the lawsuit. They're the ones who are going to actually use it. Keep your mouth shut.

Re:You have a contract?

[vandan]

Would you apply this line of reasoning to ALL areas?
What if the job were researching Bush's all-feared biological weapons?
Or GM products?

The problem is that if everybody decides to look the other way (and everyone can find a reason why they should take their money and shut up), then some pretty fucked up things get done, and people are left wondering, "How did it come to this?"

Now I'm not saying that the world is going to end because someone's harvesting prices off the net. I'm just questioning your "Me first, no-one else 2nd" argument.

Re:You have a contract?

[gpinzone]

IT's an ethical and legal issue, but not one that is life threatening. Goodwin's law ought to be ammeded to include types like you.

Re:You have a contract?

[stiller]

I do hope that every /. reader at least can discriminate between building life-threatening devices and products that may or may not just fall under some crazy copyright law. It's the user's responsibility not his. They assigned him, they carry the consequences.
It's your way of extreme reasoning that gets us nowhere. We must be reasonable, and it's reasonable to expect that no judge in his right mind would convict anyone for this.

Re:You have a contract?

[Larsing]

Short of resigning and devoting the rest of his/her life to working for the red cross, exactly how would this poor sod growing an over-sized conscience help the rest of the world?

Re:You have a contract?

[slackr]

In most cases this is absolutely correct *unless* your expert advice is part of the contract. If you are a coder, go ahead and code and let their legal department handle their end of the issue. After all, you don't want to stifle their innovation by throwing up walls that they might well be able to surmount.
However, if you are considered an expert in this particular type of application development, then you are obligated to provide your expert advice. That does not mean that your advice should be: I don't think we can do this. Instead, you simply say: Your laywer(s) may need to develop a strategy to be ready in case this problem arises.
The fact that you've gone to Slashdot with this question proves that you are not the person to defiitively say one way or the other (and that's a Good Thing in sloppy issues like these).

Re:You have a contract?

[swillden]

As long as you get paid, let them worry about the lawsuit. They're the ones who are going to actually use it. Keep your mouth shut.

I completely disagree. Turn the tables around: suppose you were the manager making the decision to pay a sub to build this tool, not realizing you may get blindsided with a court case. Would you like to know?

Telling them about this particular issue might very well mean they decide not to go ahead with this work. However, telling them will also make clear that you are an individual with integrity and one who puts his clients' interests above his own. It will also point out that your value to them goes beyond just cutting code.

Short-term, keeping your mouth shut might be the best thing to do, but long-term you'll be better off if you treat your customers the way you would like to be treated.

However, there are different ways to tell them, and some ways provide you with more opportunity to keep working, and billing, than others. Telling them "You can't do this, it's illegal, we'd better stop everything until you get an attorney to analyze this court case" is the wrong thing to do. It puts the worst possible face on the situation and recommends that they stop paying you.

On the other hand, telling them "There may be a legal question here. I recommend that you have your attornies examine this case and decide what steps you can take to avoid the problems that company had. Meanwhile, here's my initial thoughts on the design...". Thus, you get to be the hardworking, honest and conscientious contractor who's doing his best to both push the project forward and to assist them in avoiding any future problems.

Oh, and you'll sleep better.

Re:You have a contract?

[mvdw]

it's reasonable to expect that no judge in his right mind would convict anyone for this.

Maybe so, but the question wasn't whether it was illegal, but whether he should tell his contractee(?) of the possible precedent. I think he should let them know, for a number of reasons, some selfish, some not so.

Re:You have a contract?

[vietbob]

This is irresponsible. The company that hired you did so in the belief that you would work in their best interests. Since you are aware of this case, you should bring it to their attention. Worst case they terminate your contract? You still have an excellent reference for future jobs. If you have pertinent knowledge that you choose not to reveal to them, you will be putting your reputation on the line.

Rules of Thumb to Live By

[release7]

I'm not going to pretend to know what the laws are in this case. It seems that the only law these days is "He who can afford the best, most impressive lawyer wins." Here's some other cynical words of wisdom I've come to believe:

If powerful people get screwed, it's illegal.
If it forces large corporations have to work harder to earn a profit, it's illegal.
If it give the little guy a leg up or levels the playing field in any way, it's illegal.
If it's illegal and you're big and powerful, don't worry about it, you can probably get away with it with little damage to your business or career and keep almost all of you cash minus legal fees.

Re:Rules of Thumb to Live By

[urbanRealist]

It seem like you're right. In no way shape or form did anyone break either the spirit or the letter of the law here. Yet,

First Circuit holds that prohibitions found in a website's Terms of Use can be used to establish that a visitor to that site exceeded his authorized use thereof for the purposes of establishing a violation of the Computer Fraud and Abuse Act

Which says to me I can say, "oh, you can't read my website while wearing sunglasses." I could be wrong here, but according to this ruling, I can just put up any restriction I can dream up and you're bound by it when reading my website. That ruling truly must have cost a fortune. The other down side is that data mining is probably one of the most useful developments in recent times. Look at Google News.

Re:Rules of Thumb to Live By

[poot_rootbeer]

It seems that the only law these days is "He who can afford the best, most impressive lawyer wins."

I can see how someone could be fooled into believing this, if the only news they get about what's happening in the courts is the fearmongering they read in "Your Rights Online."

I suggest you obtain a copy of the verdict records from a court that deals with a wide variety of cases. Ten bucks says you'll come to the conclusion that regardless of who the legal counsel is on both sides, justice is truly served far more often than not.

Re:Rules of Thumb to Live By

[thirdrock]

I suggest you obtain a copy of the verdict records from a court that deals with a wide variety of cases. Ten bucks says you'll come to the conclusion that regardless of who the legal counsel is on both sides, justice is truly served far more often than not.

I have done this (spent time at the local courthouse), and I disagree, but not in the "the rich get away with anything because of the quality of their legal representation" way that the previous poster claimed.

No, it's more along the line of, at a given time in history, twelve people (or one judge) tend to have a set of fears, desires, loves and hates that tend them towards making personal decisions regardless of the law. This is what some people call "natural justice", where judicial decisions are made on the basis of people's feelings. Something that Janet Reno used with great success in Florida to railroad people into jail on child-sex-abuse charges, despite the fact that she faced counsel of greater competence than her, and in many cases there was no evidence or even likelihood that most of those people were even guilty.

So while legal justice is very rarely served in a court (with the exception of Judge Judy of course), natural justice tends shaft rich and poor alike on a regular basis.

huh?

[Anonymous Coward]

Who is this "Illegal" person and why are we asking him questions about Star Trek characters?

I'm not "Illegal," but I'll answer. Seeing as how he was killed off in the last movie, I think it's safe to say that no, Data is not mining for product pricing.

(In other words, you illiterate clods need to be more careful with your commas.)

Re:huh?

[Anonymous Coward]

What, do you mean?

well

[Vej]

The Albertson's I walked into today said No video taping or cameras on "premises" without prior permission.

This is obviously their perogative.

Similiarily, wouldn't it be up to the site in question to at least post up-front rules to the conduct on their site? Such 'adult-material' warnings seem a comparible attribute to a site.

And in that case, would they be enforcable or even proveable?

Re:well

[Anonymous Coward]

here we go again, another moronic techno-dweeb script kiddie who can't spell.

It's prerogative.

Re:well

[Vej]

Good job, you missed a few.

What's the point of Anonymous Coward post ability?

well....duh

[mrpuffypants]

Look, I I can visit a web site and the business (Let's say Amazon) publicly posts their prices for anybody to see then you sure as hell can use them! If suddently using bots to do work are illegal then I'd wadger that every shell script that I write is an affront to US Laws. Rotating log files and all sorts of other "make my job easier so that I can play Quake" scripts are perfectly legal, so how the hell can it be questionable just to go to a site and record prices???

Jebus, please help the Unites States Gub'ment!

Re:well....duh

[Some Dumbass...]

so how the hell can it be questionable just to go to a site and record prices???

Let's not forget that this was a legal ruling, and some of judges don't know squat about technology. The lawyer for the defense probably showed the judge a web page with prices on it and the judge assumed it was "hacking" or somesuch.

Re:well....duh

[mrpuffypants]

good point...we truly do need a lot more lawyers who are technically adept so that the future and present intellectual-property and tech-related cases can be argued by competent attorneys.

EFF

Re:well....duh

[Jeremy Erwin]

Amazon says you can't.

Amazon.com grants you a limited license to access and make personal use of this site and not to download (other than page caching) or modify it, or any portion of it, except with express written consent of Amazon.com. This license does not include any resale or commercial use of this site or its contents; any collection and use of any product listings, descriptions, or prices; any derivative use of this site or its contents; any downloading or copying of account information for the benefit of another merchant; or any use of data mining, robots, or similar data gathering and extraction tools. This site or any portion of this site may not be reproduced, duplicated, copied, sold, resold, visited, or otherwise exploited for any commercial purpose without express written consent of Amazon.com.

I am guessing that the prohibition on "visit[ing] for any commercial purpose" precludes me from actually purchasing their wares.

Re:well....duh

[Anonymous Coward]

According to that, you can't even save a copy of a page or an order.

Re:well....duh

[Duckling]

But at least they have made an explicit statement regarding this, and can use that to argue a case
against someone mining or scraping their site.

Did the company in the original article?

Re:well....duh

[Marcus Brody]

"or any use of data mining, robots, or similar data gathering and extraction tools"

So how come most of amazon is cached in google (and elsewhere)?

Do I bypass amazon's stupidly fucking ludicrous & legally dubious legalese if I say, scrape the information from google's cache rather than from the amazon website?

Re:well....duh

[lefthand50]

According to this logic, amazon should bring suit against google. Check out froogle.google.com -

Re:well....duh

[Zirnike]

Luckaly, they published their page. I'm not bound by their terms of service unless I sign it. The web is a library. A huge collection of 'books' to browse around in. Just because one of the books says 'if you read this, you must yell at the top of your lungs 'Drugs are bad!'' doesn't mean I have to do it. Amazon's 'terms of service' MIGHT be considered to apply at the moment you make a purchace, as that's part of a sales agreement. But just looking? No way.

This is a LOT more clear-cut than shrink-wrap, and that's been shot down a number of times.

Re:well....duh

[wierdling]

Unfortunately, due to hazy and unclear laws on what is considered to be unauthorized access in regards to computers and all things computer related, it is very likely that this technology (screen scrapers) can be considered "hacking" and the person/company who does it can be prosecuted. Is what we need is a much better definition of what is considered unauthorized access (if you spend 23 hours trying to guess my username and password and do it, that is unauthorized in my opinion, if I put a click through on my site that says only people who love science fiction can access the site, yet you hate science fiction but are doing a report on it so you click through to see what I have to say,that is not).

Re:well....duh

[daviddennis]

Oddly enough, they also supply amazon.com web services, which is legal and makes most of this stuff dead easy.

You'd be nuts to screen scrape from Amazon. I know because I tried it, and breathed a BIG sigh of relief when I found out how to do what I needed not only legally, but easily too.

D

Re:well....duh

[Anonymous Coward]

Amazon says you can't.

Then again, Amazon can say whatever they want. They are more than welcome to put in their license a statement that says that by visiting their site you are willing to give up your first-born son to them.

Of course that doesn't mean it'll hold up in court.

Re:well....duh

[zentigger]

I'm pretty sure GWB is a 'tard!

Few points.

[mindstrm]

First, the obvious: IF you are concerned about YOUR obligations towards your clients, talk to a lawyer competent in the areas of law this involves.

Second.. a cursory reading of the case you linked says that case was not about scraping in general, but about a consultant doing scraping on behalf of a client who was not permitted lawful access to the site/service being scraped by way of a terms of use.

In other words: one party was indirectly accessing the site by way of a second party... it was determined that in the particular instance of this case, their action was not permitted.

Do I notify the company that hired me...

[djupedal]

I think they'll get a clue RSN.

Actually, sometimes they do.

[Anonymous Coward]

Two specific cases in point.

1) At many of the deal sites (i.e. slickdeals.net, etc) once in a while this offer appears where after getting back your rebate, you have more money than you spent for the product.

2) Grocery coupons - in some cases, a store will run one of those "triple coupon Thurdays" promotions, and if you have the right coupon, the money-off total will exceed the price of the product. Depending on the the store, money is returned, or a credit is.

web servers are not protected

[Provincialist]

accesses a protected computer without authorization, or exceeds authorized access [18 U.S.C. 1030(a)(4) of the CFAA]

How does one receive authorization to access a web server? Hmm, maybe with a simple html GET? The basic fact here is that of judicial cluelessness. If I put information on a public web server, pretend to "protect" it with a disclaimer (of everything) at the bottom of the page, and then get pissed off because somebody browsed that information, I'm an idiot. In addition, I am legless in court. Web servers make information available to the world. If I had wanted to make information available to certain parties that I trust not to compete with me, I should have set up a secure server with some provision for authentication and authorization.

It really is that simple

later,
Jess

Easy fix.

[Photar]

The problem is that when you sell a commodity like a TV or a vcr and the only difference between them is price you can't exactly maintain a high profit margin. What they need to do is obfuscate the prices so that its next to impossible to compair products. Thats how cell phones work.

This phone has 500 any time minutes for 3 cents a minute from your calling area roaming is 10 cents a minute, unlimited text messaging, 800 night and weekend minutes is free for the first 6 months and has rollover.

This other phone has 1000 minutes for 2 cents a minute but with out rollover and text messaging costs 1 cent per message, night and weekends are free but don't start till 9pm.

See its not exactly easy to just look at the plans and see which one is the better plan.

Re:Easy fix.

[HBI]

Appliance companies do this with unique product numbers, if they are large enough.

"We'll meet or beat your best price" doesn't mean much if the model isnt' sold elsewhere.

The bottom line is that pricing of commodity items is cutthroat, evidence all the audio/video joints that have gone out of business/bankrupt in the New York area over the past 20 years.

Retail being unprofitable is just another way the fat is being cut out of the economy. It isn't a bad thing for the consumer. Unless you value service and full employment, even for the kind of dweeb that sells A/V equipment or appliances for a living.

Re:Easy fix.

[canajin56]

Scott Adams refers to this as a "confusopoly" Telephone companies and airlines use them. Since you can never tell who has the best price, they can all remain in business.

Re:Easy fix.

[Anonymous Coward]

Hmm when I think about it... If you really want to keep your information away from data-mining but still be able to show information to the surfers, Macromedia Flash becomes the right tool for the job. Sure it's not search engine friendly.. but that's the point.

Re:Magic Eye Poster!

[Anonymous Coward]

why, you little! gakk, gakk, gakk (bart choking)

Best buy is a really really bad example.

[MegaHamsterX]

I had a friend pricing out some equipment in Best Buy as well as all the other chains, when he pulled out his pda and started writing down prices for the things he was going to buy, store security rushed him and escorted him to the door.

Re:Best buy is a really really bad example.

[MegaHamsterX]

Oh, I forgot to add something, if you've ever shopped on the best buy site you know that quite a few of their prices can only be accessed by adding the item to your shopping cart, the price is displayed as a strikeout.

Re:Best buy is a really really bad example.

[BuckaBooBob]

Would be interesting to see what would have happened if he would have refused... Or better yet... talk to the press and see if you can make the news for carring/using a PDA in bestbuy :)

Re:Best buy is a really really bad example.

[Anonymous Coward]

I used to work for Best Buy and would "shop" the competition in my area (Circuit City) on a regular basis. This was in 1995, so PDAs weren't really an option, and pencil and paper were right out. I had to wander around their tiny (at that time) computer section, memorize a few models and prices, and then wander to another less-busy department to mutter the information into a microcassette recorder in my jacket pocket. After a few months they got wise to me and at one point (around Christmas) an employee walked up and made cracks about what I was doing and the fact that they had stock of several popular items we were out of. No one ever threw me out, though, since it was understood that we would then bar their employees from shopping our store. As someone pointed out in another post, we often had unique model numbers for computers anyway, so I was mostly shopping printers and other items that we had in common.

Re:Best buy is a really really bad example.

[mcheu]

Last I checked, if you're asked by the store to leave and you refuse, you're trespassing. You can be arrested and charged by police for that (at least in the US and Canada). You can always bring your story to the press, but unless it's a slow news day or you can tie some other issue that's in vogue, it won't make the news. Eg. the Peacenik T-Shirt guy that got busted for wearing a peacenik T-shirt in the US at a mall where he bought it. Some papers chose to omit this part, but the guy didn't get busted for wearing the shirt. He got busted for not leaving when mall security asked him to leave.

Easy answer

[Lurgen]

Once their prices hit the Internet, they're in the public domain. It would be like posting your prices in the window, and complaining that a car driving past could photograph them.

We all know that bots crawl the web - Google, Altavista, spam-bots... they're all common knowledge. You put information on a website, and it's going to be viewed by an automated process. Surely with that knowledge, it's ridiculous to think you can ban people for using the information you've posted publicly in whatever way they desire.

Perhaps these companies (airlines, computer stores, whatever) need to start offering their services at the price they really mean to sell it for, rather than this stupid haggling they expect from us. Or maybe it's time they focused on quality of service, value-add, etc rather than price wars (which never help anybody in the long term).

Bottom line? If you don't want your competitors seeing your prices, don't make them available to them - this means no junkmail, no spam, no website, no prices in the store window, no prices inside the store, nothing.

Re:Easy answer

[mgrassi99]

If sites are so worried about scraping, etc, they should just post their prices as gifs. Done and done.

Re:Easy answer

[gpinzone]

Didn't ebay sue auction spiders becuase they used eBay's resources? How the heck are price scrapers any different?

Free Information or not.

[BuckaBooBob]

If information is placed on a web site without some method to secure that information being used other than its intened purpose then It should not be data mined... But if that information is in the open with no steps taken to secure that information then it should be ok to datamine.

There's been several lawsuits on this.

[LOTHAR, of the Hill]

All variations ona theme. they claim pricing info is copyrighted info.

http://twincities.bizjournals.com/twincities/sto ri es/2002/12/16/story5.html

Examples...

[oaf357]

Buy.com was built around a huge server farm that scoured the web and found the best prices for products it sells and then beat those prices (to the best of its ability). That has changed a little now but buy.com was built from that.

Also, Pricewatch, Pricegrabber and Froogle scour the web for prices and create search engines out of them so consumers can find the best price.

I'm not saying just because everyone else is doing it means you can too (and you might have a slightly different objective causing these examples to be weightless) but it's being done all over the place.

Hope that helps.

Re:Examples...

[Anonymous Custard]

Buy.com was built around a huge server farm that scoured the web and found the best prices for products it sells and then beat those prices (to the best of its ability).

Plus, "to the best of their ability" was quite lower than other sites. Buy.com's business plan, for the first few tech-boom funded years of its existence (1997ish?), was to sell products at a loss in order to drive other startup e-commerce sites out of business.

The US Code...

[Pettifogger]

It seems that the US Code and the First Circuit make this one pretty clear. If you have to agree to some sort of "terms of use" to get onto a website, you are bound by what those "terms of use" say. By clicking through, you have agreed to a contract and you have to abide by it. If your competitor's site requires you to agree to such terms, and those terms prohibit data mining, then you can't data mine there. Simple.

As for the ethical part of telling your employer about this... well, first, remember, this is just a decision of the First Circuit. If you live in a different Circuit, then it may or may not be binding on you. I know this jurisdictional stuff can be a little confusing, but a decision by a Circuit only affects the jurisdictions within it. Only the US Supreme Court (generally, I know there are federal tax, patent, admiralty, etc. courts, too) can make decisions that are binding on the entire country. If you're not sure, check with your corporate counsel. And it might be a good idea to forward the case to him anyway, you might be able to pick up some "bonus points" from your boss for being an especially conscientious employee.

Re:The US Code...

[geekoid]

what if I never click through? Instead follow a link into the site?
there are technical ways to stop this, the law shouldn't be involved.

i've been doing the same thing for clients...

[Anonymous Coward]

A client recently hired me to scrape the real estate listings off of a multiple listings service, and substitute his name and photo for the name and photo of the actual agent who had originally listed the properties. All of this gets shoehorned into his site's look and feel.

As hesitant as i was to initially do this (i did it for a friend who had promised this without understanding what it involved, and was under the gun), apparently several other agents are now dying to have the same treatment done to their websites. I wrote off this entire project the first time, since it is likely to choke the first time the original content site changes their layout or page structure. But now, others want the same thing, and i'm stuck whether to provide it with the same caveats, or to decline these crappy wrapper jobs.

Re:i've been doing the same thing for clients...

[Lt Razak]

It's your karma, dude. (And I don't mean slashdot karma)

it's called paper

[Provincialist]

I have a feeling that the reason your friend got the Best Buy bum's rush is his use of a pda to record prices. People are in general more scared of anything electronic than they are of the simple pencil-and-pad method. Perhaps they suspected that the pda was wirelessly linked to Circuit City's intranet. I've written prices on a piece of paper in a Best Buy before without suffering such a dire consequence.

later,
Jess

"Automated" can mean a browser too.

[Anonymous Coward]

Just the act of clicking on a link sets in motion a series of automated software tasks that will deliver me a price. Millions of shoppers do this every day. Are they in violation as well?

hmm, anybody rfta?

[lingqi]

on the ground that Explorica had used confidential information obtained from EF to assist in obtaining this pricing information in violation of confidentially agreements executed by former EF employees now working for Explorica.

seems like it's the using confidential information part that got the scrapper capped.

I don't see why accessing *public* information be problematic.

the only thing that may be of trouble is the website EULA, but then the EULA would be saying the same thing as "don't visit my store unless you intend to buy," which would be rediculous in brick-and-mortar world (and should be similarly in cyberspace).

last question, though - why the heck would you ask this kind of stuff HERE? wouldn't a law-forum be a better choice?

Re:hmm, anybody rfta?

[Ian Peon]

the only thing that may be of trouble is the website EULA, but then the EULA would be saying the same thing as "don't visit my store unless you intend to buy," which would be rediculous in brick-and-mortar world

What, never heard of a 2-drink minimum?

Re:hmm, anybody rfta?

[Anonymous Coward]

I work for one of the major online travel sites.

If you screen scrape us, and we notice it (and we very often do), your IP address will be blocked by our firewalls.

What most people don't realize is that very often any search performed on these sites costs the company money. In many cases, if you search for, say, a hotel on Expedia or Orbitz or Travelocity, those companies are paying one of the major Hotel reservation systems for their results from that search.

So, if someone is screenscraping our site, each search they perform to grab prices for a hotel for a day requires us to send some money to Pegasus or Travelweb or one of the other biggies. In hotels, for example, Pegasus is the big CRS. In order to get enough bookable hotels to make thier site useful, Expedia and Orbitz and Travelocity all need to buy search results from Pegasus.

So it's not just a matter of taking their bandwidth and CPU time.

We do allow some companies to screen scrape our site for some various reasons, but they all pay us for the privilege. We don't tend to take legal action against unallowed screen scrapers, but we will do what we can to make their life harder.

Re:hmm, anybody rfta?

[Zirnike]

What, you beleive the claims of the plaintif on a lawsuit? They'd claim the price that they had skywritten over every major city in the US plus Des Moines is confidential so they can procede with their SLAPP.

It might or might not be true, but don't try to claim that ignoring their claim is a bad idea...

Re:hmm, anybody rfta?

[Anonymous Coward]

Hmmm... that depends... what exactly is rFTa? ;o)

Re:hmm, anybody rfta?

[Anonymous Coward]

No, I have not Read Fucked The Article... not that it's any of your business.

I'm not entirely sure what Read Fucking is
Some ideas come to mind, though....

Have you RFTA?

Re:hmm, anybody rfta?

[Anonymous Coward]

--the only thing that may be of trouble is the website EULA, but then the EULA would be saying the same thing as "don't visit my store unless you intend to buy," which would be rediculous in brick-and-mortar world (and should be similarly in cyberspace).--

Wanna bet. ...by clicking through this link, said person agrees to sell soul to Devil..now put head between legs and kiss your ass goodby..because we now own it.

froogle seems to do exactly this

[dakainivanua]

Doesn't Google's Froogle service do exactly this?

Airline prices may be copyrightable

[Anonymous Coward]

With respect to the question of whether prices are "in the public domain":

Airline prices are not like other prices. They involve complex calculations, calculations sufficiently complex that only a very small number of computer programs are capable of doing them, programs maintained by hundreds of developers that depend on data that costs companies like Expedia and Orbitz literally millions of dollars per year.

There have been court cases that have disallowed copyrights on numbers created by un-creative processes; the standard examples are telephone books and baseball scores. But it's not clear that airline prices would fall under the same categorization: there's a huge amount of "creativity" involved in their calculation, as evidenced by the different prices that appear for the same flights on different web sites.

So even if you can legally screen-scrape Best Buy, I wouldn't assume you can legally screen-scrape Orbitz or Travelocity or Expedia. And note that those sites quite explicitly prohibit robots and screen-scrapers.

Re:Airline prices may be copyrightable

[Adversive]

They involve complex calculations, calculations sufficiently complex that only a very small number of computer programs are capable of doing them, programs maintained by hundreds of developers that depend on data that costs companies like Expedia and Orbitz literally millions of dollars per year.

You honestly believe this? How complex could this possibly be?

The prices between sites are wildly different because there is no benefit for the airlines to compete for price in most situations. Because it is not easy to find the lowest price, people are much more likely to settle for a lower price.

The airlines could easily show you the cheapest prices. The more confusing and complicated this process is, the more likely a customer will pay more money.

Terms of Use

[justzisguy]

IIRC, if the web sites include a statement forbidding the collection of prices from their website by bots in their terms of use, they might have a civil case against the company that you hired, but I wouldn't worry about it as the developer of the bot.

Some stores do discourage comparing prices...

[Parsa]

Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually?

I read an article a year or so ago...can't remember where, maybe here, that Besty Buy and Circuit City do discourage this activity. The problem the article brought out is how do they tell the difference between comparing prices and people data mining? J

Non-issue

[Anonymous Coward]

Prices cannot be copyrighted, since they are facts. Prices are not trade secrets if they are advertised in public (even on a shelf). Prices cannot be patented.

So, why can't they be researched again? That's right. It's a non-issue.

Next.

Seek real legal advice.

[Gerad]

I am not a lawyer.

Slashdot is not a lawyer.

Slashdot is not a replacement for a lawyer.

Individual posters on slashdot may be lawyers, but are you really willing to trust your future to what some random person online says, when they could be a lawyer, but could also be some 14 year old kid who thinks it's amusing to screw with people?

Repeat after me:

I will seek proper legal advice.

Seriously, this comes up time and time again. If you're in a situation where you need actual concrete legal advice, SLASHDOT IS NOT THE PLACE TO GO. Sending in an Ask Slashdot is fine for theoretical questions, but when your ass is at stake if a lawsuit comes around, do you really want to trust your future to the legal advice given to you by Anonymous Cowards and karma whores?

Re:Seek real legal advice.

[BuckaBooBob]

Depends on his contract too... If they own all rights to said produced software.. Its thier ass not his.. Also depending where you are... Its ok to develop the software.. Just don't use it :)

Re:Seek real legal advice.

[Timesprout]

Christ are you trying to kill all our fun. Just look as the way we get to pontificate about they way things should be and how we would fight the system if it tried to screw us. Think how many people got to post recently that there was no way they would have settled out of court with the RIAA because they had a bullet proof case. Real slashdotters would have fought all the way to the supreme court and then taken to the hills with a a weapon and a few supplies as a last resort to continue the fight. Or so their posts would have us believe anyway.

This is Slashdot please stop trying to introduce reality. It wont work and you will only frighten the children.

Re:Seek real legal advice.

[Gerad]

Yeha, but you're still going to want a lawyer interperting the contract, and the relevant law for the city/state where you live. I'm sorry, but going to Slashdot for legal advice seems about as ridiculous as asking people off the street for advice on how to build a space shuttle. Sure, you might find someone who actually has the proper background, but you're just as likely to find people who don't know and guess. And I would imagine that Slashdot, you're even more likely to find people who will give you wrong answers on purpose.

Re:Seek real legal advice.

[sigwinch]

If you're in a situation where you need actual concrete legal advice, SLASHDOT IS NOT THE PLACE TO GO.

And how many lawyers know the legal implications of soliciting anonymous HTTP GET requests? How many know the semantic differences between GET and POST requests, and their relationship to the computer fraud and abuse statutes?

Furthermore, most law in this area is being made right now by the people actually working in this field. I.e., a large proportion of /. participants. The last thing we need is for the future to be built behind our backs by a bunch of expensive stuffed suits.

If you don't care, fine. Fuck off. Go build your own website dedicated to Fluffy Technology News That Doesn't Matter.

Pricewatch is not a scraper.

[Anonymous Coward]

They simply take info given voluntarily by vendors and list it. They do not actively go out looking for this info, or actively ask for it, nor do they verify it. Inclusion on their pricing list is strictly voluntary.

Re:Pricewatch is not a scraper.

[oaf357]

True. But, Pricewatch has a huge name so not submitting to it is silly.

Re:Pricewatch is not a scraper.

[EvilBudMan]

Exactly right. The vendor sneds them this info. Many times I've been able to beat PriceWatch by checking Buy, Outpost, etc.

Are you getting paid?

[g0hare]

STFU and do your job.

Why waste time in the legal system?

[x00101010x]

Filing lawsuits to protect your price information is just dumb, not to mention waste (if not abuse) of the legal system.

Personal feelings about freedom of information aside, and just from a coder's POV, here's my solution.

If they really want to avoid getting scraped, they should just get their existing, underpaid web developers to create a backend setup that generates the prices as gif's that give OCR hell (such as those used to prevent automated registration of say Yahoo! email accounts).

Coders are cheaper than lawyers (at least those needed to write such code as this).

Sure, the compition could pay more money to get somebody to develop better OCR to read each and every dynamically generated GIF, but most people require proof reading of OCR data, which leads to even more cost.

Something I learned from my Uncle who works with the DOD is this: Any lock can be picked; Any encryption can be broken. It's just a matter of if it's worth the time and money to get what's inside.

In short, with a little one time cost, the company that doesn't want it's prices scraped can just make it so hard to scrape their prices that it's not worth it. The price of scraping the graphically displayed price tags would also be an ongoing cost of software and proofreaders that would dip into profit margins, which management at the company that desires the scraping won't like.

It's not perfect, but it's better (and more bankable) than going whining to the legal system. (Especially since coders are generally cheaper than lawyers).

Re:Why waste time in the legal system?

[BuckaBooBob]

Shit thats brilliant dude... Better patent it! "If they really want to avoid getting scraped, they should just get their existing, underpaid web developers to create a backend setup that generates the prices as gif's that give OCR hell (such as those used to prevent automated registration of say Yahoo! email accounts)."

Re:Why waste time in the legal system?

[x00101010x]

I think the use of this for yahoo.com email and other such uses would count as prior art.
Besides, I posted the idea from a Coder's perspecitve, from a personal perspective, I'd patent it perhaps to keep it from being used (if it was actually patentable). =P

Re:Why waste time in the legal system?

[Afty0r]

If they really want to avoid getting scraped, they should just get their existing, underpaid web developers to create a backend setup that generates the prices as gif's that give OCR hell (such as those used to prevent automated registration of say Yahoo! email accounts).

If they were to do this, would they not be falling foul of accessibility regulations for users with disabilities? People who use screen readers, or anyone who needs to increase text size to read off the screen will be unable to purchase their products.

I'm a UK resident, so not entirely sure, but I believe you have something called "508" in the US, which is a government regulation mandating accessible websites for certain companies, I'm not sure what the criteria are for selecting companies though I'm quite sure this would break the regulations.

Re:Why waste time in the legal system?

[Lt Razak]

why not just develop dynamically changing HTML to make them update their screen scraping programs? Fill the source with garbage that doesn't show up in the final screen?

Re:Why waste time in the legal system?

[jafiwam]

Section 508 mandates that federal and some other types of government web sites fit the guidelines. It is not a rule that has to be applied towards any other site, although it is a really good starting point for non-government web sites. It is often used by webmasters to convince the Pointy-Hairs how to build a web site correctly and keep the Flash and other crud as decorative only... not the whole site itself.

A side benifit is that being Section 508 compliant makes really friendly robot fodder, so the site gets in Google and other search crawler sites faster.

Any high-profile webmaster is going to follow something like Section 508 anyway, just to minimize the chance of getting a lawsuit. You can find out more here:

http://bobby.watchfire.com/bobby/html/en/index.jsp

and the 508 Guidelines here:

http://www.access-board.gov/sec508/508standards.ht m

Re:Why waste time in the legal system?

[BuckaBooBob]

Well It seems to utilize a current "technology" in a new manner seems to be patentable these days... even though no innovation has actually takes place.. Just common sense use of a technology. :)

Re:Why waste time in the legal system?

[x00101010x]

I think such rules exist, but they don't get enforced much. If you ever try looking at online stores such as buy.com or walmart.com you'll see what I mean.

But, you're right, I completely forgot about that issue... of course you could always check the UserAgent of the person visiting, but that would defeat the purpose, if scrapping prices is against the TOS then what's to keep them from faking their UserAgent to be an audio web browser?
/me goes back to the drawing board.

Re:Why waste time in the legal system?

[x00101010x]

The gain isn't in the complication of updating the scraper program, its in the cost of proofreaders for the OCR data. Most corporate types won't trust OCR data that hasn't at least been skimmed by a proofreader. A proofreader is a whole person (or outsource contract) that requires wages, benefits, etc.
In other words, you're not creating a problem that will be solved by a few existing programmers, it's a problem that will require additional staff.
Every encryption (which this kind of is through obscuration [sp?]) can be broken, the goal of real world encryption is just to make it more expensive to decrypt it than the value of the data locked inside.

Ah, just the comment I was looking for.

[ahfoo]

I think what you have to look at is the media context in which the prices are displayed.
It's quite true that many stores will try to prevent you from making recordings of any kinds on their physical premises. I've been reprimanded by store managers many times for taking photos in the store. But their right to prevent me from creating media on their premises is based on their property rights, not any some legally backed authority to censror the media.
The web is a totally different story. I use web scrapers all the time and a site that doesn't like it can kindly take its ass off the web. Once you place material on the web, it is published. If you don't want to publish your prices, you don't have to. That's like publishing a book and complaining the readers read it too fast.
The people who compain about such things are the idiots who create unworkable business plans based on their own assumptions about how people are going to use the resource. This is an interesting issue with news media that want to sell access to their archives. There's no way they can both publish to the web and prevent me from caching old copies. If that's the business plan then web publishing is an inappropriate business decision and guess who should pay for bad business decisions: the consumer, or the fool who pursued an ignorant business plan?

Re:Ah, just the comment I was looking for.

[Vej]

That's what I thought of it as, a public publishing. That's how copyright law treats it anyway, esp in documentation of using the data.

I mean, I suppose if you decided to republish it, you might need to state "some" documentation details. But look at reviewers online, they can even call the product bad and the price. How is it any different than reading a million reviews and using their data? That's the whole purpose.

What a consultant should do.

[pete-classic]

If you have a genuine concern you should raise it. (As always, if it isn't in writing, it never happened.)

Then let it go. If you want to be extra-double sure, get your own lawyer.

You, however, are not their lawyer. It is not your job to advise them on legalities.

-Peter

Same as in the Physical World

[serutan]

Anybody who publishes information about their business runs the risk that a competitor will get hold of the information and use it in some way. This has always been a fact of life in the physical world. As computers came online in recent decades many companies have maintained databases of information about competitors' products. The Internet doesn't change any of this.

Don't you realize this is Slashdot?

[Anonymous Coward]

These people all think they know more about any topic than the actual practitioners and experts in any given subject. Jehovah himself could appear here, the the /.ers would lecture Him on how Heaven is made.

If they don't want the price stolen.

[Anonymous Coward]

Output the price in a un-OCRable jpeg image.

Re:If they don't want the price stolen.

[Theatetus]

Output the price in a un-OCRable jpeg image.

What did blind people ever do to you?

Explorica case

[RedWizzard]

Reading the summary of the case it looks like the court ruled that Zefer (the developers of the scraper software) could not use the scraper on EF Cultural Travel's website because their client Explorica was legally enjoined from using a scraper. Zefer are not permitted to assist Explorica in violating the injunction against them. The injunction against Explorica was issued because Explorica used confidential information to assist in gathering the pricing information when, as former EF employees, they had signed confidentiality agreements in favor of EF.

So to me (IANAL) it does not look like any precedent against data mining for pricing information has been set. The closest this case comes to doing that is the First Circuit's opinion on what constitutes authorised use of the site under the Computer Fraud and Abuse Act. They say that the terms of use of the site can restrict the use of scrapers. This would be a weakening of the district court's opinion which was that the authorisation question should be looked at in light of the parties "reasonable expectations" (i.e. the website owners could "reasonably expect" the users of the site to be human and not scraper software).

click-through considered non-binding

[Provincialist]

If you have to agree to some sort of "terms of use" to get onto a website, you are bound by what those "terms of use" say. By clicking through, you have agreed to a contract and you have to abide by it.

Aside from the well-known problems with any click-through agreement (contract between unknown parties, software circumvention, lack of notarization, etc.), the additional flaw in this case is provided by web archives. If you don't want to have to look at a click-through page before reading your competitor's deep dark secrets, just download what you want from a public web cache. Are these jokers going to turn around and sue Google, as well?

Actually, that brings up an interesting point. When Google gets sued for forwarding information to competitors without click-throughing them, they will probably deny that such was not their "intent" in providing the web archive. Of course, the competitors do have an "intent" that the original site doesn't condone. But there is not a technical means of determining intent over the current version of HTTP. If the original site wants to do this, it is using the wrong technology. Of course someday if the ebXML folks get off their collective butts, we might have some sort of contract-negotiation protocol. I doubt a consumer e-commerce site would be interested in erecting such barriers to entry, but this would probably be useful in certain B2B contexts. Until then, honoring click-through pages in the breach will only harm the internet. Any court case that declares that particular intents make a party ineligible to download particular material served over the web (that's my understanding of the agreement that we're clicking through here) will only harm the web and all open systems.

later,
Jess

Re:click-through considered non-binding

[sigwinch]

Aside from the well-known problems with any click-through agreement (contract between unknown parties, software circumvention, lack of notarization, etc.),...

You are talking out of your ass. An enforceable contract has four elements:

1. A meeting of the minds between the parties to the contract. Evidence of meeting of the minds can be provided by incorporating a unique element in the contract and requiring the web surfer to key it back in before they are allowed to proceed.
2. An exchange of "consideration", which basically means anything of value. If you look at several of the protected pages, that fact is evidence that you have received value. If the terms of the contract provide a competitive advantage to the web provider, that is evidence that they have received value.
3. A written statement of the terms. (Optional. If not present, damages are typically limited to something like $500 by the statute of frauds.)
4. Predication of consideration on agreement. I.e., they have to withhold the valuable consideration until after you have agreed to the terms. (Which means most "terms of service" pages are unenforceable bullshit. Advertising that you accept HTTP GET requests on port 80 is a solicitation, delivery of the requested page is consideration.)

If you don't want to have to look at a click-through page before reading your competitor's deep dark secrets, just download what you want from a public web cache.

How can Googlebot waltz right in if there's a click-through barrier?

But there is not a technical means of determining intent over the current version of HTTP.

Yes, there is. HTTP POST requests are defined as causing the server to take some action (and not merely provide data back). Ergo, using a POST against the will of the server operator is generally a violation of the computer abuse statutes.

Until then, honoring click-through pages in the breach will only harm the internet.

Bullshit. First, I can find anybody's competitors in a couple of seconds. If I find their proposed agreement too annoying (or their Javascript, or their blink tags...), I go someplace else. The invisible hand will guide them toward an appropriate level of onerousness. Second, randomly smashing contracts because somebody thinks they are unfair would be bad. What's the difference between John "Dumbass" Lawyer's click-through page, the agreement for real-time NASDAQ quotes, and a 1-Click purchase order? Nothing! It would wreck Internet commerce to capriciously void contracts because some jackoff decides they're unfair.

Re:click-through considered non-binding

[Mhtsos]

How can Googlebot waltz right in if there's a click-through barrier?
Googlebot isn't sentient, nor can it do a symantic analysis of an agreement to see if it fulfills some criteria, so it can't agree on anything. Either ALL bots shuld be banned or it's ok for a bot to go through an otherwise unprotected click through agreement. I see a lot of those "type the number" anti-bot fields lately.

Re:click-through considered non-binding

[sigwinch]

The implication is that if a dumb bot can get through your "agreement", you haven't attempted to withhold the resource until the client agrees to the contract, and thus there is no agreement.

It's trivial to use HTTP POST, a hidden form field, and cookies together to keep dumb bots out. You don't even have to make the user type anything in--that's just to provide additional evidence of intent to agree.

That Reminds Me...

[Esion Modnar]

of the old joke about this fella who was selling everything in his store at a loss.

When somebody asked him how he can make a living like that, he replied:

"Volume!"

Re:That Reminds Me...

[Scaba]

Joke? That's been Amazon's business strategy for years!

Re:That Reminds Me...

[mattsucks]

Was his last name Bezos?

Re:That Reminds Me...

[Anonymous Coward]

Actually it was a SNL sketch on a bank that only made change.

Trespassing laws

[LemonFire]

I worked on a similar project about three years ago actually looking through pretty much the very same websites. :) The project was finally shut down when one of these websites used an old trespassing law to stop us i.e. saying that we were using the site in a way that broke the user agreement and therefore we were trespassing.

Well...

[Anonymous Coward]

.. If the guy who hired you read ./, then you dont have to ;-)

Are Extraneous Commas, Annoying?

[reconn]

I'd say yes. Second only to apostrophe's.

Re:Are Extraneous Commas, Annoying?

[radish]

Could that headline be any more, useless?

Re:Are Extraneous Commas, Annoying?

[Anonymous Coward]

Extraneous commas are easier to explain.
I distinctly remember being told that
commas are supposed to indicate a pause
in the sentence, so I have a tendency to
put commas all over the place whereever I
feel the sentence needs to be broken up.

So blame it all on elementary teachers.

Read the case...

[anubis]

Read the case...EF Cultural Travel BV v. Explorica hinges on the fact that the defendant company hired an ex-programer from the plaintif company. The programmer had special knowledge of codes used in the pricing (which he had signed a confidentiality agreement not to disclose). When he made the scrapper program he violated the confidentiality agreement.

It was the violation of the confidentiality agreement that the court held was illegal.

As for whether you should tell your employer, it depends on your employment agreement! :) Depending on how the contract is written you could be jointly liable.

While this is a 1st Circuit case, it has been followed by the 5th Circuit (Ingenix, Inc. v. Lagalante) and cited in cases in the 7th and 9th Circuit.

Hope this helps.

--me

The Human Element

[aaaurgh]

"Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually?"

I guess it's as much about the percieved ability to stop a human and the lag in that information being used.

A number of years ago I was using my Psion to check the E-numbers on the food labels, I'm both vegetarian and have an intolerance to certain food additives. The store's duty manager came up to me to ask me to stop or I'd be ejected from the store for unacceptable behaviour, it took quite a while to persuade him that I was not an undercover operative from a competitor or investigative reporter - I was just doing my shopping carefully!

People don't like the idea that they can't stop the robot from checking their prices and providing that information back in seconds so that their competition can undercut them almost as soon as they post their prices.

With modern technology, retailers like to believe they could stop a person from gleaning too much information for a competitor in person but it's all just perception, the fact is that we're well past all that.

Re:The Human Element

[TheNumberSix]

I was also kicked out of a store. I'm working on a master's degree part time and I had to take a class in marketing.

We were doing a project on the ice cream market. I had to get prices, shelf locations, the numbers or brands and so on.

I got kicked out of two stores before I got enough data to do my project. (Albertson's and Vons kicked me out but Smith's was nice and let me collect data.)

So the answer to the story submitter is that you most certainly cannot walk into a store and record pricing manually.

Re:The Human Element

[Mhtsos]

I was in a Vodafon shop, looking for a mobile phone. I've been shopping around and was naturally writing down prices to compare later. The emploee politely told me that I couldn't write down the prices she was telling me according to company policy. So aparently it has something to do with the storage medium (neurons are ok, paper and databases are not). Needless to say that this shop was not included in my comparison later on and lost a potential client because of its policy.

price mining not okay ?

[ibennetch]

I remember reading something about a year ago about someone who got arrested at (I think) Circuit City for writing down the prices; on a laptop he brought with him, as I recall. A quick Google search didn't turn up anything; maybe someone will remember more information than I can recall.

Also, I've seen some online sites seem to have "this information is for your personal use only" type 'licenses' for their prices; the notices I've seen are usually in tiny print in some legal page link no one ever clicks; and the legality of such a clause is something I'm not sure about either.

Re:price mining not okay ?

[kasek]

if i recall correctly, it was actually best buy...i saw it on 60 minutes or 20/20 or something...

anyhoo, also a search on google turned up nothing, but a search on google groups for 'man arrested "best buy"' turned up lots...it was back in 1997 or so......

he was recording prices on his laptop in the store, was arrested, sued for 90k, lost, spent tons of money, made bad publicity.

Re:price mining not okay ?

[gl4ss]

as always, the 'rights' of the us citizens keep to amaze me... arrested for taking down prices?? that's pretty tough. prices for products(that a place sells) and services they provide (for consumers) are required to be shown clearly around here, and 'scraping' for prices doesn't seem to be a problem either, as such info they would be required to give when phoned up or visited. how can you buy anything without getting to know the prices?? beats me.

heck, would they make illegal saving newspapers(with ads) too?

you can add any legal notices to your pages but the effect they have is a whole different matter. republishing prices shouldnt be illeagal anyways either as long as you state where you got those prices and when..

Re:price mining not okay ?

[gozar]

It was Ronald Kahlow and Best Buy. He was arrested twice and sued Best Buy, but lost.

ebay has an anti-scraping TOS clause

[phr2]

and I think they have taken people to court over it.

Re:ebay has an anti-scraping TOS clause

[Animats]

Yes. It's a well-known, and controversial, case,EBay vs. Bidder's Edge. That was only a preliminary injunction, though. Bidder's Edge shut down on February 21, 2001, so the issue was never really decided.

Re:ebay has an anti-scraping TOS clause

[galacticdruid]

but that was only because the site ( some auction aggregator ) was _nailing_ them, using something like 1% of their entire server load, and using evasive tactics _after_ getting a cease and desist.

I'm reminded

[Nexzus]

of a man named Ronald Kahlow and his troubles with Best Buy back in 1997.

Re:I'm reminded

[shumacher]

Both Best Buy and the judge were off their rocker. Best Buy can ask him to leave. The store is private property. If he is a customer, they should have allowed him to continue to calculate prices. If Best Buy asked him to put his laptop away, he should have complied. Further, the article mentions other stores that claim to not have a policy against Mr. Kahlow's behavior. I know for a fact that one of them actually is okay with Mr Kahlow's behavior, but does have a policy (and not just an "unwritten" one)against competitor's doing what Mr. Kahlow was doing. (Which is a different thing - and more on topic - one means a potential sale for them, the other means potential sales for their competition) Their policy is to ask the suspected competitor what they are doing, and then if confirmed as a competitor, they are to make a specific statement essentially revoking that individual's right to visit the store, and then, they are to ask them to leave. When that store shops a competitor, policy compels them to wear normal street clothes, to not use any automated devices, and to admit their affilliation and leave if asked.

Today we learn that

[Anonymous Coward]

Orin S. Kerr is a lawyer.
In fact, he's a law professor, too (after being a member of the DOJ's cybercrime division).
He taught my computer crime law class.

--Dan

courts vs. open systems

[Provincialist]

...it does not look like any precedent against data mining for pricing information has been set. The closest this case comes to doing that is the First Circuit's opinion on what constitutes authorised use of the site under the Computer Fraud and Abuse Act. They say that the terms of use of the site can restrict the use of scrapers.

To me, this is a loss for open systems. Nothing in any of the relevant RFCs mentions a method of specifying or obeying provisions against automatic downloading for a particular purpose. I'm sure one of them mentions robots.txt, but that would prohibit all automatic downloads, and I'm sure most e-commerce sites don't want to chase off Froogle and its ilk.

I'm against any judicial action that changes by fiat the conventions under which the internet operates, especially in the jursidiction in which I happen to reside. It is fundamental to the web that one only serves information one wants public. Someday there will be protocols that deal with issues of trust, priveleges, and negotiation programatically. They will be used in certain circumstances, but HTTP or its descendants will be far more common. When the courts trade the future of open systems for a temporary convenience to businesses that are careless with their proprietary information, we have lost.

later,
Jess

Competition

[Anonymous Coward]

Competition is the whole idea in capitalism. In order to make capitalism work, your competitor must compete with your product/prices, in a way that is transparant to the consumer.

Trying to avoid competition should be punishable, since it is pure terrorism :)

In my country they are even trying to prohibit certain non-transparant pricing methods (banks telephone companies), since they disable the competition by obscuring the price to the consumer.

go ahead ! those guys are communists ! :)

Re:Competition

[Anonymous Coward]

Say that to Microsoft, please.
And louder!

Data Mining for profit

[Anonymous Coward]

I wrote a simple data mining/aggregation app last year that scans Hot Deals forums (ie. Fatwallet.com, Anandtech.com, Gotapex.com, etc.) and emails the new posts out each hour to my subscribers. The service has always been free. I maintain the original links and titles, so I don't feel as though I am stealing anyone's links. A few months ago, I decided to put some ads at the top of my newsletter to make a few extra dollars to pay for my time in writing the app. I immediately received a response from one of the main forums I aggregate, stating that I needed to stop running my newsletter for profit, or remove their copyrighted and trademarked name from my list of links, as they didn't allow others to benefit from their name.

Well, not knowing what I might be getting myself into, I removed my advertising.

So my question is this: Are forum threads copyrightable, even though they're written by users of a forum and don't necessarily represent fact? Can I advertise without being sued for using someone else's links in my newsletter, even though I am not representing them as my own?

-- dan
http://groups.yahoo.com/group/dansdeals/

Re:let me ask you this.

[jjoyce]

You would then need an additional comma after "Data".

Not the really data mining?

[fatwreckfan]

What he's doing isn't really data mining. Data mining is the process of discovering patterns in data which are not known ahead of time, such as the infamous "beer and diapers" correlation.

That said, I don't understand why the author is worried. I can't see how looking at publicly posted prices could be considered illegal.

Adam Smith and perfect information

[viniciusxp]

People who studies economics faces some irreal hipothesis in text books. The first topic most students have to deal with when taking the microeconomics course is when you have a big group of firms selling the very same product. If the buyer has perfect information about prices hi will choose the lowest price. The buyer's choice will influence the behaviors of all other firms that will tend to get their prices down to beat the one choosed by the buyer. We will have a dynamics that will make the price go down until the item will cost to the user the same it costs to be produced. In the real world it is very unrealistc to believe someone could have information about all the sellers prices. But with Data Minig we can have MORE information about sellers than in the real world, and we can access this infrmation with a smaller cost. We should then be nearest to perfect competition books theorize than in the real world. There is although some problems to solve before jumping to this conclusion: There is not that big number of firms competing, delivery fees, warranty and time of arrival of the product can be very different from seller to seller. Could a "perfect bot" could handle all this information. If the answer is positive firms can folow two paths : cartelization or dumping. The first one happens when firms pacts prices together and force buyers to py more, because competition is "freeze". The second one hapens when the firm artificially gets down the price to a lower level than the costs to force the competition to bankrupcy. Both behaviors are dangerous to consumers and are forbiden in most countries. IMHO a site's EULA can't go agains market law. I presume that, at least inside the same democratic country, it is legal to data mine in that way. And I can't see why a competitor can't use it as tool to build it's price strategy. It's the invisible hand Adam Smith's intuished about. The WWW is evolving, maybe in a way some people can dislike, and is using the same rules we use in the real world to make money. And I'm sure competitor will soon find solutions to prevent data mining from their sites, at least information they don't want to share. IT solutions. That do not require lawyers but intelligence and insight.

Re:Adam Smith and perfect information

[Anonymous Coward]

In game theory, we learn how all those firms, knowing that they compete only on price & sell the same thing can all adopt a strategy of "double your money back if you get it cheaper." By doing that, they effectively collude, and raise the price they can sell it for. It works because they'd be cheaper than the other businesses would be if they sold it at cost (thus, those businesses would lose all their business), forcing the others to adopt the same strategy to compete, allowing them all to have the same price, one higher than they could sustain without that policy, even though none of them ever have to actually pay out that fee, since they all choose the same, unique price which happens to be the optimum for them (the Nash equillibria) ...

Nice, huh? :)

Re:Adam Smith and perfect information

[mhayman]

Yes, price spiders are good for consumers. But I think you're confusing retailers (who sell convenience) with manufacturers. Perfect competition is not simply an product of perfect information.

Perfect competition assumes that every competitor has the same product -- all VCRs made by all firms are the same, etc. In the "real world", as you put it, products are differentiated, that's why the model is closer to monopolistic competition than perfect competition, and why the same technology (VCRs) can be sold for different prices, because each brand is slightly different than another (e.g. Sony vs Hitachi). If people had perfect information, *and* all products were exactly the same, then anyone that charged a higher price would be unable to sell any of the goods they produced.

Sony spends a lot of money differentiating itself and establishing a name and association with quality, and that's why it can charge a higher price for its products and people will buy it.

Perfect competition is really a reference to efficiency -- the goal (as you state) of the invisible hand is to make products cost as much at market as they do to produce. In nonopolistic competition however there are still and will always be wasted resources because the firms are spending money on advertising to differentiate themselves, thereby inflating the prices of their products. In perfect competition all resources would go towards the creation of the output, and economic profits would be zero.

MOD PARENT UP!!!

[Anonymous Coward]

WTF? Mods on crack. This post was fucking hilarious. I had to read it twice before I got it, but once I did it cracked me up. You guys need to get the stick out of your butts.

House rules

[scgops]

You see signs everywhere you go:

-Shirt & shoes required.
-No loitering.
-No soliciting.
-Check all bags at counter.
-No more than two students allowed in store at one time.
-Parking lot, bathroom, etc. for customer use only.

Just because a building (or a web site) is in a public place doesn't mean that everyone is free to do whatever they want. Business owners are free to create house rules that everyone needs to follow.

Similarly, web sites can legally restrict what you are allowed to do when you visit them without having to build security measures to force compliance. If web retailers don't mind robots harvesting their inventory and prices, great. If, however, they want to place restrictions on who can access their site and how, that's entirely their prerogative.

Think about it. Leaving the door to your home unlocked would make it easier for people to steal your stuff, but it still wouldn't make it legal for them to do so unless you put up a sign saying something like, "Free for the taking."

Web scrapers are legal to develop and they're legal to use on sites with acceptable use policies that allow them. However, your customers should be prepared for the possibility that some or all of their competitors could make them stop using it at some point. And, in the interests of maintaining your own professional ethics, you should probably call their attention to the issues surrounding the job they're asking you to do.

Re:House rules

[Zerth]

Stupid question, how do they tell if you are a student? (I mean besides if you happen to be carrying a bag full of textbooks or have your uni ID tagged to your forehead)

Re:House rules

[akadruid]

while they cannot prove you are a student, they will try and enforce the rule on anyone of typical student age.
obviously hidiously inaccurate, but it serves the purpose to reduce vandalism/theft/staff abuse in stores near schools/colleges/universities.
Where I live shops near primary & secondary schools use this rule to prevent kids shoplifting sweets, fags, booze and porn mags.

Re:House rules

[gurps_npc]

The problem is ANY "sign" they put up to prevent the robots from doing this ALSO prevents people from buying.

Try writing the sign and you will see.

Example: You are not allowed to copy our prices. WHOOPS, everytime your browser accessees that site it copies the prices.

The real problem is that what they want to prevent you from doing is COMPARING their prices for identical goods in an EASY TO READ manner.

But they do not have the right to affect what you do with their compeitors' prices, so you they can neither prevent you from comparing them, nor require that if you do compare them, you must do so in an awkward manner.

you are stating the obvious

[g4dget]

Every time someone asks a legal question on Slashdot, people like you come out of the woodwork and point out that Slashdot is the same as going to a lawyer. You don't say. Do you really think everybody else is stupid?

Discussing legal issues is not just a business for lawyers. Non-lawyers can give each other useful pointers. And non-lawyers actually have an obligation to determine whether their legislators are doing a good job with the laws they enact and judges they appoint, and a healthy discussion is a good start.

Re:you are stating the obvious

[Jerf]

You don't say. Do you really think everybody else is stupid?

Stupid? No. But the number of people who seem to think they are lawyers is very large, and not just on Slashdot either. I can't count the number of times in my real life I've discussed intellectual property issues and not only has the other person been very, very wrong, but I was not even able to get them to listen to me.

I'm not a lawyer, but I've taken a close interest in that sort of thing and I know the basics very well.

As a Slashdot example, basically, if someone is insisting that they have a "fair use" right to something, unless they are jusitifying it with reference to the four criteria used to determine if something is fair use, they're wrong.

People seem to need periodic reminding that they aren't lawyers, and other non-lawyers aren't lawyers.

For computer examples, how many times have you heard someone around you give an incredibly wrong reason for a crash... and stick by it, even after you fix it, because of course they're right?

Discussion of issues is one thing. Talking about something that could make or break a career, that's a time for a real lawyer, not hundreds of people who think they are lawyers.

Please read Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments. I don't know if that's risen to "classic paper" status but as far as I'm concerned it has.

er, BS:Re:Best buy is a really really bad example.

[randyest]

Sorry, I call bullshit.

Over the past 6 or 7 years I've used a palm (handspring visor, to be more precise) hundreds of times, in every Best Buy (and Circuit City, MicroCenter, etc.) in the Boston area to record prices. I've never had anyone even look at me funny.

Maybe it's related to how guilty (or difficult to remove) you look, but I really doubt that happened to anyone ever (note the once-removed story -- it's always a 'friend of mine' in these types of stories.)

In any case, what kind of wuss would leave without making a fuss and forcing them to call the police over something so ridiculous? I could be using my palm to look up my friend's number to call and ask which video card to buy. Fsck them if they don't like it.

Or, maybe this particular Best Buy was located in an airplane and the event happened during takeoff or landing. Or your friend lied to you. One or the other.

Even walk-in data mining will get you in trouble

[rat_axe]

While it's not illegal, collating price information by hand will get you booted from a store. It being private property, they don't have to have a reason to ask you to leave, and if you refuse it's trespassing (in the U.S. anyway). I remember one of the TV newsmagazines detailed a case where a guy was entering price comparison info into a PDA at one of the big chains (Circuit City?) and they kicked him out. He sued and lost. Since they can't trespass you on the Net, I guess Big Corporate will resort to whatever cockamamie methods are at their disposal.

Public servers are really private

[mypalmike]

While I agree wholeheartedly with your argument, the courts do not. Another example from the same paper is even scarier. In a similar case (Register.com vs. Verio), the court said the use of search robots to gather information may be illegal regardless of whether it broke the site's terms of service:

"Instead, the Court concluded that the mere fact that Register.com had decided to sue Verio meant that Verio's use of the search robot was without authorization: 'because Register.com objects to Verio's use of search robots,' the Court held, 'they represent an unauthorized access to the [Register.com] WHOIS database.'"

[Ironically, the pdf for the paper apparently uses some feature of Acrobat which disallows copying text from it. I guess they don't want robots scraping text out of it or something. First time in quite a while I've had to type a quote from the net by hand!]

Re:Public servers are really private

[Anonymous Coward]

Hrm. I'm wondering if it makes sense to consider
this from another angle. To sum up the court's
apparent position:

1. put information on a publically accessible
webserver
2. AS LONG AS there is a suitable "terms and
conditions" notice, public access to this
information is bound by these T and Cs - so
potentially even looking at a particular page
counts to misuse.

So, if I put a big-screen TV in my front window,
with a notice next to it saying "do not watch
the TV" and then ran copyrighted videos on it
all day, would I be breaking the "not to be shown
publically" clause in the copyright, or would I
be safe - everybody peering into my front room
would in fact be violating my privacy, and
therefore I could get away with this scott free?

I think somebody should run an experiment... :-)

Re:Public servers are really private

[Anonymous Coward]

I am afraid your experiment is faulty -- you are doing it in the real world. Every judge knows that the rules in the internet are totally different from the rules of the real world.

Cost of data access is damaging to company

[CowbertPrime]

In writing a bot, you are using an automated system to mine a company's website while the benefit to them is questionable - since, of course, you're just mining prices *for a competitor*, there's no chance that they'll see an increase in sales - and if the data is being used by the competitor effectively, that company will lose sales.

In addition, it is important to remember that while the "freeness" of data itself can be debated, accessing data is *not* free. To provide the pricing data over the web - intended as a service to their customers, no less - costs the company money in terms of bandwidth, hardware, software, and human resources. They are clearly spending money offering this service for the purpose of profit. Since the company doesn't know anything about the bot, the consequences on their response times, system performance, and stability can not be predicted. When then system is negatively affected by the competitor, that is tantamount to DoS which has definite legal repercussions.

Obviously, a technical solution would be to block the bot, but when direct cost of even accessing the data is hurting their bottom line, asking for compensation appears to be justified in the light of the future indirect cost. Having to spend resources trying to block the bot would also increase cost.

Re:Cost of data access is damaging to company

[radja]

if a competitor walks into the store, it's not for free either. rent has to be paid. employees have to watch the store. in short: bullshit. if you can't know a competing price, you cannot compete and the free market will cease to exist. everyone should be allowed to know all prices for all items..

Re:Cost of data access is damaging to company

[CowbertPrime]

yes, but the amount of resources a given competitor can use up by walking into a store and scanning all the prices by hand is relatively low compared to the number of legitimate shoppers in the store. Over the net, however, hammering a server is considerably different. Also rent and employees exist at a fixed rate (the monthly cost of rent doesn't increase with increased number of visitors, the infrastructure isn't taxed, the number of employees per shift stands at the same number whether there is a single shopper in the store, or a few hundred), whereas this is not true on the internet, especially if the infrastructure is getting heavily taxed by a webbot compared to "normal" access.

some sites may prevent this from happening

[Anonymous Coward]

Many years ago I was working for a book store. We didn't have pictures for all the books, so I went to amazon.com to download the cover pictures of the books, and did so succesfully for about 4 hours. After that I noticed our IP address being blocked from accessing amazon.com.

JEEBUS......

[_ph1ux_]

EXACTLY what I have been saying for a time...

there should be this exact type of service. actually what i was thinking of was a service that you would "bid" on an object - say only for items valued over 500 bux.

Then that service would scour the NET, ebay and every other normal avenue looking for the lowest possible price for the purchase you want to make. then buy it... and you would pay a service fee % based on the value of the item and the obscurity etc...

but all in all - attempting to make it as automated as possible so that you still really get a good deal on the item.

hello USPTO.

Prices are a fact...

[Kris_J]

Prices are a fact and therefore should not gain any protection offered by the copyright, trademark or patent systems, broken as they are.

If a bot activates a click-through agreement, does anyone hear it fall?

Re:Prices are a fact...

[agedman]

They may be a fact, but the terms and conditions of the site may restrict you from scraping.

A few years ago I went into a (real life) gocery store and was writing down prices on a notepad.

I was asked to leave. When I protested that this was just for my personal info (it was), they said that didn't matter. When I said that I could buy the stuff and get the info from the tape, they said fine but put down the notepad or leave the store.

I didn't push it (after all, it's their property, their rules).

Please tell me how to mine for prices on the Web

[Taco Cowboy]

Can anyone please tell me how to set up my machine on mining prices for any selected item on the web ?

Thank you !

The reason why it's different

[Anonymous Coward]

Quoth the poster:
"Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually?"

Ask yourself why you're being paid to write a scraper instead of being paid to walk into stores and record pricing manually.

The answer is this: because using a scraper robot is faster, cheaper and easier to repeat systematically. Using a scraper makes it possible to check competitor pricing with a scope and speed that is so impracticable in the brick and mortar world that it hasn't been necessary to prohibit it in the brick and mortar world.

For airline prices, there is "creativity" involved

[Anonymous Coward]

"A modicum of creativity" is part of the standard test of copyrightability. Telephone books are not copyrightable on this basis (see Feist Publications v. Rural Telephone Service, 499 US 340 (1991)).

But airline prices probably are.

Calculating the price of an airline ticket is a difficult process, part of the reason travel agents exist and make use of sophisticated computer programs to calculate prices. Compare 5 different web sites' prices for a complex ticket and you'll get 5 different numbers, often differing by thousands of dollars -- for exactly the same flights. So it would seem that creativity is involved. Certainly one could argue that the selection of what tickets to display on a web site like Orbitz is a creative process.

On this basis, I'd say airline prices are copyrightable. And all the major travel web sites specifically prohibit robots in their terms-of-use legalese.

So I think that screen-scraping travel web sites is a very iffy proposition, legally. Courts seem to agree: American Airlines was granted an injunction prohibiting FareChase from scraping them and selling their prices to competitors.

Actually I had this explained to me

[lingqi]

It's not even an old Joke - I believe it was a quotable thing said by, probably, Micron CEO a while back.

This, if nothing else, shows how thin the DRAM margin is, and why RAMBUS trying to skim 2% *gross* will never fly.

anyway, the point is that by pushing out a large volume, they keep their lines running, which means that they can actually make SOMETHING off those lines and get some kind of profit because the cost of running them things through arn't so damn high. This can be demonstrated by how 512M SODIMMs are 135 at crucial while the 1G SODIMMs are 1000... How much you wanna bet the 1G ones are the real money makers?

That, and you are grabbing market share.

to be honest I am not 100% sure how this works out, but by keeping the lines oiled and running, if it's nothing else, it's preventing the company from losing the *real* big bux.

What is a "scraping robot" ?

[Anonymous Coward]

Can anyone tell me what is a "scraping robot"? Where to find them? How to use them?

Thank you !

Mod Parent Up!

[FFtrDale]

This guy (?) is (non-redundantly) describing an important influence on the phenomena being discussed here. Also, the smartest guy I ever worked with couldn't spell either {g}.

Nonsense - it's Advertising!

[FFtrDale]

Contrary to your point that

while the "freeness" of data itself can be debated, accessing data is *not* free. To provide the pricing data over the web - intended as a service to their customers, no less - costs the company money

these are not private data. They've put their prices on the web in order to convince people to become their customers, and that's called Advertising . Advertising costs money; it always has. There's no "two-drink minimum" on the Net. Whether the company knows everything about how the Net operates is irrelevant.

Advertising costs what it costs, and the Net behaves the way it behaves. I'll laugh if I ever get sued by a grocery store who sent me an advertising flier in the mail, because it cost them money and I didn't buy anything. The Net is a public forum. That's why it's attractive to commercial entities.

Re:Nonsense - it's Advertising!

[CowbertPrime]

Sure, it's public, but if some competitor were to walk into your grocery store and carried off the majority of the advertising fliers, and used those fliers for their own negative advertising purposes?

Re:Nonsense - it's Advertising!

[FFtrDale]

The majority? No; that's removing the physical objects that I've had printed to give to a whole bunch of people; it's effectively stealing the advertising dollars I've spent, because such an action makes a whole bunch of the fliers unavailable to my customers. It's the functional equivalent of simply destroying one of my displays. If they come in and take one, however, and then they make whatever use that they may of all the information it contains, then that's an expected part of advertising. It's always happened. There's no legal way to make something available to the public while denying it to my competitors. They're part of the public. All I can say is, "one to a customer," and I'd better treat the competitor as if he were a customer as long as he's in my store. Anything else has too great a PR downside. If I recognize him, though, it's fair game to point him out and crow to all within earshot, "Hey, look! Even my competitors shop here!" I'd love to run a commercial showing him in my store, too.

ATTN: SEE A REPLY TO SOMEONE IN THE FUTURE

[MegaHamsterX]

Please read the parent to understand this post, I am in fact talking to someone from the future, isn't that cool

Let's go over your post so I can show you just how wrong you are.

You've used you visor in Best Buy...etc.. hundreds of times for 6 or 7 years, damn dude, did you set you clock wrong, or are you writing your post from the future, or were you in the future when you aquired your visor?
Being that Handspring was formed in 1998 and the products were not rolling until 1999, oh so you have infact misstated a fact, future traveler.

Does it work like the classic HG Wells Time Machine, or is it more like Back to the Future, well of course the original one, everyone knows time machines can't fly!

I take it the hundreds of times you've used your palm (ahem handspring visor, to be more precise)is correct. Wow! Security never harassed you, I mean you must have picked some exceptionally busy stores to case, oh um, ya see when I was in retail that's what we called it when someone would wander around the store taking notes as they go, SEVERAL TIMES A WEEK. I guess Boston is the place for shoplifters and retail shrink.

NEXT!

Well to be honest with you, calling me or my friends liars would most definitely result in an altercation(hypothetically speaking), this is one of the nice things about slashdot, we can have an argument and you can hide.

Hmm, the next one that would result in an altercation (hypothetically, of course) is being a smartass, but you're not here and I haven't a clue where you are, eh. But in any case when you are asked to leave a private establishment, you do. Failure to leave when asked constitutes trespassing and you will be arrested. Ever been in jail, I highly doubt it as if you had ever seen the inside of a squad car you wouldn't make stupid comments about doing something that would constitute arrest.

Raising hell makes you look like an idiot, I did it when I was younger, but it didn't produce a solution, so I stopped, but I see you're all for the flail your head against the wall approach to life. At least it gives your neck muscles something else to do when you're away from your day job.

KARMA to burn, karma to burn, yes it's worth it!

Re:ATTN: SEE A REPLY TO SOMEONE IN THE FUTURE

[MegaHamsterX]

Well, at least we know the moderators are still awake and working, but overrated, jeeze, that just isn't right, I expected flamebait and got overrated, but moderators will be moderators, myself included. :-)

Re:ATTN: SEE A REPLY TO SOMEONE IN THE FUTURE

[Anonymous Coward]

It's only because Slashbot moderators fear metamoderation because of their unfair mods and use overrated with impunity to enfore the group think here.

Re:ATTN: SEE A REPLY TO SOMEONE IN THE FUTURE

[Anonymous Coward]

calling me or my friends liars would most definitely result in an altercation

You and your friends are LIARS.

I'm not too worried about an altercation with some dorky slashbot, believe me.

But they don't want you to do it to them!

[r39525]

This is from the Google Terms of Service:

Personal Use Only

The Google Services are made available for your personal, non-commercial use only. You may not use the Google Services to sell a product or service, or to increase traffic to your Web site for commercial reasons, such as advertising sales. You may not take the results from a Google search and reformat and display them, or mirror the Google home page or results pages on your Web site. You may not "meta-search" Google. If you want to make commercial use of the Google Services, you must enter into an agreement with Google to do so in advance.

No Automated Querying

You may not send automated queries of any sort to Google's system without express permission in advance from Google. Note that "sending automated queries" includes, among other things:

• using any software which sends queries to Google to determine how a website or webpage "ranks" on Google for various queries;
  
• "meta-searching" Google; and
  
• performing "offline" searches on Google.
  

Please do not write to Google to request permission to "meta-search" Google for a research project, as such requests will not be granted.

Secondhand legal advice

[yerricde]

Slashdot is not a replacement for a lawyer.

Slashdot is useful to get a sense of what the legal landscape is like. Some comments are to the effect: "I am not a lawyer, but my lawyer told me this." Or "I am not a lawyer, but here is the statute [cornell.edu], and here is how a court has interpreted it [eff.org]." When you do see an attorney after reading the comments, you don't have to wait for the attorney to explain the basics. This saves time, and time is money, especially at the typical copyright and trade secret specialist's rate.

That said, you're right about one thing: anything you read on Slashdot is not legal advice.

Wal-mart is world wide not just redneckvillie

[DrSkwid]

Wal-mart bought out one of the UK's biggest supermarket chains - Associated Dairies (ASDA)

I read at the time of the merger that this made Wal-mart the worlds largest company by square footage of shop floor.

m

Re:Wal-mart is world wide not just redneckvillie

[Zerth]

Is, unless they change your name after you die now.

Re:Wal-mart is world wide not just redneckvillie

[Zerth]

aww crap... nitpicking a nitpick and I reply to the comment below it.

P2P

[upt1me]

P2P networks mine my data of mine all the time.

Re:Please tell me how to mine for prices on the We

[ajs318]

Pipe lynx -source http://somewebsite.co.uk through appropriate grep, sed and awk filters, or a Perl script, or Python if you think Perl isn't trendy enough. Then edit your crontab so all this happens by itself. Easy enough :-)

at best buy

[standsolid]

Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually?

well for one thing, you can't be sold service plans, monster cables (you need them), msn internet access... and hell why not get that surround sound system and plasma with those prices you are obtaining?

all joking aside that is the difference -- by having a bot that scours the internet, you are not being sold on that service -- which is why the page is there. it can conflict with usage agreements

/me shrugs. i think it's lame and i would do the same thing

GOATSE ALERT!!!

[jazman]

...for those at work who open it, think "can't be bothered to try that one", then go off and get a coffee.

Guess who else that just happenned to :-(

Wasn't this supposed to be a market economy?

[thomas_klopf]

The thing I find disgusting about companies that try to prevent price comparisons is that this is in violation of the very principles of the market economy that have made them so wealthy in the first place. Market economy theory dictates there should be competition, and with that competition comes lower prices because people choose the lower-priced goods over the higher priced ones. Lower prices, however, cannot be found if we do not allow people to compare prices. In sum, companies simply can't have their cake and eat it, too. Allowing people to compare prices is not just a priviledge companies give, but it is a right these people have.

Moreover, when you put your store on the Internet, you take advantage of the possiblity of millions of people looking at your wares, and all you have to do is put a webpage and database out there. You MUST be willing to accept the fact that with this conveinence, others should be able find ways to "conveniently" look at your data. I will not accept someone dictating to me how I can look at their data - you're the one who made it public, right? Again, these companies can't have their cake and eat it, too...

rant rant rant :)
-tk

AA.com v FareChase

[FredEFF]

Since you mention that you may be building a screen scraper that gathers airline fares, you may be interested to know that American Airlines has already sued (and won a preliminary injunction against) a software company that built a tool that does much the same thing. The case is American Airlines v. FareChase, and was discussed on LawMeme:

American ... sued FareChase in a Texas court (American is based in Dallas, so that's its home turf) and got a preliminary injunction against FareChase's screen-scraping practices. The court decided that the screen-scraping constituted an "interfer[ence] with American's personal property," also known these days as a trespass to chattels. The court also noted that FareChase's actions might be a criminal violation of Texas Law, which states, "A person commits an offense if the person knowingly accesses a computer, computer network, or computer system without the effective consent of the owner." Tex. Penal Code 33.02(a).

The injunction order is posted on EFF's site, and the briefs are posted on Bag & Baggage.

Insurance pricing and competitive advantage

[Malfourmed]

I used to work for a major insurance company. One of that company's major advantages was the technical ability of its underwriting/risk assessment/rating (actuarial) area. Ie, the business invested a lot of money in hiring very smart people and giving them nice computing toys to examine different risk factors to formulate the likely claim rate for a certain mix of factors (eg age, location, etc) which was then used as the basis of pricing insurance products.

Getting product pricing right is very important in insurance because unlike most businesses you incur most of your costs (in the form of claims) after the customer has paid you their money, not before, which is more typically the case.

These prices were not published, partly because the pricing ended up so complex to describe and so personalised to the individual customer that it would have been difficult to come up with a way of presenting it simply, and partly because the pricing structure embedded one of the company's competitive advantages, ie its underwriting function.

What we found was that at least one of our competitors organised a campaign to "reverse engineer" our company's rating book by paying a bunch of people to ring us up and get quotes. Ie, in the case of car insurance, they'd ring up and say they were a 25 year old male living in suburb X driving model Y. Then they'd ring up saying they're a 25 year old male living in suburb X driving model Z . Etc.

This way they'd be able to consistently match or beat our pricing merely by investing in 500 phone calls, rather than by investing in millions of dollars in brains and gear.

So this would be at least one instance where data mining for product pricing would be, if not necessarily illegal, then certainly mercenary.

(Note that merely matching another insurance company's prices won't necessarily always help you because there are other factors that affect your cost:income ratio, but certainly this tactic has given me food for thought over the years.)

Re: Is Data Mining Illegal?

[TheQuietOne]

I would doubt very much that price harvesting is illegal. For a comparison I would refer to the case a few months back where a large company attempted to use the DMCA to protect its price information. From what I recall it was deemed that information published on the internet was in the public domain and therefore freely available. It is why the best buy bargain sites are still around. TQ1.

Yeesh.. give it a rest. (OT)

[poptones]

I'd say yeah cuz your mom's a better lay ;)

But before that I'd say it's uncanny how so many people think it's just fine to throw around terms like "redneck" when they'd never think of saying other things - things like "OH, only niggers watch BET."

I realize it's expecting a lot, but I really thought the people who frequent /. were more evolved than that. For those of you who have apparently slept through the last decade, Mississippi ("redneckville") begat one of the largest telecom companies (and wall street scandals) in history.

Yeah, there's still problems here - but none I've not seen in just as grand a fashion in NY or Michigan or California (What do you call a redneck from california? "Officer.") Meanwhile, Worldcom owned (at least as far as wall street was concerned) "the internet" - and Wal-Mart owns retailing on a global scale. Both these companies came from the deep south. Pretty damn good for "a buncha rednecks."

Re:Yeesh.. give it a rest. (OT)

[Anonymous Coward]

When "nigger" is exclusively a label for observed voluntary behavior, I'll start saying it casually.

Telecom is based on interfering with the free market through absurd regulation and backroom cartels, and dominated by "butt-crack economics" (cue usual union/mafia intrusion). Wall Street is hardly a judge of intellectual achievement or even fair dealing. What noteworthy good things have smart people done there?

Re:Yeesh.. give it a rest. (OT)

[poptones]

Since when is "redneck" exclusively used as a label for "observed voluntary behavior?"

Your notion of telecom is absurd. Without those "evil nasty corporations" building out the internet it would still be little more than a toy of the "elite" in government and universities. They don't do that shit for free, you know.

Have you wired everyone in your neighborhood yet? Why not?

Re:Yeesh.. give it a rest. (OT)

[Anonymous Coward]

[quote] But before that I'd say it's uncanny how so many people think it's just fine to throw around terms like "redneck" when they'd never think of saying other things - things like "OH, only niggers watch BET."[/quote]

As soon as folks in the South stop assuming that anyone with a non-Southern accent isn't from New York City -- no, then it must be Boston -- no!??! -- and then following that up with the assumption that (i) that person's rude, (ii) is pushy, (iii) is bigoted against Southerners, (iv) knows or cares about the events of the 1860s.

I grew up in Connecticut, and went to college at NC State. It was the same crap over and over again. Many southerners use Yankee like they used to use Nigger -- and it's just as wrong.

Here's the difference though -- Yankee and Nigger are all encompasing. If you're from north of (Maryland/New Jersey/Virginia) you're considered a Yankee, regardless of action. If your skin is dark enough, you're considered a Nigger. However, being a Southerner != being a redneck. Having a mullet, less than a full head of teeth, relating to the Jerry Springer show, living in dodgy housing often on wheels, etc. gives you that redneck label. There are plenty of rednecks in upstate NY (more often known as hicks), or any other state in the union. A person can stop being a redneck -- a bath, a haircut, a dentist, a new wardrobe, a few AA meetings, and a new abode later, he's a new man.

Dude, its slashdot, the land of one sided solutions and hurting oneself jumping to conclusions. Obviously not only rednecks shop at Wal-Mart. That being said, the density of rednecks at a Southern Wal-Mart is only surpassed by gun shows and monster truck rallys.

Re:Yeesh.. give it a rest. (OT)

[Lt Razak]

Are you kidding me? I'd love the internet to "only" be the green screen hooked up to a MUD and IRC that I had back in college.

The corporation will continue to hump the internet like a dog in heat until it becomes as regulated, watered down, and crapped out like the NBC news.

Re:Yeesh.. give it a rest. (OT)

[Anonymous Coward]

Since when is "redneck" exclusively used as a label for "observed voluntary behavior?"

Evolution doesn't mvoe fast enough for cheap beer, mobile homes, and the KKK to be genetic.

Your notion of telecom is absurd. Without those "evil nasty corporations" building out the internet it would still be little more than a toy of the "elite" in government and universities. They don't do that shit for free, you know.

They aren't evil because they charge money. They're evil because they control who may compete with them (so they can charge vastly more money than needed to do the job while preventing most disruptive innovation) and now they want to control how their own customers use the only services they grudgingly provide.

Have you wired everyone in your neighborhood yet? Why not?

No, I don't want to go to jail. ILECs (along with cable TV providers) have a monopoly on laying wire for communication. Did you think CLECs wanted to rely on adequate service from their most ruthless competitors?

Re:Yeesh.. give it a rest. (OT)

[poptones]

Wow, I didn't know there were an luddites left.

I'll take google, thanks - and MSNBC and BBC and FOX and the NYTOnline and /. and WIRED and Sourceforge and...

The internet will only become as watered down as (allegedly) intelligent and skilled people like "us" allow it. It's really sad so many in this industry seem to think the borders of the internet begin and end at the .COM domain.

the can block/firewall you ...

[Lazy Jones]

So, no specific legislation is needed. ;-)

-LJ

[crawling the web since '97 to gather pricing information - check out geizhals.net]

Price-Bot for Ebay

[Chokma]

The same question occured to me...

I was writing a bot to collect all ebay auctions of a given category [ie, DVDs] and build a database to get the current market price of any DVD and archive the auction texts for more than the ususal 90 days [along with the option to detect false bidders using multi-accounts].

Of course, there would be some filtering for the top and lowest prices [to remove the $1000-DVD signed by Agend Smith from messing up the Matrix-prices].

Then I saw the Ebay terms of service, which seem to forbid this type of spidering. Ebay would not answer my mails, so in the end I stopped the project *sigh*. Would have been nice to know what kind of price a DVD/CD/Game will fetch on Ebay.

Remember: Never build your business on Ebay or Google without obtaining their permission first :)

hm?

[Anonymous Coward]

"Is Data Mining for Product Pricing, Illegal?"

This sentence no verb, has. :-)

well. kinda.

Re:hm?

[Call Me Black Cloud]

Or maybe:

Has this sentence verb, no?

Other Rule of Thumb to Live By

[Rick.C]

If you fear that it might be the wrong thing to do, it probably is.

Prices are public information...

[Anonymous Coward]

So they don't have ANY kind of copyright protection if they are gathered from public channels of communication or from the shelfs. Of course, if you get the prices from a not yet released price cathalogue... then, you aren't in violation of copyright of those prices, but you are violating trade secrets laws and the like... In any case, just SUE!

Re:Money + Lawyers

[CrazyDuke]

^It has a tendancy to make a great many things legal or illegal that would not be otherwise.

Define a "scraping robot"

[Quixote]

What is a "scraping robot"? A program that downloads data from a website without human intervention? (It sure as hell isn't a robot that primes a surface for painting, it would seem). My point is: modern browsers (like Mozilla) will pre-fetch pages from a site, even though you may not actually view them (and instead head off somewhere else). What about a caching proxy?

The question is, where do you draw the line? The data already is being accessed in an "automated" fashion (by the browser).

Plus, who will speak up for the rights of robots? Who are we to deny access to all this goody-goody data to robots? What if a robot really wanted to travel ?

Airline pricing is different

[themightythor]

What all of the replies thus far have failed to realize is that pricing an airline ticket is not the same as pricing a VCR. As someone who works for an IT company that writes software for the airline industry, I know better. The reason that companies like Sabre and Orbitz exist and are in direct competition with each other is that finding the lowest fare between point A and point B is hard. In this case, it's not the prices that are the secret, but rather the pricing system. For instance, say you're going between Minneapolis International and O'Hare. Common sense would tell you that the lowest fare would be a direct flight. But, depending on the day and other conditions, you might get a cheaper fare if you go through Detroit.

Also, I don't see how the OP's employer hopes to beat Sabre's price if they discover that Sabre is cheaper. The airline industry requires that money collected for a ticket be divied up a certain way: x goes to taxes, y goes to commission, and the the rest is split amongst the actual fare (which is set by the airline). the OP's employer would have to find the fares such that they could actually beat the price. If not, the airline is within their right to ask for the difference.

Re:Please tell me how to mine for prices on the We

[Anonymous Custard]

Set your homepage to pricewatch.com, then train a monkey to type in your chosen product names and write down the prices. Easy!

Hrrmm.... HTTP GET...

[Anonymous Coward]

not html. Did you just guess a little to try and earn some karma?

Do your homework next time, wannabe.

Re:Hrrmm.... HTTP GET...

[Provincialist]

You're right, http. Is the point somehow invalidated by this?

No, No, and No.

[moehoward]

This is clearly illegal if you are some how, some way, making money from the price gathering or providing it to the public. It is clearly protected in law. In the exact case (airline pricing), they are backed up by previous court rulings. It is a no-brainer. Don't compare this to gathering prices on apples at the local grocery store chains.

Don't go there. I worked on similar projects in the travel industry and what you are doing will get a cease-and-desist as soon as they find out about it.

If you see others doing it (priceline, etc.), they are paying for it in some manner. All the airlines have legal means for you to pull pricing info from them (it may be from a 3rd party service). That's where you want to start.

Check those user agreements

[tbase]

Most web sites have end user agreements, especially the bigger ones. For example, here's a snipet from Expedia's:

This Web site is for your personal and noncommercial use. You may not modify, copy, distribute, transmit, display, perform, reproduce, publish, license, create derivative works from, transfer, or sell any information, software, products, or services obtained from this Web site.

"Personal, non-commercial use..." so I guess if you're looking for a business flight, you better not use it. Point being that even public info, or info in the public domain can be protected when it is provided in a special format. While you may be able to publish a companies prices, sucking that info from their web site will most likely violate their terms of use. It's like the old shareware and freeware floppies - the programs can't be sold, and the publisher can't stop you from distributing them - but you can't make duplicates of the disk as a whole, because the compilation and format is what's protected. Look at it another way. Say you manually type 400 pages of parts pricing information into a database or web page for a local appliance dealer. You don't charge them much, because you can sell copies of the database or web pages to 100's of dealers across the country. But when you start making sales calls, all your leads tell you "No thanks, I just downloaded it of their web site". Companies should be allowed to publish information to be used by their customers only without fear of their competition using it against them. Otherwise it's the consumers who suffer from a lack of resources. It's already happening- how many times have you been to a web site that makes you call them to find out who the dealer is in your area? It's because they don't want their competition going after their dealers. Also remember this - forget about the legality for a moment - anything you write will most likely be easily broken by small changes in the format and/or code of the source web sites. If they don't want to (or can't) bother with going after you legally, they could make your life a never-ending nightmare of coding updates. And if your site is successful, you'll be a big red flag in their web logs.

Price Scraping is illegal... according to TV

[PurplePhase]

Even though the other replier says Sam's book says he never did it, etc., the TV documentary I saw on Sam showed that he tried to grab competitor's prices but that once anyone in the store saw him doing it he was immediately (politely) asked to leave the store.

Apparently, to those TV producers at least, price scraping is protected under law. That's why no one can bring a camera into a store and start snapping pictures. Those surveillance cameras aren't there only to discourage shoplifting.

Wish I could find the reference, though. Anyone?

8-PP

Re:Price Scraping is illegal... according to TV

[I8TheWorm]

As a former (underpaid) programmer for Walman Marcus, I can tell you about a couple of company policies. One, is to beat the competitors (what used to be K-Mart, Target) etc... on similar items. There are many stories circulating at the home office about light bulbs selling for $.01 because WM Manager would cut the price, then KM Manager would cut the price, in an almost endless loop. The addendum to that policy is finding out competitor prices. One way is to scour sale fliers. Another is to take a handheld with a barcode reader to zap an item and record it's price. Managers are constantly being tossed from competitor's stores for doing that very thing. It happens to WM Managers, KM Managers, Target Managers, etc...

Re:Price Scraping is illegal... according to TV

[serutan]

You don't have to be doing anything illegal to get thrown out of a store. You can get thrown out of a store for doing anything they don't want you to do in there, like drinking a Slurpee.

Low-tech prohibits this.. Sometimes..

[dimer0]

Try walking into Best Buy with a pad of paper and a pen.. Walk through the isles..

I've done this before - and have been told to remove those items from the store..

Screen scraping is not data mining

[Call Me Black Cloud]

Screen scraping is data gathering. Data mining is looking for trends or patterns in data you already have. Getting the nuggets out of your data to continue with the mining analogy. From this presentation titled "An Introduction to Data Mining Technology" data mining is defined as "The automated extraction of hidden predictive information from (large) databases".

The bottom line is this: when you put this work experience down on your resume don't say you were data mining. Companies looking for that experience will ask you hard questions you don't know the answers to and you will be embarrassed.

Not the pricing -- the timing

[RalphSlate]

There may be precedent for this. eBay was able to convince a judge to bar spidering of their site.

There is another legal concept called "Unfair Competition" which links copyright and facts.

Normally, facts cannot be copyrighted. However, this law seems to kick in when one company compiles and publishes time-sesitive information that it has taken from a direct competitor in a way which "free-rides" on the efforts of the competitor. It is usually applied to news organizations, when one newspaper sends a reporter to Iraq and a second newspaper (perhaps an evening edition) uses the "facts" in the first newspaper's article to publish the very same news.

I could see the instantaneous publishing of all competitors' prices as a violation of this legal theory.

Re:Not the pricing -- the timing

[jkabbe]

Keep in mind that eBay lost their first case against Bidder's Edge's because it was solely a copyright case. Then they brought a conditions of use case that they won.

Robots and Handheld: The Difference

[nalfeshnee]

"Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually?"

Well duh-to-the-nth! It might be a *teensy* bit faster using a robot as compared to, uh, manual input. Hence the word, uh, 'robot'.

<heavy sarcasm>
In other news: "Consumer surprise as Consultant proves buying on the web *can* be faster than going out of your house, taking money from the bank, catching the bus, entering a store, choosing a product, walking to the counter and paying for it, waiting for the bus again..."

ok, ok, you get the point.
</heavy sarcasm>

Nalfy.

Son of Goatse

[Anonymous Coward]

Did you know "The Receiver" actually has a son?

Former CI Professional Opinion

[jlaprise]

As long as the information is available to the public in some fashion, there is nothing wrong with scraping/data mining as described. Both ethically and legally, this methodology would be completely acceptable to CI pros. Check out www.scip.org

Exactly right...but judges are idiots

[siskbc]

You can't really keep them secret, so "trade secret" is right out. It's not a identifying mark

Obviously, you're right. The four biggies (patent, trademark, copyright, and trade secret) all eat it. So why does shit like this happen? Because the judges in these cases half of the time have never used email. Their aides do it for them. Computers are nebulous devices that other people use. When you get right down to it, they don't have the experience to see the obvious analogies that you made and we all understand.

In 20 years when we get rid of these dipshits, maybe crap like this won't happen. Might be too late by then, but oh well.

Protecting against scrapers

[nalfeshnee]

Two things that come to mind (depends on the site and what they currently support technically browser- and serverside):

-- use front-end technologies that prohibit or at least inhibit the workings of server-side scraperbots. examples: flash, javascript.

-- use sessions to control how often a given client can access prices, e.g. 'a 10 and you're out' rule: most 'ordinary' users have no need to view a certain page of prices more than X times in a browser session. here, cookies provide even more protection since some scrapers won't be set up to handle them.

both systems may have their drawbacks (no flash allowed), weaknesses (against sessions i can simply make multiple logins), but i've incorporated similar systems on sites for clients with prices that need a sensible level of protection (i.e. one shouldn't be able to grab the whole damn price list with a one-page GET e.g.). guarding against SQL injection is also something which is often forgotten.

Cheers,

Nalfy

Re:let me ask you this.

[slamb]

You would then need an additional comma after "Data".

Follow your own advice. If you put a comment after the subject if your sentence it'd read:

You, would then need an additional comma after "Data".

You put a comma after the name of the person whom you're addressing (listener? addressee? target?) if it's at the beginning of the sentence or before if it's at the end.

The subject gets no comma love. If he's also the one being addressed, he can get this:

Data, are you mining for product pricing?

But "you" is the subject.

Related question...

[Anonymous Coward]

Hypothetical scenario:

a client hires you to write a scraper to grab prices and other visible data from a competitor's website. BUT: in a password-protected section (for that competitor's registered customers). the client you are working for happens to share many of the same customers, and thus obtaining access is as easy as receiving login information from a friend of a friend of a friend...

as the consultant, you have no actual knowledge about where the login comes from, other than some reasonably educated guesses that they are probably from a chain of employees that leads to one of these customers, and you write a scraper, and then execute it to build a db to mine.

if you get caught, is anything about this illegal, who would be at fault in this scenario?

let's assume that perhaps the terms of the site say something about the data being private.

read the cases before commenting please . . .

[declana]

EF Cultural Travel does not stand for "you cant scrape". It depended on one key thing - Explorica was started by ex-EFCT employees that were bound by a non-compete agreement. The circuit court rejected the argument that there was any generally implied term that forbade a company from scraping. Note that the circuit court did say that a TOS that restricted scraping may be enforcable. In short, check the TOS' of the sites you are proposing to scrape. Incidently, in a seperate appeal, the injunction against the programmers were upheld based upon their knowledge of a previous injunction against the employees (not based upon a general "thou shall not scrape rule).

And then destroy web standards

[hellfire]

Set the prices as GIFs, then:

1) make it take too long to download webpages
2) Make it impossible for physically disabled individuals to use your webpage, thus possibly losing business.
3) Make it take forever to actually come up with the GIFs and link them to the appropriate flight/trip/fare. Imagine the prices.
4) make it only marginally harder for the scrapers who then find out that either you have a pattern to which GIFs are used for which digits in your numbers for prices... OR
5) Show everyone how much money you are willing to waste by encoding every single fare for each flight into its own GIF with a unique identifier and thus blowing your investment.

Re:And then destroy web standards

[x00101010x]

Yeah, i forgot on 2.
As far as 1 goes, a few more gifs isn't going to be noticed in the bloat of ecommerce sites out there today, which factors into the title (lots ecommerce titles take a big dump on web standards and are non-functional/ugly with anything other than MSIE or NS4.79 and below because that's all the webmonkies get taught at their local community college).
On 4, the big issues isn't the scrapers, it's that the scrapers will use OCR which may have a higher error rate due to the scramble/fuzz images which will cost in proofreading. Most corporate types require anything input by OCR to be proofread, which costs money. The code for a better scraper isn't the money eater, it's the proofreading of the OCR data.
3,4,5: Properly done the images could be spat out pretty quickly with less backend lag than it takes to look up the prices and generate the html page anyways.
Anywho, not like I even like the idea anyways, i think that if information is posted on the web it should be free game (with the exception of copywritten origional content). I really only thought up the idea because i read the article after a 4 hour coding spree on a stupid web project for a big stupid monolithic company and it sounded like the sort of thing they'd ask of me (Make it scraper proof!) =P

Future of the internet?

[IWannaBeAnAC]

under what circumstances is price harvesting off of the internet not allowed?

The answer to that is, "when influencial businesses can dictate the law to their own ends". I am sure that a great many (non-internet) businesses would love to ban people from walking around their store with a notepad jotting down prices.

The promise of the Internet was that it would make everyone equal: the vastly increased flexibility in how online salespeople can rip you off was supposed to be counterbalanced by the consumer's (or groups of consumers) ability to counteract this by actively extending the functionality of the internet by policing these actions, an important component of which is keeping track of prices/goods/services offered by various merchants.

This 'ideal' absolutely must be enshined in law (probably international law is the correct forum), otherwise it will be whittled away in some juristictions where business has a controlling influence on the legal process (read: USA), and then pressure on the rest of the world to conform.

It is a failure of humanity that we always choose the most optimistic outcome for an upcoming technology. The 'promise' of computers and automation was the paperless office, no more menial tasks resulting in increased leisure time for everyone. The reality is more wasted paper than ever before (printing another copy of the whole document, just to eradicate a typo, is but a mouse click away), and a smaller fraction of the population working much longer hours, while the rest suffer unemployment.

In hindsight, both of these effects can be seen to be at least as likely as the 'promise'. A proper analysis back in the days when it might have been possible to make a difference, might well have shown that it was in fact the far more likely outcome.

The same mistake has been made with the Internet. While the 'promise' of equality and empowering individuals is a possible outcome, the underlying technology also allows unprecedented restrictions on freedom. Given the track record, which do you think is more likely?

What will be required to reverse the course, if the 'promise' doesn't come to fruition? Are there any comparable examples from history?

Of course its legal....

[gte910h]

By putting it up on the web without a password, they are PUBLISHING the information. This means all things you can do to a store cirular's data, you can do to web site published information

Re:er, BS:Re:Best buy is a really really bad examp

[Oliver Wendell Jones]

There was an incident of a man being arrested in a Best Buy (for trespassing, I believe) after being asked to stop writing down prices for large screen TVs. The irony of it, and I explained this in a post a long time ago, is that I used to work at Best Buy, and on weekends, we were asked to bring a non-blue shirt with us to work so that we could go incognito to the local electronics store (H.H. Greggs, before we had a Circuit City locally) and use a micro-cassette recorder to 'steal' all their prices so we could mark down the items in store to compete. Now they're telling people that they can't do what they themselves do (or did). Reminds me of a local story about a guy who was wearing one of those fancy NASCAR leather jackets with either Home Depot or Menards as the sponsor of the team going into the store that wasn't the sponsor and who was celebrating their grand opening (i.e., wore a Menards jacket into a Home Depot, or maybe it was the other way around) and was asked by management to leave because they thought he was a spy from the other company. Made the local paper when it happened...

Not necessarily illegal....

[MerlynDavis]

But I used to be the network admin for an e-tailer, and we discouraged people from webcrawling our site. Not because our pricing was proprietary, or anything, but for the simple fact that I had to pay for all that damned bandwidth. Sure, one crawler doesn't add up to much, but we had as many as 20 crawlers at any one time, some of them obviously on T1+ links, and using every ounce of speed they could.

We did provide pre-formatted price lists for those people who asked, prepared daily, and available via FTP. That way, instead of having to wade through our HTML code and try and locate the pricing, they could get an SQL data file, or maybe a CSV file, or any one of a dozen formats...

Since when do 3rd-graders take microeconomics?

[Anonymous Coward]

he he...

Antitrust Laws May Be The Root of The Problem...

[thosss]

I think much of this discussion may have missed this point:

According (or most likely legally inferred) to section 1 of the Sherman Antitrust Act, a company isn't permitted to get prices directly from its competitor.
--
Exchanging Information with Competitors
It is important to avoid the exchange of sensitive business information with competitors without guidance from legal counsel. The exchange of price lists or prices charged to customers may violate Section 1 of the Sherman Act even though there is no agreement to fix prices, due to the natural tendency that such conduct will produce uniform or stabilized prices in the industry. Of course, you must obtain this information from some source in order to compete. But you should be able to show that you did not obtain it directly from your competitor and that you did not make your lists available to competitors.

http://www.hhrmlaw.com/antitrust.htm
--

I don't know how extensive the links inbetween the company and the competitor need to be, but if the company is using the program (that you made) I would think that is more direct than indirect. If you do it and give them the prices, who knows.

Wrong prices with Froogle - but I don't understand

[harmonica]

Froogle detects a certain camera as USD 499, although it really costs 30 USD less, which you can only see if you put that camera in the shopping cart. Here's the explanation - what's wrong with the price being an advertisement? Maybe someone can shed some light on this for me.

Data-Mining

[anythings-possible-b]

21:42 15/5/2546

TPOIC: DATA-MINING

i don't think it illegal. free info forever. but instead, now, beating their price, they should maybe
extend their services. i assume, it's something with travel, so who cares if it's cheaper, if
the other guy has the way cooler tours?
i do data-mining all the time.
how about this: i check what tour they offer and do it on my own ...
maeh: i'm not allowed to LOOK! what?
so why are they posting their prices on the web? hire a secretary. call 1-800-travelling, or something.
-
it might be, that this fight between the two companies is a "personal" one. maybe they used
to be one comapny until the TWO boss got in a fight and they seperated and started their own
business. maybe it's personal ...
-
i would like to know how the found out they were data-mining?
-
so in the future all the computers are doing the work,
but who is acctually earning money to be able to spend it? human nil, computer infinity?
-
i don't get it ... i am buying a ticket from an airline, which sells it to another comanpy first.
really why is it cheaper? if i go to the airline direct it costs more ... scratch scratch ...
oh and why aren't the airlines buying ORBITZ and the other online ticket reservation services?
why haven't hey got their own?
"something-bulk" garenties a full airplane ...
maybe in future we just need ONE powerplant, ONE airplane, ONE car, ONE house, ONE shower for whole humankind.

Re:Even walk-in data mining will get you in troubl

[Anonymous Coward]

I remember that story! Ironically enough, it was a Best Buy, not a circuit city. And they had the guy ARRESTED for trespassing because he refused to leave when the manager told him he had to leave.

exactly

[MemeRot]

it's not a contract if you don't agree. you read the license, you say 'no'. you then proceed to browse the site. it doesn't even have the power of a click through.

No fair!

[aziraphale]

If I'm not allowed to use a computer program to automatically make web requests, then I don't see why they should be allowed to use a computer program to automatically respond to them. If they want the convenience of having a computer sit there answering anybody's query of 'how much does this cost?', then they should not be surprised when somebody writes a computer program to simplify the process of asking a number of different retailers what their prices are.

So, if they don't have a person sitting at a terminal in the server room personally typing in all the HTML for each web page, I'm not going to type in URLs personally. Those terms and conditions are, of course, available to anybody who cares to look at my web site, so I don't see any reason why websites won't comply with them.

More seriously, web site terms and conditions are always written in appalling pseudo-terminology that talks about allowing people to 'access' the web site, but prohibiting them from 'downloading' content from it, or 'storing' it. Quite how one accesses content on a website without downloading and subsequently storing it (if only in my local computer RAM) is beyond me.

There's an implicit assumption here that using a web browser to generate and send your HTTP requests is okay, but any other program is not; quite how the border between browsers and non-browser user agents is drawn is completely ignored. The terminology used in the legal documentation should at least, surely, bear some relation to the terminology used in the HTTP RFCs. For example, I'd respect a web site whose T's and C's (which were perhaps available from a URL identified in a header in every HTTP response issued) said something like this:

'You may submit HTTP GET and POST requests to port 80 of the server provided they are correctly formed according to current IETF RFCs; HTTP responses transmitted by this web server must be interpreted in strict accordance with the prevailing IETF RFCs. The content of any response issued by the server is copyright this website.'

Frankly, any attempt to require any more than that on the part of your users is a futile effort on the web.

Ah well. I can dream.

How could they be googled?

[MemeRot]

If not clicking the click through actually prevented you from accessing the site, then the googlebot wouldn't be able to get in.

Amazon.com

[tpugh00]

Amazon also marks certain items so that you cannot see the price unless you add it to the shopping basket then view your shopping basket. This makes it more difficult to automate mining for prices for some of the products. If companies see this type of price mining as an issue, I would not be surprised to see companies start putting their prices in bitmaps -- similar to some sites that ask you to retype a value displayed in a bitmap in order for the submit to work.

My two cents

[jhines0042]

If the prices are made avaiable by one means then that same means should be allowed to view them.

So if the prices are available to the average customer by allowing them to walk into the store, browse the aisles, then leave, then I should be allowed to do the same.

If the prices are available electronically then I should be able to read them electronically however I choose.

If I have to get the prices by speaking to a salesperson then that is the means that I need to use.

This is fair because the cost (time, money, inconvenience) is equitable across the board. If you want to go cheap on me by not putting a salesperson with every piece of merchandise then I'm allowed to go cheap on you and just look at the sign.

Price Fixing

[Anonymous Coward]

I think this has more to do with Price Fixing laws than anything. Especially of the gathering of competitive information is used to raise prices in cases, rather than lower.

Personally I think it's a dangerous business practice, sooner or later your competitors will begin to do the same and your whole industry will suffer severe margin errosion.

Hey, that's what downsizing is for right?

Thought this was settled

[k12linux]

I thought a court case had already been won which said basically "If you put it on the net, it isn't private/secret information anymore." Wasn't it about a company's report that was "leaked" when they put it on their web server but didn't intend people to read it yet?

Rebates are real enough

[87C751]

I've received many rebates over the years. And yes, I paid sales tax, but that's not the purchase price. It's just the local gangsters taking their rake.

Use the Client ID against them...

[illumin8]

What if you change the client id of your data mining spider to something like "By granting me access you hereby waive all rights to privacy. Any data sent to me is mine to do with as I wish."

Then, if their web server still serves up the page to you it would seem to me that you would win a court case if it ever came to that.

ASK A LAWYER!

[Anonymous Coward]

Another example of another nimrod seeking legal advice from /.ers instead of doing the smart thing and asking a lawyer.

Re:ASK A LAWYER!

[wessman]

Well, as soon as you remove the rod from your anonymous a**, you can step back and see that this question was not about seeking legal advice from /.ers ... instead, it was about sparking a conversation among web developers, programmers and consultants, and more importantly, pointing out to our community how ridiculous access, authorization, copyright and licensing statutes have become and how they threaten consumer rights and innovation.

Licenses are not golden

[padukes]

It drives me crazy how everyone thinks that you can put anything in a license and if someone accepts/clicks-through/whatever then it is binding. If I put in my license that you are not allowed to breathe while viewing my site, and I have proof that you did so, I will not win a case against you. Even if you accepted that license. Even if you wrote me a letter telling me how you agree to every clause. Even if you specifically mention the breathing clause and how great you think it is, and how you hate people who breathe while viewing web sites. Just because I (or my lawyer) puts a restriction in a license, does not mean that I have the right to make that restriction. P

Bandwidth costs..

[Anonymous Coward]

Access to websites costs money because the bandwidth and systems aren't free.

On that basis alone, I believe they can restrict your access. Minimally, they can tell you to cease and desist.

When I worked at a major E-tailer, over 10% of our traffic was due to robot activity. That 10% utilization of the systems represents more 'hits' than the vast majority of websites get.

The bandwidth and systems cost to support that 10% were *substantial*.

I currently monitor the performance of 20 websites from my home broadband connection. This amounts to around 100 Gigabytes of traffic per month. At some point, I expect to be told by a site to *STOP*.

As for your service, for some time I have wanted to do an 'Orbitz watch' that would monitor specific route prices on a regular basis. Essentially, telling subscribers what they can reasonably expect to pay if they time it right and then telling them when to buy..

robots.txt

[mrmike37]

IANAL, but I'm sure there is a good reason why all search engines I've run across allow your site not to be spidered with robots.txt present.

Current Research Project

[AustinTSmith]

Currently, I am conducting some MIS research at the University of Arizona that does just this: mines information from a website and stores it to a database. I am mostly concerned with the pricing information, however I do not believe it to be illeagal to mine such info. I am thinking of some commercial applications I can use the information for.

If the information is posted on a webpage then it is public knowledge. Hence mining of this information is perfectly ok. Look at how a search engine crawler works.

Some stores frown upon such things

[mcheu]

Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually?

There are some stores that actively discourage people from bringing in notebooks and PDAs to record their pricing information. While I wasn't working for anyone at the time, this happened to me about 2 years ago at a local Future Shop (division of Best Buy Canada). The experience suggests that some stores are very protective of their pricing info. I don't know to what extent this is legal.

I was shopping for a CD burner and a joystick at the time for Christmas (and maybe a little something for my own rig). I don't like shopping online and wanted to know what was available and at what prices locally, so I went around to stores gathering this info. Most didn't have a problem with this. One of the stores I stopped at was Future Shop. I was in the middle of writing down the information when I was suddenly surrounded by 3 salesmen in business suits asking me what I was doing. I told them, and one of them said I couldn't do that. Another fellow in a blue suit IDing himself as a manager came over and told me the same thing. He then said he wanted to know who I worked for, and what to see what I had written in my notebook. I told him I was shopping for myself and refused to hand over my notebook. I left without being challenged. Weirdest Christmas shopping experience I've ever had.

Since the manager joined in, I'm guessing the gestapo stuff is (or was, haven't been back in a while) store policy. It certainly didn't make me want to shop there again. The only thing I can think of is that their competitors often sent "operatives" into the store to collect pricing info and they tired of it. The only thing is, if that store didn't allow price comparisons, how the heck else can you find out what is available locally and at what prices?

=======================
MCheu
-Internat ional shopper of mystery

Re:Some stores frown upon such things

[mcheu]

There are some stores that actively discourage people from bringing in notebooks and PDAs to record their pricing information.

Just to clarify, this wasn't a computer notebook. What I had was a pocket size coiled paper pad and an orange HB pencil.

Damn that would be a feedback loop!

[gpoul]

I guess we'll soon have really fucked-up e-commerce websites if more people start to use this technique and robots from multiple sites start to underbid each other.

Will they start to sell items for $-1??

missing the point

[Anonymous Coward]

I think the point here is being missed. Read the brief. There were confidentiality agreements involved which ultimately decided the direction of the original injunction. As for the second injunction, a third party cannot aid in the circumventure of an injunction. Period.

You or I could gather price data from sites and use it as a guidelines without legal worries. Compiling and selling that data outright would probably get you in trouble because of its time sesitive nature.

WTF?

[AyeRoxor!]

"Ask Slashdot: Is Data Mining for Product Pricing, Illegal?"

Who, wrote this, subject, William, Shatner?

Re:let me ask you this.

[jjoyce]

I was not claiming that one should always follow the subject with a comma. The rule is that commas should be inserted to separate parenthetical phrases and to indicate natural pauses. If there is a comma after "pricing", then there must be one after "Data". However, both commas should be omitted because the proper way to express that sentence would be to recognize that "mining", not only the subject of the sentence but also a gerund, requires a possessive:

Is Data's mining for product pricing illegal?

It is a far clearer sentence now.

"Data Mining"?

[Ominous Armed Cow]

This is just "web scraping".

You already know you are looking for very particular information (prices) which can reliably be found in a particular location. Data mining involves slicing and dicing data to discover new information which may or may not be there.

BTW, [and yes, IAAL], it doesn't make a whole lot of sense for non-lawyers to fret over dubious caselaw from what is probably some other jurisdiction with a very different set of facts.

Just because one judge was dumb enough to go along with a bad decision doesn't mean you have to worry what the next one would do. Don't worry, they'll send you a letter if they really don't like it, and then your legal department can worry about it. With very few exceptions, just collect your paycheck, live your life, and move on.

Same thing at Fry's, more or less

[Anonymous Coward]

I was with a friend at Fry's where we were examining a number of DVDs to see if they had a few items in some out-of-print sets. Since I can never remember which of these items I am looking for I have a list on a PDA. After I pulled out the PDA and we started looking through the poorly-organized shelf of DVDs one of the obvious-looking Fry's "loss prevention" or whatever they are Fry's employees appeared. We ignored him and he didn't say anything either, just stood there looking. After a while another one showed up and they had a conversation together in some non-English language which ended with the 2nd one saying 'collectors' in heavily-accented English, after which time they both left. Don't know what would have happened if they had decided otherwise, or if they had actually spoken English well enough to communicate with customers.

Can They Stop Disabled From 'Scraping'

[roboneal]

I'm sure there are a few products that assist disabled persons to "surf" websites by disecting the web page (through essentially screen scraping techniques) and performing one or more the following:

1. Adjusting text size.
2. Dictation of content.
3. Numbering of links.
4. Numerous other alternate presentation of the same data (changing colors for the color blind for example).

An outright ban of automated scraping techniques would eliminate these uses. (While I am at it: What is a web-browser but a form of screen scrapper?).

If the basic technique is allowed, all that can be debated is the use of such data and I think that is a much more dubious area. Facts are public domain.

Maybe they can use the "my bot is blind defense".

More specifically, what's wrong with scraping?

[0x0d0a]

Forget price protection for a moment -- the thing the stores don't like is specifically price scraping. Why do we have any justification to put up legal barriers to block price scraping? It benefits consumers and drives prices down. I'd call it a *positive* factor that is necessary for a free market.

Hell, I could see *requiring* retail outlets to make their prices publically available.

Advise, document, release.

[Anonymous Coward]

It is VERY common practice to explictly forbid the use of scrapers in the Ts&Cs you quickly pass over on such sites -- and perfectly reasonable. You are using that company's hardware and software under conditional use. Both Expedia and Travelocity prohibit ALL commercial use.

I'm not a lawyer, but I would cover my legal ass before I released a single line of code and I would be hesitant to proceed on this project until I talked to a lawyer. If it's a work-for-hire project, you probably have less of a liability problem.

James T. Kirk would ... use ... an ellipsis.

[Anonymous Coward]

And what would you put there - a semicolon?

Sheesh.

here is a similar case

[mcguyver]

Here is a related incident:

http://news.com.com/2110-1017-944258.html

Bargain Network spidered real estate prices on homestore.com/realtor.com and posted them on the bargain.com website. Homestore sued and the case was settled out of court. I wish it was not settled out of court because that would set up a precident.

In my opinion you are asking for the problems. Taking a case like this to court and winning would be difficult. At the very least it would be a serious legal expense.

The last time I checked the rules for Froogle you had to be the actual merchant that ships the product in order to show up in their index. If you are spidering a merchant then you are an affiliate, the products do not originate from you so you would be exluced from Froogle. Froogle does not allow you to sort products by price - so obviously what you plan on doing is different. Froogle also gives merchants the option to be excluded from their index.

My advice is this - get a lawyer because one will surely be contacting you. Familiarize yourself with these phrases: false advertising, breach of contract, and unfair competition.

changing text font, etc

[nsushkin]

Don't use gnuplot. Use Grace. It can be scripted and is much more powerful and flexible.

Re:For airline prices, there is "creativity" invol

[Anonymous Coward]

Compare 5 different web sites' prices for a complex ticket and you'll get 5 different numbers

That is because part of the price calculation involves the second and third letters of the name or IP address. The reason for this is a trade secret, so nobody will tell you about it.

If you dont want me to buy there stuff...

[SkewlD00d]

1) Use $1100 - $1000 rebates
2) Keep their prices and stock a secret.
3) Use deceptive advertising to lure me into the store.

The question in the article is bogus, it's like the whole "deep-linking" scare. What's next? Registration and signing an NDA to view some amateur site where the search feature doesnt work? BS. I'll shop somewhere else. Whatever, if it's publically accessible and doesn't copy (or IFRAME) any content, then you can point to it. It's like the whole "you can't take pictures" or "write down prices" in our store BS.

I've worked retail... the best customer come-back ratio and most profitable stores are *always* UPFRONT and HONEST about their pricing, products, service and policies. To me, anything less is a store/business shooting itself in the foot. You might hood-wink a few customers some of the time, but word-of-mouth, the internet and economic forces will surely take you out.

That's my two pennies

I just wish...

[SkewlD00d]

... Digikey would let me sort by price.

Therefore, I will spiderbot them. ;)

You could just make all your prices on your site into OCR-hardened, dynamic images. But it's lame to force your customers to check every site. It's anti-competitive.

You misread the headline, silly!

[alienmole]

Using typical Slashdot grammar, it's asking whether it's illegal for Data to mine for product pricing.

See, the problem is that Data's positronic brain and direct computer interface gives him an unfair advantage over humans who are mining for product pricing. So, the obvious solution was to make Data mining for product pricing, illegal.

a simple and clear cut solution

[a302b]

A simple solution is to check a robots.txt file or similar. If there is none, it should be legal to grab prices online.

However, if through such a file, someone disallows price leeching, then people should respect that and be liable if they ignore those instructions.

You're Fried

[Anonymous Coward]

No matter whether you tell your client or not, you've just announced to lawyers in the future that you knew about the legal situation. Lawyers looking for violations similar to what you described will be trying to figure out who you are. Let's hope that you twisted the story and are actually working in a different field (such as competitive Peeps prices).

Check your contract

[Anonymous Coward]

You probably don't have to worry about it but some consulting contracts contain clauses that might lead to trouble if your client is nailed for using this system because you did something illegal in creating it. Usually this is limited to IP violations (stealing code, voilating patents) but the language can be pretty broad sometimes.

I don't agree.

[Futurepower(R)]

Lawyers are often 100% adversarial with people whom they detect cannot supervise them well. You MUST know the law yourself. You MUST think about what should and should not be legal. You cannot sensibly leave important questions about legality to people who make $350 per hour for dealing with confusing legal situations. There is a HUGE conflict of interest. Lawyers make more money if non-lawyers cannot understand the law.

title has CONVERSATIONAL grammar

[oliverthered]

Slashdot is a conversation web site, so, I would expect to see a lot of conversational grammar in
use.

In the case of the headline, I believe that the comma represents 'a DRAMATIC pause'.

Much like:
The inhabitants of Mars, do they realy care?

It was not price mining that the court prohibited!

[theNAM666]

AARRRGGH. I hate ./ sometimes.

From the court summary of the decision:

The court affirmed a preliminary injunction enjoining defendant Zefer Corporation ("Zefer") from utilizing a "scrapper" tool it designed to obtain pricing information from plaintiff's website on the ground that Zefer was doing so to assist defendant Explorica, Inc. ("Explorica"), which was itself enjoined from such activity by virtue of its improper use of confidential information obtained from plaintiff to aid it in gathering this information.

This means that Zefer was prohibited from mining the data because their client (Explorica) was prohibited, because Explorica and Zefer had gained access to the data by exploiting confidential information. Which is another issue entirely from data mining...

Re:Please tell me how to mine for prices on the We

[Anonymous Coward]

me too!

Actually, WalMart sued fw over pricing:

[mandalayx]

from link:

In a blatant misuse of the Digital Millennium Copyright Act, over the past two weeks a group of national retailers forced FatWallet.com (www.fatwallet.com) to remove Day After Thanksgiving sales information from its site. In letters sent to FatWallet, each retailer claimed that the Copyright Act gives it a monopoly over this price data. Today, the Samuelson Clinic and Gray Matters, on behalf of FatWallet.com, challenges those letters as abuses of federal law, insists on damages, and refuses to disclose identifying information on the individuals who posted the sales information. For more about this issue, please read the press release [PDF], FatWallet's online story or the Chilling Effects story.