Facebook Crawler Speaks Back

Last week we ran a story about Facebook suing to get a crawled dataset offline. This week we have a bit of a response written by Pete Warden, the guy who actually did the crawling. He followed robots.txt, and then Facebook's lawyers went after him. It's actually a quite interesting little tale and worth your time.

Comments

Pretty naive

[elrous0]

Did this guy really think he could just give away the data that Facebook sells (or intends to sell) to third parties and NOT have them sue him for it? It's no secret that the business model of most of the social sites and big search engines factor in the massive amounts of data they collect on users as a major corporate asset, to be used internally for data mining and also sold (supposedly after being anonymized) to advertisers and other third parties. It takes a babe in the woods to think he can just waltz in and take that away with a "But your robot.txt didn't say I *couldn't* do it" defense, without expecting a big legal fight.

Is the guy in the right? Probably. Would he have a case? Probably. Does either of those facts matter if he doesn't have the big $ needed to hire lawyers and fight through several courts? Nope.

Re:Pretty naive

[Anonymous Coward]

If he's in his right, but not having as much money as a big cooperation means he'll lose anyway, then your U.S. court system is broken. Please fix it.

Re:Pretty naive

[qoncept]

matter if he doesn't have the big $ needed to hire lawyers

Thank you. I ran an open source project for a few years and came home one night to find to find that my webhost had taken its site down after being contacted by a company with a similar name. The company claimed they'd tried to contact me, explained how my project was causing them harm, but the simple fact of the matter was that my project's name did not infringe on theirs.

I ended up renaming the project. I've told the story dozens of times, and the response is always the same. "That's BS! They can't do that! Go to court!" People don't understand that $20 a month in unmanaged Google ads doesn't cover lawyers the same way that company's actual paying customers do.

Re:Pretty naive

[julesh]

It takes a babe in the woods to think he can just waltz in and take that away with a "But your robot.txt didn't say I *couldn't* do it" defense, without expecting a big legal fight.

Yes. Apart from anything else, he's just about entirely missing Facebook's point. Facebook don't give a shit how he accesses their site; this has nothing to do with the fact that he spidered it in a way that their robots.txt file allows, and everything to do with the fact that he was *redistributing their data* without consent.

Now, the question becomes whether what he was distributing falls under fair use. This is a very tricky question, and has nothing to do with how he acquired it.

Re:Pretty naive

[elrous0]

I hope you're being sarcastic. If not, I have some bad news for you.

Re:Pretty naive

[Pharmboy]

American justice might be blind, but it know what money smells like. One more reason why we need judicial reform to prevent abuses like this. Of course fighting it wouldn't be worth it, as even if you won, your "winnings" would have only been the ability to continue using the name. Another good example is http://www.nissan.com, where he actually fought and won, at a great price. His name is Nissan, and his computer business and name existed back when the cars were called "Datsun", but they sued anyway. This is another one of those "We are bigger than you, thus more deserving of the domain name than you" cases.

Re:Pretty naive

[whisper_jeff]

If they want to sell data (as they clearly do given that's what their business model is built upon) then they should take greater precautions to ensure that it is protected. If they leave that information out in the open, for anyone with a hint of insight to find, then they should not be surprised to find their valuable data in the hands of someone else. He didn't delve into their private information - he simply accessed publicly available information that anyone with an internet connection could view.

Facebook got lucky - the data was gathered by just an average Joe without the backing to fight a legal battle. Had it been someone significantly larger, the result may have been "go ahead and sue - we'll see you in court." And, quite frankly, I'd be shocked if Facebook would win that sort of battle. And that's a battle that Facebook decidedly does not want to lose - it would mean the end of their business...

I'd be curious to learn if that information is still available (as I am certain it is...) because someone/some company might decide that's pretty valuable _PUBLIC_ information and might, just might, decide they're willing to battle Facebook's legal team for it... Expensive legal battle over very valuable marketing data... If you have the resources for the fight, it might be a fight worth waging...

Facebook may have gotten lucky once but they may not be so lucky next time...

Re:Pretty naive

[Lumpy]

WE have the best court system money can buy!

Re:Pretty naive

[Archangel Michael]

Want to fix the ELECTION laws, while not breaking the First Amendment Rights to Free Speech? It is really quite simple. One simple rule.

Only People (persons, not legal entities)who are eligible to vote can donate to political campaigns.

This doesn't deny corporations from running ads, they just have to do it on their own, and out in the open where everyone can see who they are telling people to vote for. They have to buy their own ads to tell people to vote for Harry Reid or Mitch McConnell.

This also goes for Unions and all other organized groups. Make them buy their own ads for their own causes.

Simple rule, clear, concise, straightforward and solves all sort of problems with current campaign laws, without any bias towards or against anyone.

AND that is why it won't ever be implemented.

And I'm sure that there is someone that is going to be upset because their favorite group won't be able to donate money to a candidate/campaign while at the same time restricting anyone that might oppose them (it) from doing likewise at the same time.

Re:Pretty naive

[FlashBIOS]

[...] where were they when Bush and the Republican congress took Clinton's balanced budget and ran up the biggest deficit in history? [...] Where was their cry to vote against incumbents when the Republican held the majority and were running up the debt?

So because they are late you're going to discount their argument? Doing the right thing in spite of it of it being timely is still better than not doing the right thing because you think you may have taken too long.

Mark Zuckerberg

[prayag]

Mark Zuckerberg is the most unethical guy in the industry today. As is obvious by the origins of Facebook, his infamous hacking of the journalists passwords during the the-facebook era and countless other fiascoes that come to news from time to time. Everyone who has ever dealt with him says have bad things to say about him.
If he is the face of the next generation entrepreneurs, then god saves the industry.

Arachnophobia

[mfh]

I might be alone here but spiders revolt me to a point where I simply respect them and leave them alone.

But that said, Google operates a spider, pretty much. So we have to look at any potential spider on the internet like we might look at Google. If he followed the Robots.txt as Facebook set it up and he didn't try to misunderstand it, then there isn't anything they can do. Although, I'm pretty sure the Facebook EULA says you can't spider them so he's SOL anyway if that's the case. This should be a long and drawn out case unless there is a settlement.

Facebook is ripe. People put up EVERYTHING about themselves on there. I never accept a friend request unless I know the person and I offer a challenge question often. If it's not responded to adequately, I simply ignore them. But in the end there isn't much you can do. If you put it on Facebook -- consider it public, like if it was in the phone book.

Re:Arachnophobia

[OnlyJedi]

From the Statement of Rights and Responsibilities, Section 3 "Safety":

2. You will not collect users' content or information, or otherwise access Facebook, using automated means (such as harvesting bots, robots, spiders, or scrapers) without our permission.

The question then becomes how enforceable is the agreement? Sure, if he has an account Facebook can close it, but if he is just accessing Facebook without an account do they have a case? Last I saw you can browse parts of profiles without being logged in, and without ever agreeing to any terms.

obviously this is abusive

[circletimessquare]

this is what the guy should do:

1. engage the lawsuit

the downside is financial exposure. so incorporate your work in such a way that it can't hit your personal finances. the upside is massive exposure. you will achieve some level of fame: the guy who finally gave the robots.txt convention a legal status quo. this will help you professionally, as well as make your life story

2. whine to google

you are completely right that google shouldn't have to get permission every time it wants to crawl the site. therefore GET GOOGLE TO DEFEND YOU

Re:obviously this is abusive

[ikoleverhate]

how about if he rejigged his crawler to get the data from the google cache instead? So he'd never get anything from facebook or enter into any implied agreement with them.

Re:Ballsy.

[hansamurai]

Not really ballsy considering he didn't actually let Facebook's challenge of "The only legal way to access any web site with a crawler was to obtain prior written permission" go to court. Maybe he should have gone to the EFF for help as the repercussions of a judge actually deciding in Facebook's favor would have been devastating to the web.

Ooo, deja vu

[lxt]

It's sort of ironic that Facebook is trying to stop someone crawling public profiles on their site, because that's exactly what Mark Zuckerberg did while he was at Harvard (I was a grad student in the CS department at the time).

Pre-Facebook, Zuckerberg created a site that let Harvard students compare each other, a bit like Hot or Not. Obviously nobody was going to go to a site that wasn't populated with their classmates, so he basically crawled the websites of the various residential houses that put their students info online (but behind passwords and auth) and copied it into his own site.

He actually got into a fair bit of trouble for this, and ended up being sent to Harvard's ad-board for discipline (I think he got put on probation, but I'm not entirely sure).

The key difference here is that this guy actually did everything by the book and followed robots.txt, whereas Mark Zuckerberg didn't.

Facebook's privacy policy

[whencanistop]

Facebook's privacy policy says:

“Everyone” Privacy Setting. Information set to “everyone” is publicly available information, may be accessed by everyone on the Internet (including people not logged into Facebook), is subject to indexing by third party search engines, may be associated with you outside of Facebook (such as when you visit other sites on the internet), and may be imported and exported by us and others without privacy limitations. The default privacy setting for certain types of information you post on Facebook is set to “everyone.” You can review and change the default settings in your privacy settings. If you delete “everyone” content that you posted on Facebook, we will remove it from your Facebook profile, but have no control over its use outside of Facebook.

I'd also like to point out in their terms:

When you publish content or information using the "everyone" setting, it means that everyone, including people off of Facebook, will have access to that information and we may not have control over what they do with it.