LinkedIn Says It's Illegal To Scrape Its Website Without Permission (arstechnica.com)

A small company called hiQ is locked in a high-stakes battle over web scraping with LinkedIn. It's a fight that could determine whether an anti-hacking law can be used to curtail the use of scraping tools across the web. From a report: HiQ scrapes data about thousands of employees from public LinkedIn profiles, then packages the data for sale to employers worried about their employees quitting. LinkedIn, which was acquired by Microsoft last year, sent hiQ a cease-and-desist letter warning that this scraping violated the Computer Fraud and Abuse Act, the controversial 1986 law that makes computer hacking a crime. HiQ sued, asking courts to rule that its activities did not, in fact, violate the CFAA. James Grimmelmann, a professor at Cornell Law School, told Ars that the stakes here go well beyond the fate of one little-known company. "Lots of businesses are built on connecting data from a lot of sources," Grimmelmann said. He argued that scraping is a key way that companies bootstrap themselves into "having the scale to do something interesting with that data." [...] But the law may be on the side of LinkedIn -- especially in Northern California, where the case is being heard. In a 2016 ruling, the 9th Circuit Court of Appeals, which has jurisdiction over California, found that a startup called Power Ventures had violated the CFAA when it continued accessing Facebook's servers despite a cease-and-desist letter from Facebook.

Comments

then dont' make it public

[Anonymous Coward]

don't make it public fi you don't want it read

Re:then dont' make it public

[Anonymous Coward]

don't make it public fi you don't want it read

They want it read. By people. (And search engines.) They don't want it read by companies that take the information and then sell it as their business model.

If we support hiQ, saying that scraping publicly-accessible content from another site and then using that for profit is permissible, then doesn't that mean it's also applicable to other sites? Slashdot's content is public: can I scrape everything, host it on my site, insert ads, and make money?

Sorry hiQ, as much as software and internet legislation is behind the times and technically inappropriate, there are some things in law which follow common sense - and one of them is you can't take someone else's stuff and sell it for yourself. If you want to use their content then you need to follow the (common) practice of establishing some sort of licensing agreement.

But anyways, what about their user agreement?

You agree that you will not: [...] Develop, support or use software, devices, scripts, robots, or any other means or processes (including crawlers, browser plugins and add-ons, or any other technology or manual work) to scrape the Services or otherwise copy profiles and other data from the Services;

Is that not enough for at least an injunction and civil suit?

Re:then dont' make it public

[BronsCon]

They don't want it read by companies that take the information and then sell it as their business model.

What do search engines do, then?

Re:then dont' make it public

[tattood]

What do search engines do, then?

Search engines create an index that is searchable and make money by selling ads on the search page. Search engines are NOT collecting the website data, and make correlations about the data on the website and selling that data to companies.

Re:then dont' make it public

[saloomy]

Even so they do (display titles), and even so they haven't (figured out a way to display excerpts without collecting data), Google respects a sites robots.txt, which clearly linked.in as asked hiQ to do, but hiQ is flouting the EULA. On one hand, I agree: if you don't want it read, keep it private.

On the other hand: Posting something in the public space effectively gives the readers a right to consume it (akin to reading a book). It does not give the reader the license to freely copy or build upon it (akin to republishing the book under their own name, or writing a sequel to a book they have read) using the same characters and titles. For that, you need a license from the original content creator (Linked.In in this case).

Re:then dont' make it public

[BronsCon]

Right, the issue here is the "we want to let search engines use it without license, but want to require a license for anyone else" attitude. Either require everyone who uses it to license it (even if you don't charge some of them), or require nobody to license it, that's more or less how copyright law works.

Re:then dont' make it public

[BradleyUffner]

Exactly, "403: Forbidden" exists for exactly this reason.

A program asked a web server for a particular page. The web server said "here you go!", instead of returning a 403 error.
If this is about copyright infringement, then sue under a law that applies to that.

Re:then dont' make it public

[JaneTheIgnorantSlut]

I guess you can't print a page either.

Re:then dont' make it public

[sexconker]

No, only one side has legitimacy.

If you complain about people using information you post PUBLICLY, you are an idiot.
This doesn't even rise to copyright infringement.

Re:then dont' make it public

[smooth wombat]

"we want to let search engines use it without license, but want to require a license for anyone else" attitude.

No, that is not correct. Search engines point to a page and may give a very brief line or so from the article, but one still has to click on the link to go to the real page and read everything.

hiQ goes to the Linkedin site and rather than pointing to the pages in question, takes the data, packages it, and then sells it to someone else, having left Linkdedin to do all the heavy lifting.

The two are not close.

Re:then dont' make it public

[F.Ultra]

Where exactly in the complaint by Linkedin are they telling search engines to not index linkedin.com?

Re:then dont' make it public

[Dog-Cow]

If MS asserted a copyright claim, that would be different. There is no fraud and no hacking taking place when scraping publicly-accessible data.

Re:then dont' make it public

[BronsCon]

The two are not close.

They really are, though. LinkedIn has copyright on all of their content, in whole and in part, not just as a whole. That's how copyright works, otherwise I could change a single word in a book and republish it as an original work under its own copyright. It is also important to keep in mind that (most) search engines -- and Google specifically -- don't just grab the page title, META description (or first couple lines of content) and a word/phrase count, they grab the entire content of the page, and they do so in order to display the exact part of the content that contains your search term(s) -- as I mentioned earlier -- rather than a likely irrelevant summary or intro.

To do this, search engines must necessarily use the entire page and not just key pieces of data. That is, Google et-al get away with using more of LinkedIn pages without license than hiQ is using. Therein lies the problem.

Re:then dont' make it public

[BronsCon]

Not relevant to the argument being made, which was in response to someone claiming they want search engines to be able to do what search engines do then, in the very next sentence, claiming they don't want search engines to be able to do what search engines do.

Re:then dont' make it public

[Gavagai80]

Slashdot's content is public: can I scrape everything, host it on my site, insert ads, and make money?

Copyright law clearly makes that illegal. This case is a little different in that it seems to be about the kind of data that can't be copyrighted.

Re:then dont' make it public

[F.Ultra]

But it seams that Linkedin want precisely that, i.e for search engines to continue to do what they do but not let hiQ do what hiQ does.

Re:then dont' make it public

[BronsCon]

Right. I was replying to this, though:

They want it read. By people. (And search engines.) They don't want it read by companies that take the information and then sell it as their business model.

I was pointing out that search engines "take the information and then sell it as their business model."

Sorry you missed that.

Re:then dont' make it public

[omnichad]

Slashdot's content is public: can I scrape everything, host it on my site, insert ads, and make money?

Plain old Copyright law is enough to put an end to that. However, facts are not copyrightable, and Linkedin has a lot of valuable facts in its database.

But anyways, what about their user agreement?

Can you put a EULA in a document folder (page 50 in a stack of 200 pages) and throw it on the ground in the park, and expect to enforce it when it tells people not to read the other pages in the envelope? That's the physical-world equivalent.

Re:then dont' make it public

[bws111]

It doesn't matter what search engines do. The owner of the site is perfectly within his rights to say 'these accesses are allowed, these are not'.

Being indexed by a search engine is probably beneficial to LinkedIn. Both parties gain from being indexed, it is a symbiotic relationship.

HiQ is probably not beneficial. By ratting out LinkedIn's user to their employers they are potentially decreasing the number of people who will use LinkedIn. That is a parasitic relationship.

Re:then dont' make it public

[omnichad]

1) collecting the website data. Check The spider downloads all of the text content and stores an index along with contextual relationships.

2) make correlations about the data on the website Check The hyperlinks on the web site are used to evaluate the relative importance of the linked web site.

3) selling that data to companies. Nearly They don't charge for the search engine directly - they charge for advertisers and then provide the data for free to visitors. More or less the same thing effectively speaking.

Re:then dont' make it public

[BronsCon]

The owner of the site is perfectly within his rights to say 'these accesses are allowed, these are not'.

Yes, and they can do that with HTTP200 and HTTP403 status codes, respectively.

Re:then dont' make it public

[buss_error]

No, that is not correct.

I'm of two minds about LinkedIn.
In the first place, I'm required to have an account by my current employer.
In the second place, LinkedIn in my opinion does a ton of scraping themselves (asking to access your mail box contacts, for instance.) But at least Linkedin ASKs to access it. Still, it feels creepy to me. The "psycho" girl friend kind of creepy.

On the third hand, LinkedIn told the to stop. So they should stop.

Re:then dont' make it public

[F.Ultra]

I didn't miss that, just look like you thing that extracting the title of a page constitutes "take the information and then sell it", something that is covered by fair use. It would be a whole different affair if i.e Google extracted and resold the amount of information that hiQ does, which of course was the point of the GP.

Re:then dont' make it public

[bws111]

Sure, they CAN do that, but they don't HAVE to do that. Once you have been told you don't have permission, you don't have permission.

Re:then dont' make it public

[BronsCon]

I didn't miss that, just look like you thing that extracting the title of a page constitutes "take the information and then sell it"

No, it looks like you missed where they're taking the entire content of the page and not just the title, since you keep coming back to "just the title".

It would be a whole different affair if i.e Google extracted and resold the amount of information that hiQ does, which of course was the point of the GP.

So, if Google took only key pieces of information, rather than the entire page, that would be problematic? Because Google takes the whole page, while hiQ takes key pieces of data; Google is actually taking, repackaging, and profiting from more of LinkedIn's data than hiQ is.

But, all of that is still highly irrelevant to what I was replying to.

Re:then dont' make it public

[BronsCon]

Actually, anything you're able to view from a public space is fair game under current laws, with the exception of court orders stating otherwise. If hiQ's servers can view the content from the public internet (that is, if LinkedIn's servers serve it to them without them hacking around some technical measure), it's fair game unless LinkedIn gets an injunction against hiQ. That is, what you're claiming is really for the courts to decide.

Or, you know, LinkedIn could just claim copyright on their data and issue a series of DMCA notices.

IANAL but I've consulted several regarding a very similar issue in the past.

Re:then dont' make it public

[angel'o'sphere]

The question is nor what they grap/scrap but what they copy and redistribute.
Google creates a catalog, pointing with every search result to the original.

HiQ is balantly violating copy right, privacy rights and EULAs/TOSs.

If you don't grasp the difference I hope you are not a software developer. Ignorance on that scale can easy be the end of your career.

Re:then dont' make it public

[BronsCon]

Actually, both violate copyright, but the reality is that they can (and should) choose who they go after. And on those grounds, LinkedIn can and should issue a flurry of DMCA notices and sue for an injunction. They can't sue for damages without registering their copyright, including a copy of the work being registered, but they sure can get an injunction. The CFAA simply does not apply to publicly available data until and unless they get said injunction and even then it would be a hell of a stretch.

Also, I suppose it takes near average or higher intelligence to determine when someone is making an absurd argument in an attempt to point out the absurdity and incorrectness of the argument he is countering.

Re:then dont' make it public

[KingMotley]

Uh no. That's not how copyright works. They have to right to grant or deny the rights to copy their data however they see fit. Their EULA clearly states that you are not allowed to do exactly what HiQ is doing. Copyright, unlike trademarks, does not forfeit its right to enforce it just because they haven't in the past, or have chosen to enforce it in specific instances. HiQ has been directly told they don't have permission, end of story.

Re:then dont' make it public

[AHuxley]

Re "What do search engines do, then?"
Connecting people who worked on secret mil/gov projects with people looking for staff to work on other secret mil/gov projects.
So people list all the projects they worked with and can show they are trusted in plain text.
They used the same methods in the gov/mil and just expect the same results on the net.

Re:then dont' make it public

[BronsCon]

Read the rest of my comments, then feel silly.

Re:then dont' make it public

[American Patent Guy]

No, that's not quite correct. LinkedIn only has a copyright in that which (1) they acquired from their employees or other sellers and (2) constitutes a "work of authorship". They do not have a copyright in the content acquired from other sources, e.g. data, phrases, images that originate from members or other sources. The arrangement of information on a page may be a work of authorship, but only if there is some creative aspect to it. Data on a web page is not a work of authorship, and no one has a copyright in it. Not even by the one who produced or collected it.

Re:then dont' make it public

[American Patent Guy]

Data is not copyrightable, because it isn't a "work of authorship" under the copyright statutes. That's why LinkedIn is using this hacking law in a contorted way to try to stop the use of this content.

Re:then dont' make it public

[kwbauer]

No, they have given permission to search engines to do what search engines do which is to direct traffic to the website from which the data was collected. The search engines are operating within the license given to them by LinkedIn.

LinkedIn has not given hiQ permission to do what it is doing. Kind of like how a songwriter or copyright holder can give permission to an organization to reprint songs/music in a songbook but restrict others from reprinting those same songs or even photocopying them out of the authorized book even if other music in that book is openly allowed to be copied.

This really is not a new concept.

Re:then dont' make it public

[kwbauer]

They do not violate copyright if the copyright holders says they don't violate copyright. Kind of like how the police cannot charge my neighbor for stealing my lawnmower if I don't care that he has borrowed it even if I didn't give my express permission to him prior to the police asking me about it.

Re:then dont' make it public

[kwbauer]

but the "less" and "effectively speaking" are the keys to the whole thing. Along with that pesky little thing called copying without permission for the intended usage. For some concrete, everyday examples, wander into any Catholic or Methodist or LDS (Mormon) church and look through the hymnal. You will find something at the beginning explaining how all the music can be copied for non-commercial use except as otherwise noted. And then you will find that some of the songs are marked with phrases that fit the "otherwise noted" category. This is the exact same situation applied to the content on a web-site instead of the content in a book of music.

Re:then dont' make it public

[saloomy]

No. The difference between a search engine and hiQ is that a search engine (Google specifically, but I think all of them) respect the robots.txt instruction set. If you don't want a search engine in your content, then there is an "opt out". Maybe one could argue that this should be an "opt in", but there is a way to say "I don't want to authorize you to scan my page for indexing".

hiQ has not listened to Linked.In's "opt-out" in the form of a cease and desist which is a pretty strong indication the content owner doesn't grant the permission to be scraped.

I think the court will find in Linked.In's favor here, and so they should. hiQ is operating without a license to reproduce the original works. This is what Copyright is designed specifically to stop.

Re:then dont' make it public

[kwbauer]

But Google is doing something of which LinkedIn approves and has given Google permission to do. hiQ, on the other hand, is doing something of which LinkedIn does not approve and has not given hiQ permission to do. That is entirely the difference here. LinkedIn believes that they benefit from the way Goole indexes their pages and allows them to be searched but LinkedIn believes that what hiQ does is harmful to LinkedIn as it will tend to drive people away.

I understand that people who have never created anything of value or who believe strongly in socialism have no concept of ownership of property (aka, property rights) so they cannot distinguish between the two usages but, until such a time as the whole world has descended into the dystopian hellholes you have name Utopia, please try to follow along.

Re:then dont' make it public

[kwbauer]

A library shelf is a public space and so is a museum wall. Are you claiming that anybody has the right to walk in, take pictures or photocopies of anything in those public spaces and resell those copies and that they are not violating current law? I would be asking those several lawyers that you consulted with for a refund.

Re:then dont' make it public

[alzoron]

This is not a copyright issue. This is a CFAA issue. It's been long determined that you cannot copyright facts. The CFAA deals with unauthorized access to computer systems. LinkedIn told these companies to stop doing it and they kept doing it That's a pretty clear case of unauthorized access.

Re:then dont' make it public

[BronsCon]

When it comes to reading (viewing in your museum example), which is what was discussed in the argument I was originally replying to, the above is absolutely true. When it comes to copying, it's a little more nuanced than that, of course; but, then, I was writing a Slashdot post, not a fucking dissertation, I certainly was not giving legal advice and, again, was arguing against someone who claimed that merely viewing something viewable from public space, which the owner readily serves up to you with no technical measures in place to protect it, is illegal. De-facto, it is not. That is, I was talking about vieweing, not copying, a fact that becomes abundently clear when you read the entire post, rather than just the first and last sentences.

And, with regard to LinkedIn's complaint, whether or not hiQ has the right to view the data is precisely what matters. They do have that right, which should render the complaint invalid.

Copyright, on the other hand, well... I talked about that in part of my post that you clearly skipped over.

Re:then dont' make it public

[BronsCon]

You can't copyright facts, but you can copyright collections of them, provided you've done more than simply compile facts you already had. That is, if you had to collect the facts from multiple sources, your representation of those facts is protected by copyright. LinkedIn's data is a collection of facts, which they collected from multiple sources.

I'm sure you're very familiar with patents, but you've missed some nuance of copyright law in your reply.

Re:then dont' make it public

[BronsCon]

You can, in fact, copyright a representation of facts you've done work to compile from multiple sources. LinkedIn has done work to compile their collection of facts and is entitled to copyright protection.

That is unlike, for example, a local phone directory, because the local phone company is the sole source of that data. A nationwide phone directory, comprising data collected from the various ILECs and CLECs, would be a copyrightable work. The bar being so low in that case would likely mean you'd be fine putting out your own version if you can prove you collected the data yourself, but it's still protected.

As for accessing a public website, well, HTTP provides for access control and it needs to be used here. Further, we're talking about LinkedIn which, like Facebook, participates in active tracking of users of other websites, so access to their servers is practically forced on you if you don't actively seek to avoid it. I can't tell you to stop letting me fling data at your face, then hold you liable when I don't stop, which is precisely why technical measures (on LinkedIn's part) would be necessary for this to be a CFAA issue.

Re:then dont' make it public

[American Patent Guy]

A compilation of facts can be copyrighted, but not the underlying data. If this company wants to extract those facts, data or other bits of information and create its own compilation, it violates no one's copyright. It doesn't matter whether those facts came from multiple sources or a single one.

Even if there were to be a copyright here, the doctrine of implied license and the statutory exclusion of fair use upon infringement would probably apply. By making the data available to anyone over the web, LinkedIn arguably granted an implied license to use that data. (They don't publish it to keep it secret.) The courts have ruled it is a fair use to record movies on your DVR for your own personal viewing, and it would arguably be the same for extracting a collection of data from an Internet source, provided that the entity didn't compete with that source.

I missed nothing. I'm an intellectual property lawyer.

Re:then dont' make it public

[BronsCon]

A compilation of facts can be copyrighted, but not the underlying data. If this company wants to extract those facts, data or other bits of information and create its own compilation, it violates no one's copyright. It doesn't matter whether those facts came from multiple sources or a single one.

If, in doing so, a copy of the compiled data is made... well...

The courts have ruled it is a fair use to record movies on your DVR for your own personal viewing

Yes, it is.

and it would arguably be the same for extracting a collection of data from an Internet source

Sure, for your own personal use.

provided that the entity didn't compete with that source

You mean provided the use was not commercial or for profit, right? After all, you're:

an intellectual property lawyer

Your appeal to authority does not imply correctness or completeness of understanding; especially so given the argument you just made.

Re:then dont' make it public

[American Patent Guy]

If, in doing so, a copy of the compiled data is made... well...

... and because that's how LinkedIn provides the data, the scraper operates under the doctrine of fair use. (There's no other way for it to collect the data.)

Whether your copying of the data is for personal or business use is not distinguished in the law. Your impact on the market is what counts. This scraper isn't affecting LinkedIn's ability to operate or provide the service that it does. You're free to gather information over the web (or another medium) as much as you like, recompile it, and resell it if you want to. Whether you profit from it doesn't matter. Publishers of directories and phone books have been doing this for many years. Google, Yahoo, Bing and all the search engine providers do it too.

Would you like me to correct you a further time?

Re:then dont' make it public

[BronsCon]

So wait, what you're saying is if my use of a short sample from a song doesn't affect the sale of that song, it's fair use? Sorry, Robert Van Winkle wouldn't have settled out of court if it looked like he was going to win against Bowie and Queen. What's happening here is practically the same. So, no, I don't think I'd like you to correct me again, since your "corrections" don't align with reality.

Fair used is determined based on four factors, which I'm sure you're quite familiar with (but hoping I'm not aware of for the sake of your argument):
- the purpose and character of your use (e.g. commercial or not - which is relevant here)
- the nature of the copyrighted work
- the amount and substantiality of the portion taken, and
- the effect of the use upon the potential market.

Your argument is, effectively, that only the latter of those factors matters. Sorry, Mr. Slashdot Lawyer, but you're wrong and the law as written says so quite clearly. You've got the books, look it up for yourself.

Regarding phone books, I've already raised that point elsewhere.

I get it, you're an IP lawyer who specializes in patent law. It's okay, there are a lot of facets to IP law and it's important to specialize and be good at your specialty; other specialties will suffer as a result, but that's fine. You might consult one of your colleagues specializing in copyright before continuing this conversation.

Re:then dont' make it public

[BronsCon]

Ugh... I had a whole response typed out, clicked Preview and, in my sleep-addled state, closed the tab without posting. I'm not writing it all again, but the long and short of it is that there are four factors used in determining fair use and you're considering only one of them. I would argue that hiQ's use of LinkedIn's data does harm their ability to collect and sell it; I sure don't want hiQ mining shit about me from LinkedIn for the specific purpose for which they are doing so, so I'm less likely to continue maintaining my LinkedIn profile; I'm not alone in that, so yes, it does affect LinkedIn's business. Other factors include the purpose and character of the use (it's commercial, so it fails that test as well), the nature of the copyrighted work (time was spent collecting and compiling the data, time was spent designing the format and layout of the data, it's a creative work), and the amount and substantiality of the portion taken (all profile data is being scraped).

As for phonebooks, I actually raised that issue elsewhere. This isn't even a long thread; perhaps complete your research before assuming I don't know about fucking phonebooks, eh? As a patent lawyer, you should be really good at researching shit, shouldn't you? I mean, if you're good at your job, that is.

I get it, you're a patent lawyer with barely a passing familiarity with aspects of IP law outside of your specialization. Perhaps consult one of your colleagues specializing in copyright before replying again, though.

Re:then dont' make it public

[American Patent Guy]

Well, then. Post your "whole response" and perhaps I'll have something to respond to other than your sleep-addled insults. You haven't rebutted what I have said.

As anyone can download LinkedIn's data, HiQ is doing nothing special in the market. You're less likely to use LinkedIn because you've discovered that it can be used by anyone in a way you don't like. HiQ hasn't impacted the market by scraping it. Your analysis applies to the entire compilation. HiQ is downloading information for individual postings, which is not protected by copyright. (If HiQ was breaking into LinkedIn's servers and downloading the entire compliation, then your analysis would have some merit.)

If you posted such wisdom elsewhere, why not post it here? Is it so troubling for you, being so eager to make a point? (If you really had such a wise posting, that is.)

I get it. You're a person who thinks he understands copyright law, and wants to prove his expertise. Perhaps you should consult with one of your colleagues specializing in law (of any kind) before replying again.

Re:then dont' make it public

[AK Marc]

So the solution is to provide public APIs, and request scrapers use those, so the data access can be tracked and identified just like when humans and search engines use it.

If they make it public and predictable so search engines point to them, then they have given a robots.txt that allows that use, so it's "licensed" by the lack of controls, same as search engines.

But anyways, what about their user agreement?

The search engines never log in or agree to the user agreement, and this use seems to be a search engine that doesn't simply direct views to their page.

Re:then dont' make it public

[AK Marc]

I can go into the Louvre, sit in front of the Mona Lisa, and sketch an exact replica of it, down to the brush stroke (except for the fact that there are always people standing in front of it trying to take a selfie), then sell that copy. Wait, what was your point again?

Re:then dont' make it public

[AK Marc]

Then why didn't they file a copyright complaint? Instead, they are claiming "hacking" for viewing public information. (not copyright for using it, but "hacking" for viewing). Copyright is irrelevant, and not the complaint.

Re:then dont' make it public

[BronsCon]

You haven't rebutted what I have said.

You haven't read what I have written.

If you posted such wisdom elsewhere, why not post it here?

Because I did post it here, in this very thread. It's not my fault you've chosen not to read the thread in its entirety before replying to me, nor is it my responsibility to repeat everything I've ever posted here to every dumbass who can't scroll a page to find it himself.

Sorry, I reserve that level of service for my paying clients, not random armchair quarterbacks who claim to be lawyers yet can't do a simple review of what's already on the page they're looking at before replying.

Personally, I hope LinkedIn goes the copyright route on this, just so I can rub your nose in it when they win. They're not going to get anywhere with CFAA.

Re:then dont' make it public

[Wootery]

That something is 'in public' doesn't mean you're free to copy it.

Walk around a city and you might see countless TVs. That doesn't mean you're allowed to record them and sell the videos - that's still copyright infringement.

Re:then dont' make it public

[BronsCon]

But Google is doing something of which LinkedIn approves and has given Google permission to do.

Have they, though? Or have they simply not asked them to stop?

I understand that people who have never created anything of value or who believe strongly in socialism have no concept of ownership of property

Lovely assumption, but incorrect. I, in fact, have created quite a bit of value in this world. Just as a small sample, my clients value me enough to keep me employed long-term and my employees value the income and stability I provide them. So, then, you must think I'm a socialist? Why is that? Wait, no, you can't possibly think I have no concept of ownership of property when I've stated that LinkedIn has ownership of the data they've collected. Speaking of trying to follow along... So, then, I'm left wondering why you wrote this, as it certainly does not apply here.

Re:then dont' make it public

[BronsCon]

Look again, they're given a lot of explicit restrictions and a handful of explicit permissions. In Google's case those are limited to:
Allow: /psettings/guest-controls*
Allow: /psettings/guest-email-unsubscribe*
Allow: /psettings/sms-unsubscribe*
Allow: /psettings/guest-controls/retargeting-opt-out*
Allow: /settings/loid-email-unsubscribe-router*
Allow: /settings/loid-email-unsubscribe*
Allow: /help/

For reference, the first 6 are pages where one can unsubscribe from various forms of marketing and the last is LinkedIn's support section. Anything else Google indexes (and they have indexed a LOT of LinkedIn's content) is without explicit permission, possible even contrary to the 45 explicit restrictions they've been given. For example, I found this in Google's index, and /profile/ is listed as a Disallow rule.

Most of the search engines listed in that robots.txt have the same set of rules as Google. The only obvious exception is deepcrawl, which also has the following Allow rules:
# Profinder only for deepcrawl
Allow: /profinder*
Allow: /profinder/*

Re:then dont' make it public

[pnutjam]

Have you seen google's "snippets" they do scrape content from sites, sometimes alot.

Re:then dont' make it public

[AK Marc]

Facebook is letting someone in that they know is going to make a copy, then calling it "hacking" that a copy was made.

Even if the door is open, once the owner says "Do not enter", coming in is trespassing. Accessing the data once you have been asked not to is still illegal access.

More like:

Facebook has a used goods store. The sign says "open". Bob comes in, buys a chair, then re-sells it on eBay for twice as much. Facebook says that Bob is welcome to come in the shop, and buy things, but if he sells them, then he was trespassing when he bought the item.

Facebook is arguing that getting the data is legal, but the use of the data after retroactively makes the legal act illegal hacking.

You can't invite someone into your house, find out they took a smelly poo in the bathroom, the file charges against them because they trespassed to leave the poop, as you later claim that the invitation implied that there'd be no pooping. Facebook's door isn't just unlocked, when someone knocks (sends a get), Facebook opens the door and says "come in", then sues them for coming in.

Re:then dont' make it public

[kwbauer]

Because the Louvre does not hold a copyright on the Mona Lisa? Many other works of art are still under copyright so let's focus on those.

Re:then dont' make it public

[AK Marc]

The claim does not talk about copyright, so let's focus on the facts Facebook asserts.

SEO?

[Anonymous Coward]

I guess they don't care about being found on a search engine.

Scrape or Scrap?

[DontBeAMoran]

Because if it's not illegal to scrap their websites, black hat hackers will have a field day.

I've done several scraping projects

[GerryGilmore]

Using some add-on python packages it is ridiculously easy to scrape any web page, even those that use ASP (It's a PITA to get set up the first time, but...). The ONLY thing - aside from legal action, apparently - is to have a login mechanism in front. Without authenticating, it's no-go.

Re:I've done several scraping projects

[iggymanz]

hahaha, you imagine login is a cure?

no, scripts can log in. with sites having millions of users you can make as many logins as you need, it's a whack-a-mole the site can't win

Re:I've done several scraping projects

[im_thatoneguy]

You can have terms of service though on a login to make it easily illegal.

"By logging in you agree to not republish data that you view."

Re:I've done several scraping projects

[Gr8Apes]

That's not illegal, that's merely a violation of the user agreement.

Re:I've done several scraping projects

[Anonymous Coward]

Just keep in mind the language of the Computer Fraud and Abuse Act.

Permissions, login systems, and any technical specification or description of what you are and are not allowed to do not at all come into play.

Only the wishes and desires of the system operators, possibly taken long after the fact, are what get applied.

You really need a written contract showing the system operator both wishes you to have access, and is authorized to have their wishes upheld.

Simply giving you a login and a click-through screen saying you are allowed to use that login is not enough to reflect the wishes of the system operator and does not legally grant you access to anything under the CFAB.

Anything showing the implied wishes of the system operator may get you out of any claimed damages since it shows you were acting in good faith, but will still have them taken into account.

A contract is the only way to really defend your past access as legal.
Although it should be obvious in such a court case the contract will only cover the past, the date it was signed to the day of the claimed illegal access that is being argued in court.
If you continue to access their system after they tell you that you aren't supposed to, the time from that moment forward will still be considered as illegal and include damages.

Re:I've done several scraping projects

[Zero__Kelvin]

Which gives them standing in court. It *might* not be a crime but it creates a contract that doesn't exist without it. This is far from the first time a company has tried the old "The Internet doesn't work the same way for us as it does for the rest of the world. Callsies, no take-backs!" defense.

Re:I've done several scraping projects

[im_thatoneguy]

Not criminal but breach of contract is grounds for a civil cause of action.

Re:I've done several scraping projects

[Zero__Kelvin]

You forgot the "IANAL". I know this, because if you were you would also know about EULAs and other ways to enter into a contract, including verbal agreement.

Re:I've done several scraping projects

[Cajun Hell]

no, scripts can log in. with sites having millions of users you can make as many logins as you need, it's a whack-a-mole the site can't win

There's no rule that says getting login credentials needs to be trivial. Can you make a throw-away account at your bank?

LinkedIn can authenticate people if they want, assuming they don't mind having a barrier to entry that keeps people from using their site. But keeping people from using their site does seem to be the agenda item here...

Re:I've done several scraping projects

[Rockoon]

It *might* not be a crime

Breaking a contract is not a crime. Full stop.

Re:I've done several scraping projects

[Zero__Kelvin]

... and, start again. If the contract says that any breach of the contract means you are no longer allowed to access the system then continuing to access it after violation of the contract almost certainly *IS* a crime. A meatspace analogy would be if you are allowed to get merchandise and pay later, then violate the contract knowingly and take product from the warehouse anyway you have committed theft.

Re:I've done several scraping projects

[GerryGilmore]

Sure, scripts can login - if you have a valid login credential. If you do, obviously you can scrape away. If not, well....

Re:I've done several scraping projects

[h33t l4x0r]

Not if they verify with SMS, which I believe they do.

Re: I've done several scraping projects

[Zero__Kelvin]

You couldn't be more wrong.

Re:I've done several scraping projects

[Zero__Kelvin]

That's not correct. You directing your son to click the button is no different than you directing him to commit a crime. The culpability and responsibility rests with you. You will be held to the contract in the former case and charged with a crime in the latter. Great parenting though!

Re:I've done several scraping projects

[iggymanz]

not a deterrent, buying a block of cell phone addresses is common too (usually for telemarketing scam call origination)

Re:I've done several scraping projects

[iggymanz]

I think the volume of telemarketing scams originating in India (where they block block of cell phone addresses in the USA to use as origination point) proves there are plenty of fraudsters that don't give a crap about your words on paper ratified by a room of politicians.

Re:I've done several scraping projects

[iggymanz]

the fraudster/scammer in a 2nd or 3rd world toilet is shaking in his boots at the threat your country's legal system represents....HAHA!

Happens in other industries too

[ErichTheRed]

Airline websites have this same problem -- the online "cheap ticket" engines regularly scrape the publicly available data by essentially running the "book a trip" workflow millions of times to try to pull the entire set of fares for different city pairs. It's a cat-and-mouse game because the information has to be available for normal humans to book trips; no one is going to solve a CAPTCHA to look up fares. Basically these engines are looking for any irregularities like mis-filed fares or fares that happen to be a particularly good deal. (Airlines have to publish their fares in advance and make them available to online sources that are available to travel agents. This is why you'll occasionally see stuff like a transatlantic business class ticket for $50 or similar...)

I'm not sure if LinkedIn can actually bar someone from scraping their public data. If that was the case, no one could run wget on a website and pull down all the static content.

Re:Happens in other industries too

[shuz]

I have direct experience with this myself.

This is why companies like Akamai have products geared specifically for this problem. However stopping bots is nearly impossible unless you deal with them on a realtime basis. It would be interesting if Linkedin could get the entire world to make website scrapers illegal and then actually enforce that illegality. As of now when a bot owner is shutdown they just move the operation overnight to the ISP that will take their business in the same country or move countries all together. Likely the only way to stop this behavior is in the website transaction process or to make scraping not monetarily feasible.

Re:Happens in other industries too

[viperidaenz]

Wouldn't it just be easier to run your bots through multiple VPN's with endpoints in different countries?

Re:Happens in other industries too

[h33t l4x0r]

No offense but you are a complete noob if you're trying to scrape sites without connecting through proxies. LinkedIn will start sending 403's almost right away.

This is bonkers!

[Zobeid]

Here's why it seems bonkers to me. . . When you access a website, you are merely sending that site a request for information. That's all. Assuming it responds with the requested information, one must presume that's because the operator (and, by proxy, the owner) of the website set it up for that purpose. So what we have here is effectively. . .

LinkedIn: Don't request information from us!

hiQ: Please send the following information.

LinkedIn: OK, here you go.

LinkedIn: Dammit, you requested information after we told you not to! WE'RE GONNA SUE!!

Re:This is bonkers!

[bluefoxlucid]

Actually, LinkedIn has a point.

LinkedIn supplies service to the public at-large, in the same way that a MicroCenter supplies retail service to the public at-large. All members of the public are allowed to enter a MicroCenter. You walk up to the doors and they open automatically.

You can be trespassed for no reason by a retail center or other physical location open to the public at-large. The doors still open to you, but you're not allowed in. It's the same with a Web site: it's difficult in-practice to establish a verifiable packet identity on the Internet. IP addresses change, and you can do goofy shit like put the data scrapes in AJAX requests to distribute their source.

In other words: you're by default authorized to access LinkedIn's public assets. You're not allowed to access stuff requiring a logged-in session until you've gotten log-in credentials, because there are actual systems in place to stop you from doing that, implying that you're not supposed to force access there. Basically, civilized understanding of the expectations of your host on the face.

If LinkedIn tells you to stop, you've now had your authorization revoked. You can't claim a restraining order is invalid because someone's outside and you can also be anywhere outside, and you also can't claim that LinkedIn can't de-authorize you unless they specifically identify and block you. Blocking an individual entity from a Web site is hard and has collateral damage.

So the CFAA is actually a valid vehicle here, since "abuse" is essentially defined as "accessing a system to which you are not authorized." The reasonable person test holds up a lot of behavior, largely because it's unreasonable for a person to determine if a certain behavior or function on a Web site might not be something they're allowed to touch, or whatnot, given the reasonable behavior of people at-large. A lot of stuff happens that won't pass CFAA as fraud or abuse, even though it's inconvenient and unintended. By the same token, when somebody has told you to stop accessing their systems in a certain way and you do it anyway, a reasonable person might assume you were, you know, told not to, and not allowed to do that, and that you know damned well you're not allowed to do that.

That's not to say threats, lawyers, and other anti-social behavior are good business. Poor diplomacy here. Effective in the legal field, but not your best option.

Re:This is bonkers!

[Train0987]

Then blacklist IP's at the firewall(s) for endpoints that are scraping your site.

Re:This is bonkers!

[Zero__Kelvin]

That is exactly the situation, except add "I'm not telling you who I am, and have not entered into a contractual relationship with you." Linked in *could* reply with "Sorry, you must tell us who you are and enter into a contractual agreement with us in order for us to send that information ", but they don't. So the only way LinkedIn or anyone else wins this case is with an ignorant judge or one who has been bought. If the defense does their job only the latter is possible.

Re:This is bonkers!

[tattood]

Then blacklist IP's at the firewall(s) for endpoints that are scraping your site.

IP addresses are fairly easy to change. You can use something like TOR, so your public IP always changes.

Re:This is bonkers!

[bluefoxlucid]

Let's try this again.

it's difficult in-practice to establish a verifiable packet identity on the Internet. IP addresses change, and you can do goofy shit like put the data scrapes in AJAX requests to distribute their source.

Blocking an individual entity from a Web site is hard and has collateral damage.

Wikipedia has tried this, with collateral damage and limited success. I've seen people get sent to jail for harassment and legally barred from accessing certain sites and systems under restraining order, and then continue to access them with no reasonable way to prove their identity (i.e. could be someone else pretending to be said person).

These days, it's different. Those IP addresses are probably automatically-assigned or internal to cloud infrastructure. IAAS may share addresses across clients. The IPs may appear from a range of hundreds of subnets coming from auto-scaling AWS infrastructure, constantly provisioning and releasing addresses.

In other words: "Block it at the firewall" can easily mean "Block everything coming from AWS, Azure, DigitalOcean, and all other data centers all over the world." Difficult (nigh-impossible) and prone to huge amounts of collateral damage.

Then: the courts have already told you this is a matter of you having a public Web site, and you can deal with them "accessing" it yourself because you apparently have no right to tell people they're not allowed in due to their use of your published information. Now you have people jumping from address to address, and you're forced to play firewall whack-a-mole.

Re:This is bonkers!

[Ichijo]

Except when you enter a MicroCenter, you are stepping foot on their property. When you anonymously request a public web page from a web server, you're standing on the public sidewalk at the walk-up window. Since you as a taxpayer own that sidewalk, can the store owner restrain you from your own property as a way to make you stop placing orders at the window?

From TFA:

[Orin Kerr, a legal scholar at George Washington University] argues sites wanting to limit access to their site should be required to use a technical mechanism like a password to signal that the website is not, in fact, available to the public.

By making data publicly available to individuals to help LinkedIn gain exposure but trying to restrict it from other web sites who wish to use that same data for financial gain, it seems LinkedIn is trying to have their cake and eat it too.

Re:This is bonkers!

[viperidaenz]

A trespass notice can't stop you looking at MicroCenter from a public space.

If you want to restrict someone in a public space, you need a restraining order from a judge.

Transferring that to the internet, a cease and desist letter is like a trespass notice. Probably appropriate for telling someone to stop creating new logins to access restricted content after you disable their old ones.

Asking a judge for an injunction would be appropriate to stop someone accessing publicly available content. Of course, this is never going to stop someone from another jurisdiction.

Re:This is bonkers!

[bws111]

Many stores have doors that you can open by pushing a button. Assuming the door opens, one must presume that is because the management (and, by proxy, the owner) of the store has set it up for that purpose. So what we have here is effectively..

Store: You have been banned from this store. Do not come back
You: Push the button
Store: Door opens, you go in
Store: We told you to stay out, we're having you arrested for trespassing

This, of course, happens all the time (except for the idiotic assertion that the door opener somehow granted you permission to enter).

In both cases, the DEFAULT position is that you have access. You may request information, you may enter the store. However, once you have been TOLD you do not have permission, then you DO NOT have permission. At that point it is YOUR responsibility to not access what you have been explicitly told you may not access. If you ignore that, you may find the applicable laws (trespassing, CFAA) used against you. And you will lose, because you have no defense. 'But they let me...' is not a winning defense, ever.

Re:This is bonkers!

[Gavagai80]

Trying to make it illegal to scrape the data is beside the point -- what linkedin really wants to do is prevent others from publishing the data. Just because you can find a book in the library and the book doesn't fire lasers at your eyes to blind you and stop you reading it doesn't mean you have permission to sell your own book which consists of photocopies of that book with a few small changes.

Re:This is bonkers!

[Frosty Piss]

LinkedIn supplies service to the public at-large

OK, there's where you're wrong.

Re:This is bonkers!

[American Patent Guy]

You're not allowed to access stuff requiring a logged-in session until you've gotten log-in credentials, because there are actual systems in place to stop you from doing that, implying that you're not supposed to force access there.

Actually, if the scraper used a valid username and password (or other valid credentials) to gain access, access was authorized. It might have violated a user agreement perhaps, but that's a separate civil matter. The Computer Fraud and Abuse Act specifies criminal acts that a private entity (like LinkedIn) can't use as a basis for its suit.

Re:This is bonkers!

[bluefoxlucid]

When you request a public Web page, you're accessing and using their machinery.

My entire argument was "available to the public" versus "except you; you get the hell out right now." A technical mechanism is infeasible: if they want the data to be publicly-viewable and don't want people to do certain things, then a password doesn't work; and firewalls and the like will have to contend with modern global, auto-scaling, IP-changing data centers where you can't just single out a particular actor by IP address or any other identifier.

Legal scholars can argue whatever they want, while other legal scholars come behind them and argue that they're wrong. That's kind of what lawyers do. Your argument essentially claims that anything they put up is a public property and that they can't have any terms of use for anonymously-accessible assets. Considering I can ping your server anonymously, a DDoS would be legal under your reasoning--without even going to reducto-ad-absurdium.

Re:This is bonkers!

[bluefoxlucid]

Actually, transferring that to the Internet, you have to walk into the MicroCenter and turn the display around, then go back outside the window and look at it again to get a view of what's there. Every time you want to see it, you have to walk inside, fiddle with things, then walk back out.

You do know that nothing is actually "on the Internet", right? Do we need to explain to you how the Internet works?

Re:This is bonkers!

[bluefoxlucid]

The point wasn't that they used a password; there was a further point down that LinkedIn had de-authorized them from non-password-protected mechanisms: they told them they're now specifically not allowed to do that, which means they're not.

Imagine if you ssh'd to a bank's accounting system across the 'net and found that it just lets you log in as root, no password. Is that also legal?

Re:This is bonkers!

[Ichijo]

Considering I can ping your server anonymously, a DDoS would be legal under your reasoning...

Starting from step 1 (assembling a botnet), how do you organize an effective DDoS using purely legal means?

Re:This is bonkers!

[bluefoxlucid]

Arranging the infrastructure wouldn't be necessarily-legal. The ownership and control of the source of the attack may be illegal. The attack itself, however, would not constitute an additional illegal act beyond just having a botnet and sitting on it.

Likewise, if I just spun up several thousand micro-servers in AWS spread across all data centers and smashed the shit out of your site with requests declaring their source to a dead IP in the same subnet, I could have your servers pump out tons of HTTP responses with large images and other assets and kill itself. Totally legal, right?

Re:This is bonkers!

[Ichijo]

That sounds like an expensive way to run a DDoS. Has it ever happened in real life?

Re:This is bonkers!

[viperidaenz]

No, you'd need to yell from the street "Hey, what's in your store MicroCenter?" and they would yell right back what ever the the hell they want to.

You send a request for information and receive a response. You don't move anything around or change anything.

Re:This is bonkers!

[bluefoxlucid]

So, if I hack into your bank account, I'm just standing outside your bank yelling, "Hey, can you give me some money? Maybe viperidaenz's? He's got some!" and they're handing me cash, and I haven't done anything wrong?

Re:This is bonkers!

[bluefoxlucid]

No because it would be illegal. People do pay thousands of dollars to borrow an illegal botnet for DDoS, though.

Re:This is bonkers!

[viperidaenz]

Depends if you did it by bypassing a security measure, or by falsifying your identity.

Crawling a website doesn't require any of that. Google and Bing don't get in trouble for it. They both use the data they collect to make money. Not really any different to what this company is doing.

However, sending usernames and passwords in headers or in post data that are not yours is a little bit illegal. Lying about who you are to get something in return is "obtaining by deception"
Additionally, sending your own username and password means you've explicitly agreed to a contract to obtain them in the first place. Using that service against the terms of your contract opens up your liability.

CFAA does not apply

[Anonymous Coward]

If you read the act, it is clear that it applies to financial and government systems. It has not been tested in court that the CFAA covers violating terms of use. You really only need to go to some basic contract law to deal with people accessing your site in a way you do not like, and copyright law for distributing your material without permission.

Re:CFAA does not apply

[russotto]

It has not been tested in court that the CFAA covers violating terms of use.

Yes, it has, but only in the Central District of California as far as I know. The interpretation that the CFAA covers violating TOS was found to be overbroad in U.S. v. Drew, 259 F.R.D. 449 (C. D. Cal. 2009).

Exactly!

[Anonymous Coward]

I refuse to use any social media site including LinkedIN. A lot of companies - such as Goodwill - recruit exclusively from LinkedIN. Fuck'em.

I don't work for any company that uses social media for recruiting.

You know it is not that simple.

[Anonymous Coward]

They DO want it read, by the end-users who consume it and don't resell it. They DON'T want it read by aggregators who profit from it (and especially in a way that drives users to restrict their use of the service).

There is no solid technical solution for distinguishing between these two classes of user. So businesses are using the law to draw that distinction instead.

I personally think the "don't scrape" approach is totally backwards. They should take a "don't redistribute" approach instead. But logic commonly loses to wealth in the real world.

what about wifi scanning just looking for ssid's

[Joe_Dragon]

what about wifi scanning just looking for ssid's is on by default on many os's

Give all a bit of trust and get ripped off

[Trax3001BBS]

I refer to the Robot.txt used to tell search engines what's out of bounds. http://www.searchtools.com/rob...

Re:Give all a bit of trust and get ripped off

[HornWumpus]

But they want to be indexed by Google, just not by they company that tells employers their staff is looking.

The solution is just to never, ever, stop looking. Even if you love your job, having a current resume on Linkedin will get you better raises.

Re:Give all a bit of trust and get ripped off

[Mandrel]

But they want to be indexed by Google, just not by they company that tells employers their staff is looking.

A robots.txt file can state which HTTP User Agent strings are allowed. For example, Slashdot only allows access by certain search engines. If you're starting a new one, you have to misrepresent yourself, or you're buggered. The question is when such misrepresentation is legal and moral, and whether it is instead up to sites to more accurately detect who they want to serve, and serve errors to those they don't.

The solution is just to never, ever, stop looking. Even if you love your job, having a current resume on Linkedin will get you better raises.

Again it pays to be the selfish squeaky wheel. The basis of advertising.

They just went about this the wrong way

[93 Escort Wagon]

Now if LinkedIn had instead posted "ecto gammat", all the nerds would be in their corner.

Pot, Kettle.

[AnotherBlackHat]

LinkedIn's whole business model is "scraping" information from people. It's not like they pay people to enter that information.

When CDDB tried this sort of B.S. it led to FreeDB. Maybe LinkedIn being assholes will lead to something similar.

HiQ

[tylersoze]

Can we talk about what HiQ is doing with the data for a sec? "HiQ scrapes data about thousands of employees from public LinkedIn profiles, then packages the data for sale to employers worried about their employees quitting" I mean WTF?

Re:HiQ

[American Patent Guy]

Like it or not, if you (or an employee in your example) choose to publish information about yourself in a publicly-accessible place, then you've voluntarily relinquished whatever privacy rights you had in that information. Whatever you believe about HiQ, they are only organizing and re-releasing public information. LinkedIn has no copyright in it (as they didn't create the data, nor is it a work of authorship), and they were complicit in the act by delivering it up upon request.

It's MY data, not LinkedIns

[Anonymous Coward]

Curious how GDPR will or could play here. If the Europeans have it right, a profile isn't LinkedIns. It's the collective people who put their info there.

I'd suggest, a flag that the user can set, which says 'make x' of my profile indexable.

No standing

[American Patent Guy]

The Computer Fraud and Abuse Act is part of the Federal Criminal Code, and no private entity can use it to bring a suit. A prosecuting attorney for the government could make a criminal charge, but LinkedIn would have to persuade him/them to take that act. This is much ado about nothing.

Wrong!

[www.sorehands.com]

The CFAA applies immediately or when the defendant (or defendant to be) exceeds the permitted access. This could be also through a cease and desist letter. See Facebook, Inc. v. Power Ventures, Inc., No. 13-17102 (9th Cir. July 12, 2016) https://cdn.ca9.uscourts.gov/d...

You are permitted to grant different people different terms or access. Look at https://qz.com/981029/a-federa...

Re:Wrong!

[russotto]

Didn't read it, did you? It does say that a cease&desist can trigger the CFAA. It does not say "The CFAA applies immediately or when the defendant (or defendant to be) exceeds the permitted access". In fact, it specifically says violation of terms of use cannot trigger liability under the CFAA.

Re:Wrong!

[BronsCon]

Precisely this. Especially important in the case of Facebook or LinkedIn, which you have to actively avoid if you wish to not access them.

That would also make search engines illegal

[Kreigh]

If you don't want your content to be scraped, don't make it publicly available. If I can view it without a logon you have no claim to protect the data.

Re:A little concerned...

[Alok]

My view: If you are on the Internet, then you provide your data at your own expense. I run my own websites this way and it is working out just fine for me.

How much data do your websites have, and approximate number of people and other entities who might be interested in downloading it? Setting aside the point that you don't distinguish between public vs private data, or the method of consumption (web surfing vs scraping to sell to others); just try to calculate the bandwidth requirements if LinkedIn or any highly popular site were to actually do this.