Should You Break TOS Because Work Asks You?

An anonymous reader writes "My boss recently assigned me a project that was all his idea, with two basic flaws that would require me to break multiple web sites' Terms of Service (TOS). Part requires scraping most of the site, parsing the data and presenting it as our own without human intervention. While we're safe on copyright issues, clearly scraping like this is normally not allowed. At times it might also put a load on those sites. The other is, for lack of better words, a 'load balancing' part that requires using multiple free accounts instead of purchasing space and CPU time for less than $2,000 USD per month. The boss sees it as 'distributed' computing when in reality it's 'parasitic.' My question is: am I wrong about the ethics? If I do need to walk, how best can I handle it without damaging my reputation and future employment opportunities?"

Comments

You're Right, Of Course

[eldavojohn]

My question is am I wrong about the ethics?

You don't even have to ask that question, this isn't even one of those interesting cases or gray areas. What you're planning to do is wrong--even though you could probably escape any legal ramifications. It sounds pretty clear that this site creates profit from these overly priced accounts for information that you obviously value at some amount. Getting it for free (regardless of the TOS) could put you at some risk for litigation. Using the term "load balancing" or even "distributed computing" is hilariously misplaced here.

If I do need to walk how best can I handle it without damaging my reputation and future employment opportunities?

Look, I understand what's it like to be looking for a job when the economy is bad. If there are forces keeping you pinned to this employer, I don't know of them. What I would retort with is "How can you keep working this job without damaging your reputation and future employment?" I mean are you going to put in your resume that you coded a technically innovative but bandwidth stealing parasitic botnet to duplicate content from a website that asks for a monthly payment to normally access it at that volume?

I would suggest you propose the $2k/month route and if your boss balks at it, start interviewing with other companies. If you have to leave and you're worried about being blacklisted as a 'whistleblower' (and your boss just might be that kind of guy) then tell him it's for monetary reasons that you're leaving and wish him the best of luck in his future scams.

Re:You're Right, Of Course

[SatanicPuppy]

Should be. It depends on what kind of data they're downloading, and whether they're just crawling link by link and hoovering up everything, or whether they're looking for something specific.

Either way, spiders and scrapers usually have programmed scan intervals which have no relation to an actual human's browsing...or they just hit the page as hard as they can, but that is so easy to block that almost no one does it that way. Even if they add a little randomness, it's only efficient to run a scraper if it's hitting every few seconds at max, and even the most ADD user won't keep that up.

Ironically, the easiest way to nail 'em is to put up a subset of "no robots" pages; if the robots crawl those pages, blacklist 'em. Every legitimate spider will respect those files.

Otherwise, if you're running a site with a ton of data, and something is crawling it sequentially, you can absolutely redirect their queries to whatever you want. I'd be wary of doing something cute (if you can call goatse "cute") for fear that you'll have an occasional false positive and redirect a user from a high bandwidth location to that site.

Re:You're Right, Of Course

[GrpA]

I've also had similar requests in the past, and in both cases I did the work. I considered the request, decided they were ethical (even if somewhat unusual) and so did it. That's something you're going to have to figure out for yourself - whether you're going to do it or not.

I've been on the other side of the fence also...

If you're relying on data for commercial use, putting yourself in a position where you need that data is a risky thing...

I had a scraper once come after me. I caught them - as the previous poster pointed out, it's easy... I didn't block them. I captured and redirected their requests so I could control what they got and, well, sent them some information that made them look really, really stupid. They were angry, but there wasn't much they could do.

They were just enthusiasts - they had no business risk in their application suddenly failing.

Let your boss know the risk he is facing and then ask him if he really wants to risk being caught and shut down unexpectedly, or worse, finding someone has poisened his data.

It's just not good for business.

GrpA

Re:You're Right, Of Course

[cerberusss]

I'd advise against discussing it with HR. I've encountered the following situation: I talked to a HR manager about something that obviously should've remained confidential. However that same HR manager was part of the management team and thus had two hats on. She proceeded to inform the management team, to my astonishment.

I've come to the conclusion that HR is just a staff department and owes allegiance to, you guessed it, the management team. Not you.

Re:You're Right, Of Course

[level4]

Definitely possible!

Any company with a website that contains "regularly updated data that might be interesting for competitors" has probably already got some kind of anti-scraping system in place. This guy's boss thinks he's being clever and original - of course he's not, any company with a site of any value and popularity has already seen this a million times.

What they return basically depends on the mentality of those who work there. The "by the book" professional types will just blackhole the IP or return a "too many visits from this IP" page.

Companies with a more BOFH type guy in charge might very well start "playing" with the data. Instead of the "too many visits" page you might find yourself getting a page with some of the data changed around randomly. Believe me, there are *many* people around who think it is just the height of comedy to fuck with people who are basically stealing their stuff anyway.

They will turn it into a game - and, when the erroneous data turns up on the thieving web site (if that's what this guy's company is running), a few screenshots of that site with the modified data suddenly becomes pretty good evidence in a court, if they're of the "legal remedy" persuasion.

Scraping data is a last resort, not the first thing you try. Forget the ethics - the fact he's working for a company willing to be that insanely cheap and stupid in the first place should be a signal to run far, far away in itself.

Re:You're Right, Of Course

[Chapter80]

The proper way to document this in email is something like this:

Boss-
I'm able to do the data scraping and should have it up and running by the end of the day.
- Your faithful employee

In case you are wondering about the technical details, here they are:

The scraping is implemented with a perl script which is activated using cron.

We scrape the site twenty times per minute, which is a violation of their terms of service. By doing this, of course, we risk that they may shut us off at any time, or even provide us with fake data.

The typical PHB will read the first two lines on his blackberry, and you're golden. Worst case he or she will scroll down - but the managerial brain is set to shut down at the word "perl". The word "cron" is a failsafe - in case the PHB also has ADD.

Later when s/he comes back and says "why didn't you warn me", you can point to the text "beneath the fold" of your email.

Re:You're Right, Of Course

[SatanicPuppy]

The example that would leap to my mind is a number of services that allow you to "map" an ip address to a geographic location...I use one of those for my job search homepage, and it only allows ~200 queries a day for the "free" account...It would be plenty useful to have as a free service (targeted advertising), and if you set up enough "free" accounts, you could use it that way.

Since I'm doing all my job searching away from where I'm currently living, I use mine to make sure that my job searching page always looks "under construction" for people who live where I live. My boss actually checks it occasionally, I guess to make sure I'm not trying to leave.

If you want legal advice...

[Anonymous Coward]

...ask a lawyer.

Re:If you want legal advice...

[dintech]

No, no, no. If you want legal advice, ask Slashdot! Given enough time, you'll get an answer that is exactly what you hoped for and you can ignore or mod down the ones you don't like. It couldn't be simpler and it's a whole lot cheaper!

Just ask yourself this:

[Anonymous Coward]

Did the contractors on the Death Star deserve to die?

Re:Just ask yourself this:

[Rennt]

Haven't you seen "Clerks"

Dante: My friend is trying to convince me that any contractors working on the uncompleted Death Star were innocent victims when the space station was destroyed by the rebels.
Blue-Collar Man: Well, I'm a contractor myself. I'm a roofer... (digs into pocket and produces business card) Dunn and Reddy Home Improvements. And speaking as a roofer, I can say that a roofer's personal politics come heavily into play when choosing jobs.
Randal: Like when?
Blue-Collar Man: Three months ago I was offered a job up in the hills. A beautiful house with tons of property. It was a simple reshingling job, but I was told that if it was finished within a day, my price would be doubled. Then I realized whose house it was.
Dante: Whose house was it?
Blue-Collar Man: Dominick Bambino's.
Randal: "Babyface" Bambino? The gangster?
Blue-Collar Man: The same. The money was right, but the risk was too big. I knew who he was, and based on that, I passed the job on to a friend of mine.
Dante: Based on personal politics.
Blue-Collar Man: Right. And that week, the Foresci family put a hit on Babyface's house. My friend was shot and killed. He wasn't even finished shingling.
Randal: No way!
Blue-Collar Man: (paying for coffee) I'm alive because I knew there were risks involved taking on that particular client. My friend wasn't so lucky. (pauses to reflect) You know, any contractor willing to work on that Death Star knew the risks. If they were killed, it was their own fault. A roofer listens to this... (taps his heart) not his wallet.

Kevin Smith knows his stuff.

Hilarity ensues when...

[Anonymous Coward]

...you build a system that closely relies on this nonstandard (and unsupported) method of getting information, they change it and it breaks.

Either by accident, or because they spot a load of particular access patterns from your address, figure out what's going on and intentionally break it.

Uh...

[Anonymous Coward]

No. By your own admission you think its wrong. Next?

Sigh

[MyLongNickName]

Okay, this one is simple. You know what is right and what is wrong. The reality is that 99% of the folks will do what the boss asks without even raising a fuss. The reality is that you will be damaging your career if you don't go ahead.

Now, the other reality is that shit flows downhill. That is, if this project gets questioned, the boss will claim ignorance, and put the blame on you. Your job is to cover your ass.

Email is a good documentation tool. "Clarify" the request, asking if this is what he intends for you to do. Remove the emotion. Put in only facts. Put in a piece about your not being sure, but this may be a violation of terms of service. Ask if he wants you to proceed. Forward your sent email to a personal account.

By the book. This one is so simple that it should be in the FAQ.

Re:Sigh

[MyLongNickName]

You bring up a good point which leads to lesson #2: Written trumps verbal. If shit hits the fan, you halve your email. if your boss then says that he verbally told you not to proceed, you only have to say that you have no recollection of any such conversation. He is on the defensive as he has nothing to back it up. If he was "appalled" at the thought of breaking the TOS, then he would have written back and clarified.

Now, if you want to double cover your ass, give him status reports via email. Ask questions. You are covered.

Now to answer some other questions about whether to quit or not. You have to make that decision on your own. For screen scraping, I wouldn't quit over something so mundane. Sorry. Especially if you are a grunt. You voice your concerns, and go on. The reality is that 4 times out of 5 if you voice your concerns like this in a written manner, that the boss will back down. I have faced it twice in a grunt position with two different managers, and both times I got thanked for bringing it to their attention. It is all in how you deliver it. If it comes across as "I am ethical and you are a piece of shit", then your career is hurt. If it comes across sa "I am trying to look out for your well being and that of the company", it can be a positive. Wording is everything.

Spammer logic.

[argent]

If you can access it, it was designed to be accessed.

So you're totally behind email spam, you don't think spam should be considered unethical, let alone made illegal?

Re:Spammer logic.

[d3ac0n]

I don't think spam should be any more illegal than billboards, flyers, or direct mailings.

The flaw in this argument is that your three counter-examples (Billboards, Flyers and Direct mailings) are paid for entirely by the SENDER. IE: Billboards are paid for up-front before they are mounted, Flyers and direct mailings have printing costs paid up-front and delivery costs (either the local govt. mail service or paid people to manually give it to you) paid up-front as well.

Spam, on the other hand, is largely delivered on the backs of OTHER payers. Both through the incredibly high bandwidth costs (HOW much of the total Internet traffic is Spam now?) and through ancillary costs such as costs for software and hardware to filter Spam out, and human costs in terms of work-hours wasted manually going through spam. Not to mention the costs to people and networks infected with Spam botnets.

This is what makes Spamming SO profitable, and why it won't go away. Because the costs for Spam are decentralized to millions of people otherwise not directly involved, even a return as little as .01% will turn a HUGE profit. This just doesn't work in the regular advertising world. It's also why it's Illegal is several countries now. It is essentially stealing service from millions of other people and generating millions of dollars of expense for hundreds of companies around the globe, for what are largely scamming and phishing operations.

Get it?

Good.

Check with Compliance Officer/Department

[Ohmaar]

I work in health care, so maybe it's different in your industry, but every hospital I've worked for has had a compliance officer with an anonymous 800-number for compliance questions. This is DEFINITELY the kind of stuff they want to know about.

Why are you asking *US*?

[elrous0]

Only YOU can decide how far you're willing to go for your job. You're essentially asking us what your own ethical limits are.

Re:Anything on the web is available for access

[BadAnalogyGuy]

Babies really shouldn't be given candy in the first place.

Hey, anonymous! This is your boss.

[jollyreaper]

I told you to scrape Slashdot, not read it. Now get back to work!

one approach

[buddyglass]

1. Tell your boss it's a bad idea to break these websites' terms of service. He'll probably override you and tell you to do the project anyway.
2. Code up the project just like he asks. Demonstrate that it works.
3. Shortly afterwards, email the sites in question from a non-work friend's account and let them know (with specific information) the accounts and IP addresses that are violating their terms of service. Hopefully the accounts will be disabled, and/or your employer's IP range will be blocked.
4. Throw up your hands and tell your boss, "Well, I guess they figured out what we were doing!"

Re:Redirecting content

[Mr. Droopy Drawers]

Reminds me of a time when an Ebay'er was pointing to images on my website for an automotive auction. Didn't ask us or give us credit for the images. So, his example of "recently restored examples" became a photo of a '63 Imperial being loaded into a crusher.

How's that for Crushing the Competition?!

Re:Redirecting content

[Free the Cowards]

Somebody once pointed at a picture of a frosted birthday cake on my web site from a forum. So I grabbed my image editor and built a special edition of the cake just for him, where the frosting read "Don't link to my images!"

I also have a specially crafted JPEG which is under 1000 bytes but which produces a 20,000x20,000 pixel image filled with black. It will totally screw up the layout of any page linking to it if they haven't entered an explicit size for the tag.