Should You Break TOS Because Work Asks You?

An anonymous reader writes "My boss recently assigned me a project that was all his idea, with two basic flaws that would require me to break multiple web sites' Terms of Service (TOS). Part requires scraping most of the site, parsing the data and presenting it as our own without human intervention. While we're safe on copyright issues, clearly scraping like this is normally not allowed. At times it might also put a load on those sites. The other is, for lack of better words, a 'load balancing' part that requires using multiple free accounts instead of purchasing space and CPU time for less than $2,000 USD per month. The boss sees it as 'distributed' computing when in reality it's 'parasitic.' My question is: am I wrong about the ethics? If I do need to walk, how best can I handle it without damaging my reputation and future employment opportunities?"

Comments

You're Right, Of Course

[eldavojohn]

My question is am I wrong about the ethics?

You don't even have to ask that question, this isn't even one of those interesting cases or gray areas. What you're planning to do is wrong--even though you could probably escape any legal ramifications. It sounds pretty clear that this site creates profit from these overly priced accounts for information that you obviously value at some amount. Getting it for free (regardless of the TOS) could put you at some risk for litigation. Using the term "load balancing" or even "distributed computing" is hilariously misplaced here.

If I do need to walk how best can I handle it without damaging my reputation and future employment opportunities?

Look, I understand what's it like to be looking for a job when the economy is bad. If there are forces keeping you pinned to this employer, I don't know of them. What I would retort with is "How can you keep working this job without damaging your reputation and future employment?" I mean are you going to put in your resume that you coded a technically innovative but bandwidth stealing parasitic botnet to duplicate content from a website that asks for a monthly payment to normally access it at that volume?

I would suggest you propose the $2k/month route and if your boss balks at it, start interviewing with other companies. If you have to leave and you're worried about being blacklisted as a 'whistleblower' (and your boss just might be that kind of guy) then tell him it's for monetary reasons that you're leaving and wish him the best of luck in his future scams.

Re:You're Right, Of Course

[jimicus]

How is splitting and allowing the work to be done by someone else to do any more ethically sound than doing it yourself?

At the risk of invoking Godwin's Law so early on, how is taking this approach any different from saying to yourself "I'm just following orders"?

Re:You're Right, Of Course

[SatanicPuppy]

Should be. It depends on what kind of data they're downloading, and whether they're just crawling link by link and hoovering up everything, or whether they're looking for something specific.

Either way, spiders and scrapers usually have programmed scan intervals which have no relation to an actual human's browsing...or they just hit the page as hard as they can, but that is so easy to block that almost no one does it that way. Even if they add a little randomness, it's only efficient to run a scraper if it's hitting every few seconds at max, and even the most ADD user won't keep that up.

Ironically, the easiest way to nail 'em is to put up a subset of "no robots" pages; if the robots crawl those pages, blacklist 'em. Every legitimate spider will respect those files.

Otherwise, if you're running a site with a ton of data, and something is crawling it sequentially, you can absolutely redirect their queries to whatever you want. I'd be wary of doing something cute (if you can call goatse "cute") for fear that you'll have an occasional false positive and redirect a user from a high bandwidth location to that site.

Re:You're Right, Of Course

[mea37]

And if you ever wondered, when <insert crisis here> broke, how things could go so horribly wrong... it's because of people who think like this guy.

"Don't worry if it's the wrong thing to do; just document that it wasn't your idea!" And apparently never mind the idea of personal responsibility.

When there are no negative consequences for doing the right thing, ethics is mostly a curiosity. Ethics exist to guide you when the right path isn't easy. And yes, you are personally responsible for your own ethical behavior, regardless of whether someone with a bigger paycheck -- or even someone who signs your paycheck -- says otherwise.

Does it mean you have to walk? That depends on your boss. If you do, the best way to preserve your reputation is to avoid mud-slinging. Your current employer might want to try to harm your reputation, but it's extremely unlikely he'll get far (certainly not without exposing himself to legal liability). So just don't shoot yourself in the foot by ranting about the situation in interviews, etc.

Re:You're Right, Of Course

[level4]

Definitely possible!

Any company with a website that contains "regularly updated data that might be interesting for competitors" has probably already got some kind of anti-scraping system in place. This guy's boss thinks he's being clever and original - of course he's not, any company with a site of any value and popularity has already seen this a million times.

What they return basically depends on the mentality of those who work there. The "by the book" professional types will just blackhole the IP or return a "too many visits from this IP" page.

Companies with a more BOFH type guy in charge might very well start "playing" with the data. Instead of the "too many visits" page you might find yourself getting a page with some of the data changed around randomly. Believe me, there are *many* people around who think it is just the height of comedy to fuck with people who are basically stealing their stuff anyway.

They will turn it into a game - and, when the erroneous data turns up on the thieving web site (if that's what this guy's company is running), a few screenshots of that site with the modified data suddenly becomes pretty good evidence in a court, if they're of the "legal remedy" persuasion.

Scraping data is a last resort, not the first thing you try. Forget the ethics - the fact he's working for a company willing to be that insanely cheap and stupid in the first place should be a signal to run far, far away in itself.

Re:You're Right, Of Course

[AndersOSU]

You're absolutely right. The problem is being right, like being ethical, doesn't put food on the table.

The reason crises happen is three fold, first people with power see a competitive advantage in acting unethically, second people in charge or monitoring unethical/illegal behavior aren't up to the task, and third people tasked to do the work don't raise bloody hell when asked to do anything unethical.

In order to solve the problem you only need to fix one of those. The problem is, the first two options involve convincing people to act against their personal interests. People contain a remarkable survival mechanism, the ability to justify and rationalize difficult actions. Going after people who stand to gain by acting unethically is the business equivalent of abstinence only education.

Re:You're Right, Of Course

[interiot]

I really agree with this. If someone is already going to the effort of writing a lot of scraping code, it's already worth it to them to buy one of those $10-15/month shell accounts online that have SSH access. SSH gives them the ability to forward local TCP requests to that remote IP, their scraping app just has to have the ability to use a SOCKS proxy. This means scrapers have a proxy IP that 1) doesn't show up on any of the open-proxy DNSBLs, and 2) is fast and reliable enough for them to get real work done. And if you block them, they just pay another $10-15 to get another reliable IP.

Re:You're Right, Of Course

[Clover_Kicker]

Yeah, subtly wrong data is a million times worse then goatse, the scraper might not notice for weeks or months...

If you want legal advice...

[Anonymous Coward]

...ask a lawyer.

Short answer... "no".

[argent]

If your boss asks you to do something illegal, don't. If he doesn't agree, you should probably be looking for a new job, already. If he's willing to play these kinds of games with another company, what makes you think he won't do the same to you?

Uh...

[Anonymous Coward]

No. By your own admission you think its wrong. Next?

Sigh

[MyLongNickName]

Okay, this one is simple. You know what is right and what is wrong. The reality is that 99% of the folks will do what the boss asks without even raising a fuss. The reality is that you will be damaging your career if you don't go ahead.

Now, the other reality is that shit flows downhill. That is, if this project gets questioned, the boss will claim ignorance, and put the blame on you. Your job is to cover your ass.

Email is a good documentation tool. "Clarify" the request, asking if this is what he intends for you to do. Remove the emotion. Put in only facts. Put in a piece about your not being sure, but this may be a violation of terms of service. Ask if he wants you to proceed. Forward your sent email to a personal account.

By the book. This one is so simple that it should be in the FAQ.

Re:Sigh

[realisticradical]

Really? Were you working for the mafia?

That kind of policy makes me think of companies like Enron, Arthur Anderson, or the cigarette companies. Any company that would fire people for trying to protect themselves from company sponsored illegal or unethical activity must be engaging in lots and lots of it. The best part it it sounds like the policy is simply a CYA policy to protect those at the top.

The part I don't understand though is how did they differentiate between CYA emails and actual questions about projects? If, "Sorry boss, I was a bit confused at the meeting we concluded that I should go about the project by [illegal activity]" gets you fired how did they ever get anything done?

Re:Sigh

[MyLongNickName]

You bring up a good point which leads to lesson #2: Written trumps verbal. If shit hits the fan, you halve your email. if your boss then says that he verbally told you not to proceed, you only have to say that you have no recollection of any such conversation. He is on the defensive as he has nothing to back it up. If he was "appalled" at the thought of breaking the TOS, then he would have written back and clarified.

Now, if you want to double cover your ass, give him status reports via email. Ask questions. You are covered.

Now to answer some other questions about whether to quit or not. You have to make that decision on your own. For screen scraping, I wouldn't quit over something so mundane. Sorry. Especially if you are a grunt. You voice your concerns, and go on. The reality is that 4 times out of 5 if you voice your concerns like this in a written manner, that the boss will back down. I have faced it twice in a grunt position with two different managers, and both times I got thanked for bringing it to their attention. It is all in how you deliver it. If it comes across as "I am ethical and you are a piece of shit", then your career is hurt. If it comes across sa "I am trying to look out for your well being and that of the company", it can be a positive. Wording is everything.

Spammer logic.

[argent]

If you can access it, it was designed to be accessed.

So you're totally behind email spam, you don't think spam should be considered unethical, let alone made illegal?

Re:Spammer logic.

[BadAnalogyGuy]

Should I be against spam for any other reason than I am annoyed by them?

I don't think spam should be any more illegal than billboards, flyers, or direct mailings.

Re:Spammer logic.

[d3ac0n]

I don't think spam should be any more illegal than billboards, flyers, or direct mailings.

The flaw in this argument is that your three counter-examples (Billboards, Flyers and Direct mailings) are paid for entirely by the SENDER. IE: Billboards are paid for up-front before they are mounted, Flyers and direct mailings have printing costs paid up-front and delivery costs (either the local govt. mail service or paid people to manually give it to you) paid up-front as well.

Spam, on the other hand, is largely delivered on the backs of OTHER payers. Both through the incredibly high bandwidth costs (HOW much of the total Internet traffic is Spam now?) and through ancillary costs such as costs for software and hardware to filter Spam out, and human costs in terms of work-hours wasted manually going through spam. Not to mention the costs to people and networks infected with Spam botnets.

This is what makes Spamming SO profitable, and why it won't go away. Because the costs for Spam are decentralized to millions of people otherwise not directly involved, even a return as little as .01% will turn a HUGE profit. This just doesn't work in the regular advertising world. It's also why it's Illegal is several countries now. It is essentially stealing service from millions of other people and generating millions of dollars of expense for hundreds of companies around the globe, for what are largely scamming and phishing operations.

Get it?

Good.

Do what geeks do best

[GreyyGuy]

Fix it. He wants to do something on the cheap and look good. But the way he wants to do it is going to fail spectacularly. And when it fails, so will you. If this puts any amount of load on the services it is using, it will get picked up by the service provider. Maybe not today, but it will. And then the accounts will get turned off and possibly your IP addresses blacklisted, and then it all goes away. So give him a better solution. If he is balking at the $2k/month find a cheaper service. There is almost always one. Compare the cheaper solution to the time spent fixing it when the free service cuts you off. Provide examples of free service cutting people off.

And unless you are looking for some very specific information, I would expect someone to provide an RSS feed with something similar that is supposed to be used for this sort of thing.

Why are you asking *US*?

[elrous0]

Only YOU can decide how far you're willing to go for your job. You're essentially asking us what your own ethical limits are.

Business sense

[ThePyro]

Even if your boss doesn't care about the ethics of this scheme, he probably does care about ramifications to the business. What happens when you get caught? All your development work will have been wasted because they'll shut you down at the very least. There's potential for a lawsuit, which is an expensive proposition even if you win. Damage to your company's reputation may make it harder to do business. And as another poster already mentioned, this isn't exactly a gem of a project to put on your resume.

A character check?

[juuri]

Having been put in a position once before that an employer asked me to do something I found to be frankly quite lacking in a moral nature here's what I ultimately decided to do.

After considering the work for a while, both why I didn't feel like performing the work personally and why the company desired this functionality I finally decided to do the work, but inform my boss and his boss that I was uncomfortable creating this before hand and giving them clear notice of the whys.

Firstly I did the work because it was simply my job and I had signed onto the job. It's something a *lot* of people might not have given a second thought to creating, obviously as they both had no problems with the work since they asked me to continue even after raising my concerns. Secondly because it wasn't really "that bad" and having steady income of cash dolladolla bills allows me to have nice things like somewhere to live and food I wanted to see if it was something I was over-reacting to.

After completion? Yep, I still felt like shit. So I gave them my notice and told them in the my resignation letter why I was leaving and referred them to the early notification of my objections. So, for me, it was a good learning experience about myself and having done it in this manner I have no problem explaining it to future employers as my reason for leaving this particular job.

Who cares?

[1u3hr]

would require me to break multiple web sites' Terms of Service (TOS).

A website's "terms of service" are not the Ten Commandments. They're not laws, or even moral rules. They're just what one company wants you to do. You don't work for them, why do you care? If they notice and complain, it's your boss's problem, legally; and morally, I wouldn't lose any sleep.

Only thing to do is cover your ass and get your boss to put his instructions in a memo so he can't blame you should problems arise.

Really "scraping a website" is not a moral question on the scale of collaborating with Nazis. It's a business. Other businesses are your rivals, not your friends. They'd fuck you over in a minute.

one approach

[buddyglass]

1. Tell your boss it's a bad idea to break these websites' terms of service. He'll probably override you and tell you to do the project anyway.
2. Code up the project just like he asks. Demonstrate that it works.
3. Shortly afterwards, email the sites in question from a non-work friend's account and let them know (with specific information) the accounts and IP addresses that are violating their terms of service. Hopefully the accounts will be disabled, and/or your employer's IP range will be blocked.
4. Throw up your hands and tell your boss, "Well, I guess they figured out what we were doing!"

If you even need to ask....

[jimicus]

If you even need to ask, you've already demonstrated a trace of ethics.

Now, sometimes having such ethics will mean you have to make difficult choices. And nobody else can make those choices for you.

While ethics won't pay the mortgage, "Reason for leaving the previous job: I was asked to do something illegal and, when I queried this, was given the ultimatum to do it or get out. I got out." is probably a heck of a lot better than "The company had to sack me after it transpired I'd done something illegal" (emails to CYA notwithstanding).

Because, make no mistake, the fact that your company has done this will get out.

Re:Hilarity ensues when...

[Paeva]

I would think this would be a good way to address the issue with your boss. He wants to save some money to get, as he thinks, the same thing for free. But in fact, there are potential downsides to playing that game. He may be disregarding potential legal issues, but he should be less willing to disregard practical issues. If this other company discovers what you're doing, they could make it a little harder to access, or they could ban your company's entire subnet and send a letter indicating that if you'd like to get access again, then you'll have to start paying them for the service you've been stealing.

The key is that, in the meantime, your boss' plan will seem like a dramatic failure that should have been foreseen.

Hahaha hahahaha hahaha!

[EWAdams]

"Compliance officer" in an IT business... you crack me up. You should take your show on the road.

Hospitals have compliance officers because a) they're regulated, inspected, etc. and b) people can die and they can be sued to Kingdom Come.

The IT business is about as regulated as Somalia.