Should You Break TOS Because Work Asks You?

An anonymous reader writes "My boss recently assigned me a project that was all his idea, with two basic flaws that would require me to break multiple web sites' Terms of Service (TOS). Part requires scraping most of the site, parsing the data and presenting it as our own without human intervention. While we're safe on copyright issues, clearly scraping like this is normally not allowed. At times it might also put a load on those sites. The other is, for lack of better words, a 'load balancing' part that requires using multiple free accounts instead of purchasing space and CPU time for less than $2,000 USD per month. The boss sees it as 'distributed' computing when in reality it's 'parasitic.' My question is: am I wrong about the ethics? If I do need to walk, how best can I handle it without damaging my reputation and future employment opportunities?"

Comments

You're Right, Of Course

[eldavojohn]

My question is am I wrong about the ethics?

You don't even have to ask that question, this isn't even one of those interesting cases or gray areas. What you're planning to do is wrong--even though you could probably escape any legal ramifications. It sounds pretty clear that this site creates profit from these overly priced accounts for information that you obviously value at some amount. Getting it for free (regardless of the TOS) could put you at some risk for litigation. Using the term "load balancing" or even "distributed computing" is hilariously misplaced here.

If I do need to walk how best can I handle it without damaging my reputation and future employment opportunities?

Look, I understand what's it like to be looking for a job when the economy is bad. If there are forces keeping you pinned to this employer, I don't know of them. What I would retort with is "How can you keep working this job without damaging your reputation and future employment?" I mean are you going to put in your resume that you coded a technically innovative but bandwidth stealing parasitic botnet to duplicate content from a website that asks for a monthly payment to normally access it at that volume?

I would suggest you propose the $2k/month route and if your boss balks at it, start interviewing with other companies. If you have to leave and you're worried about being blacklisted as a 'whistleblower' (and your boss just might be that kind of guy) then tell him it's for monetary reasons that you're leaving and wish him the best of luck in his future scams.

Re:You're Right, Of Course

[Lumpy]

I say do it. but EMAIL your boss with your concerns and then continue.

when the shit hits the fan you have documentation to throw him under the bus hard and watch the wheels crush him.

Honestly It's all about CYA in the business world. If your boss tells you to do something unethical or wrong, document it every way you can and hold onto that so you can hand him over.

Why do you have any loyalty to them? they have none for you.

Re:You're Right, Of Course

[qoncept]

I would suggest you propose the $2k/month route and if your boss balks at it, start interviewing with other companies. If you have to leave and you're worried about being blacklisted as a 'whistleblower' (and your boss just might be that kind of guy) then tell him it's for monetary reasons that you're leaving and wish him the best of luck in his future scams.

How is splitting and allowing the work to be done by someone else to do any more ethically sound than doing it yourself?

"Of course we must fear evil men, but there is another evil that we must fear more... and that is the indifference of good men."

Re:You're Right, Of Course

[JosKarith]

What they're asking you to do is at the least immoral, possibly even illegal. Your employer doesn't have the right to ask you to place yourself in legal jeopardy in this way, and if the sh1t hits the fan do you really think that someone that came up with this scheme will balk at placing all the blame on you. Someone really needs to have a little chat with your boss about ethics...

Re:You're Right, Of Course

[SatanicPuppy]

All that will happen is that the site in question will blacklist your scraping application. I work for a media organization, and we deal with this stuff all the time. It's far more cost efficient for us to simply whack the application than to try and track down the jokers. It's actually pretty trivial to nail an automated scraper: they're obvious on the logs.

So the few times I've had someone ask me to do this sort of scraping, my response is usually that sure, fine, it works, but it's very easy to spot on the logs, and the information is very likely to become unavailable at unpredictable intervals.

In the long run, it's usually pretty futile to scrape in the first place. When you're stealing content just to drive traffic, you tend to have a crappy site. The only time I ever did a professional scraping app that was "justified" and "legal", the victim was another business unit within the same corporation, and we had every right to the data that they "couldn't" compile for us.

Re:You're Right, Of Course

[jimicus]

How is splitting and allowing the work to be done by someone else to do any more ethically sound than doing it yourself?

At the risk of invoking Godwin's Law so early on, how is taking this approach any different from saying to yourself "I'm just following orders"?

Re:You're Right, Of Course

[DrLang21]

Agreed. This is a case of CYA. I would also consider discussing it with HR, depending on the reputation that your HR group has for protecting internal whistle blowing activity.

Re:You're Right, Of Course

[Hognoxious]

Would it be possible to detect the scraper in real-time and redirect it to some fake/spoof data? It needn't be goatse. But it could be!

Re:You're Right, Of Course

[yincrash]

yes

Re:You're Right, Of Course

[j-pimp]

How is splitting and allowing the work to be done by someone else to do any more ethically sound than doing it yourself?

Its slightly better. There might be some merit in tattling, but presumably at large risk to your self (blacklisted) versus small risk to the party being exploited (the larger the fiscal damage to the company whose TOS is being violated, the better the chance of them catching your boss.) Also in the senario where no one will work for your boss because everyone made the "I will quit rather than do it myself" the work never gets done.

Re:You're Right, Of Course

[Andy Dodd]

Maybe not in real time, but once someone detected a scraper at a given IP, they could easily change their site to feed that IP fake data instead of blocking it.

If I were in the scrapee's position, I'd probably do that because it's the best way to attack the scraper. From order of least effort on the scrapee's part to most:
1) Blocking it makes it obvious to the scraper that they've been found out, and they'll work around it, then you'll need to block them again, on and on the cat-and-mouse game goes.
2) Feeding them mostly good data but with lots of inaccurate information scattered about is nearly impossible for them to detect until it has irreparably damaged their reputation and/or caused them to make bad decisions based on the data.
3) Suing them is a pain in the butt, even more effort than 2)

Re:You're Right, Of Course

[qoncept]

On the Eli Manning scale, they're both an "Aw shucks, I done wrong." Unless you solve the problem (rather than just reducing risk to yourself), what you're doing isn't ethical.

As an aside, I'm not saying this because I'm self righteous. I'd just do the work.

Re:You're Right, Of Course

[SatanicPuppy]

Should be. It depends on what kind of data they're downloading, and whether they're just crawling link by link and hoovering up everything, or whether they're looking for something specific.

Either way, spiders and scrapers usually have programmed scan intervals which have no relation to an actual human's browsing...or they just hit the page as hard as they can, but that is so easy to block that almost no one does it that way. Even if they add a little randomness, it's only efficient to run a scraper if it's hitting every few seconds at max, and even the most ADD user won't keep that up.

Ironically, the easiest way to nail 'em is to put up a subset of "no robots" pages; if the robots crawl those pages, blacklist 'em. Every legitimate spider will respect those files.

Otherwise, if you're running a site with a ton of data, and something is crawling it sequentially, you can absolutely redirect their queries to whatever you want. I'd be wary of doing something cute (if you can call goatse "cute") for fear that you'll have an occasional false positive and redirect a user from a high bandwidth location to that site.

Re:You're Right, Of Course

[erayd]

Is this the New Zealand National Library archival scrape we're talking about here?

Re:You're Right, Of Course

[BobMcD]

Agreed. Do it, CYA, and look for another job.

And optionally, once you're out, blow the whistle.

Re:You're Right, Of Course

[macxcool]

My question is am I wrong about the ethics?

You don't even have to ask that question, this isn't even one of those interesting cases or gray areas. What you're planning to do is wrong--

He's absolutely right. People in positions of power cannot compromise on ethics; not even a little bit. Do what's right and let the chips fall where they may.

Re:You're Right, Of Course

[GrpA]

I've also had similar requests in the past, and in both cases I did the work. I considered the request, decided they were ethical (even if somewhat unusual) and so did it. That's something you're going to have to figure out for yourself - whether you're going to do it or not.

I've been on the other side of the fence also...

If you're relying on data for commercial use, putting yourself in a position where you need that data is a risky thing...

I had a scraper once come after me. I caught them - as the previous poster pointed out, it's easy... I didn't block them. I captured and redirected their requests so I could control what they got and, well, sent them some information that made them look really, really stupid. They were angry, but there wasn't much they could do.

They were just enthusiasts - they had no business risk in their application suddenly failing.

Let your boss know the risk he is facing and then ask him if he really wants to risk being caught and shut down unexpectedly, or worse, finding someone has poisened his data.

It's just not good for business.

GrpA

Re:You're Right, Of Course

[garcia]

So the few times I've had someone ask me to do this sort of scraping, my response is usually that sure, fine, it works, but it's very easy to spot on the logs, and the information is very likely to become unavailable at unpredictable intervals.

Depends on how you do it. I tend to use tor and a random wait time between gets to bring down the data over a few hours (up to a few days) and in one instance, because the URLs were easily guessed, I randomized the list to make it seem as if the hits were going to pages all over the place. I was never banned for any scraping activity that I have done.

In the long run, it's usually pretty futile to scrape in the first place. When you're stealing content just to drive traffic, you tend to have a crappy site. The only time I ever did a professional scraping app that was "justified" and "legal", the victim was another business unit within the same corporation, and we had every right to the data that they "couldn't" compile for us.

It's not futile. Scraping provides a plethora of information in a useful format from places that aren't willing (or unable) to provide data in the necessary format. I used scraped data of course schedule information from MnSCU to develop a weekly report that showed data about how many courses were filled at other area institutions. It was to our competitive advantage to have this information and while it was publicly available, the system wouldn't provide it to us in the DW. I used that data for a variety of different reports than I originally intended and it would not have been possible otherwise.

While I wish that the data had been provided in a better format for my use, it wasn't and that's what made scraping necessary. Plus *I* was the one who got to determine what information I was allowed to glean from the data rather than whatever the system decided was appropriate for our needs.

Re:You're Right, Of Course

[dsoltesz]

If you've gotten to the point of asking Slashdot, you know the answer: it's unethical and you need to be looking for a new job if you can't get this resolved.

These first three responses are probably all you need. Start with talking face-to-face with the boss, outline the ethical and technical problems (focus on the technical "ya know Mr. Boss, this is gonna eventually break") and propose a better solution. Follow up with e-mail summarizing the meeting (definitely document).

If you can't get the boss to buy in on a reasonable, ethical solution, then go ahead and do it his way (it's what you're paid to do) or quit (if you can afford it) documenting why you're leaving (make sure HR gets a copy of your resignation notice). Either way, look for a new job and get the hell out of there. Think positive, don't worry about the economic doomsday crap the news makes up to keep us on the edge of our seats, and don't be too proud to take a pay cut or something resembling a "demotion" (you can always work your way back up). Hell, start your own consulting business.

Re:You're Right, Of Course

[jellomizer]

Well you also need to remember that 2k per month is around what it would cost to replace your job too. For a small company 2k Per month is a lot of money and you shouldn't just pass it off as pure evil, If the company is small enough stuff like thise moves from unethical to stealing bread to feed your children. (Or in this case pay your employees). There is the assumption that everyone workes in huge Multi-Billion dollar companies, however many work in companies that are happy for 5 figure profits. (2k*12 = 24,000 that is 5 figures)

Secondly your reputation isn't as big as you think it is. And if the current job does get ethic violations and even if it pinpoints to you. You should defentaly make public that you doing this work under protest. If asked you can show your protest, also the current bad echonomy is a good excuse as it is a bad echomony you coulnd't risk leaving at the time as you needed the work. However even if it did happen chances are most people hiring will not know or ask or care.

Re:You're Right, Of Course

[Anonymous Coward]

Something else to consider, while there is clearly no ethical question (it's unethical) if your company wants to some how provide "media" or "content" but has to steal it from other sources, how do you expect to make money?

I've seen this before, I've worked at security companies that did the same with with Nessus and Snort. Take their IP, paint it like our own, put some marketing on it and act like we created it. The whole story was that we'd just do it to get started and then start our own effort. Thing is, if that's your core business then shouldn't you actually invest in doing it? Ultimately the company in question was told to stop using nessus.org and had to negotiate some deals and they'll most likely never make a cent.

If the data is important enough to steal and important enough that you need it then you had better understand why it's not important enough for your company to invest in creating it. What's the business plan here? You'll never get better at creating it while you're dependent upon stealing it.

Re:You're Right, Of Course

[Phreakiture]

Tread wicked carefully!

Chip Salzenberg got his ass burned back in 2005 by grumbling about his employer's ethics regarding screen scraping. I heard him speak at YAPC::NA in Toronto that year, and from what he was saying, they were able to take his every legitimate action (e.g. logging in remotely to work from home) and twist it in court into something less than legit (e.g. unauthorized access). It's their word against his, and they hold the access logs. Your best bet, if you want to make a stand about the morals, is get the hell away from there first.

Re:You're Right, Of Course

[Anonymous Coward]

I suggested such to one of our customers to deter some really annoying scrapers. Our customer was very concerned that any legitimate customer could get hit by this, and therefore denied my request to implement some mechanism to add noise to the data the scrapers received.

Re:You're Right, Of Course

[doug]

If you work for a big enough company, get in touch with the legal department and see what they have to say. It is their job to decide which legal fights are to be fought, and which ones are to be avoided. My guess is that they'd put a stop to this if they knew about it.

If your company is too small to have staff lawyers, go over your boss's head.

- doug

PS: Get out from under that boss as his ethics and yours don't align. Whether that means an internal transfer or jumping ship is up to you.

Re:You're Right, Of Course

[Anonymous Coward]

Well you have 2 big problems.

1. The load gets noticed, the site admins figure out what you're doing. and they poison your data. And with who knows what.
2. Unless you're aggregating some sort of very vanilla index, you might not be as in the clear wrt copyright as you think.
a) the US doesn't have a sweat of the brow doctrine, but other countries do
b) even given a) if there's any element of creativity in the data you're scraping the load, and profit, probably take it out of fair use.

What keeps your neighbor from shitting in your lawn isn't just the law, it's what you might do in return. If you don't know these people, you don't know what they'll do. All you know is what you're doing is wrong, and will take money out of their pockets. Therefore, you can expect some hostile reaction. Are you prepared and willing to accept those unknown consequences?

Re:You're Right, Of Course

[alta]

I'm for giving them entirely bogus data that would cause them to loose customers. Not sure exactly what kind of site's we're talking about, but if a customer goes looking for chicken soup recipes and ends up getting porn... I think your boss will realize that they're on to you and won't suggest stealing from them any longer.

Re:You're Right, Of Course

[cerberusss]

I'd advise against discussing it with HR. I've encountered the following situation: I talked to a HR manager about something that obviously should've remained confidential. However that same HR manager was part of the management team and thus had two hats on. She proceeded to inform the management team, to my astonishment.

I've come to the conclusion that HR is just a staff department and owes allegiance to, you guessed it, the management team. Not you.

Re:You're Right, Of Course

[mea37]

And if you ever wondered, when <insert crisis here> broke, how things could go so horribly wrong... it's because of people who think like this guy.

"Don't worry if it's the wrong thing to do; just document that it wasn't your idea!" And apparently never mind the idea of personal responsibility.

When there are no negative consequences for doing the right thing, ethics is mostly a curiosity. Ethics exist to guide you when the right path isn't easy. And yes, you are personally responsible for your own ethical behavior, regardless of whether someone with a bigger paycheck -- or even someone who signs your paycheck -- says otherwise.

Does it mean you have to walk? That depends on your boss. If you do, the best way to preserve your reputation is to avoid mud-slinging. Your current employer might want to try to harm your reputation, but it's extremely unlikely he'll get far (certainly not without exposing himself to legal liability). So just don't shoot yourself in the foot by ranting about the situation in interviews, etc.

Re:You're Right, Of Course

[Alpha830RulZ]

The OP isn't in legal jeopardy. The TOS of the site being scraped at at best a contract (if the employer has a paying agreement with them) and are just words otherwise. If the contract is being violated, the employer is completely liable for the acts of the employee.

I'd just do it. I'd point out to the boss that most sites have logging and other measures in place that may render the work product unreliable, or possibly unobtainable.

I remember someone at our company writing a scraper for for Yahoo some years back. Yahoo blocked our domain, and they had to go back to Yahoo, hat in hand, to get us allowed again. Now Yahoo has (when I last checked) a 4000/request a day limit.

You could combat this with use of proxies, but at that point, you have a case to tell the boss, "you know, if the news media found out about this, what are you going to say to them?" Normal fear should solve the problem then.

An anonymous email to the scrapee's web admin, noting that they might watch traffic from IP thus an such, might also elicit a fun little "I told you so" opportunity to the boss.

I don't think this is worth quitting over. This is just an uninformed boss, who, if the OP is adroit, may become a little more informed.

Re:You're Right, Of Course

[MindKata]

"It's just not good for business."

I find this discussion yet annother interesting insight into the (lack of) ethics of some company bosses. I've often found to my surprise, the ethics of sales people, marketing people and bosses are at times very different from that of programmers and other workers in a company. Some time ago Slashdot discussed "Ethics in IT" and its interesting how it fits with this discussion. Here's the link, it gets interesting how much it fits this discussion, once you get to the part that discusses how some bosses lack of empathy towards others...
http://slashdot.org/comments.pl?sid=448546&cid=22377570

Some bosses have contempt for other people, so considering doing this kind unethical business behaviour, is well within their usual thinking.

Re:You're Right, Of Course

[hummassa]

Depends on how you do it. I tend to use tor and a random wait time between gets to bring down the data over a few hours (up to a few days) and in one instance, because the URLs were easily guessed, I randomized the list to make it seem as if the hits were going to pages all over the place. I was never banned for any scraping activity that I have done.

The other guy just didn't care enough. Even with a lot of randomness, bots' activities stand out like a sore thumb in the logs. The only scraping strategy that "mingles" with real traffic is if you only download the pages when asked by a real user (user clicks on "Info A", you go to site B, fetch info A, display on your own terms for him/her).

Re:You're Right, Of Course

[level4]

Definitely possible!

Any company with a website that contains "regularly updated data that might be interesting for competitors" has probably already got some kind of anti-scraping system in place. This guy's boss thinks he's being clever and original - of course he's not, any company with a site of any value and popularity has already seen this a million times.

What they return basically depends on the mentality of those who work there. The "by the book" professional types will just blackhole the IP or return a "too many visits from this IP" page.

Companies with a more BOFH type guy in charge might very well start "playing" with the data. Instead of the "too many visits" page you might find yourself getting a page with some of the data changed around randomly. Believe me, there are *many* people around who think it is just the height of comedy to fuck with people who are basically stealing their stuff anyway.

They will turn it into a game - and, when the erroneous data turns up on the thieving web site (if that's what this guy's company is running), a few screenshots of that site with the modified data suddenly becomes pretty good evidence in a court, if they're of the "legal remedy" persuasion.

Scraping data is a last resort, not the first thing you try. Forget the ethics - the fact he's working for a company willing to be that insanely cheap and stupid in the first place should be a signal to run far, far away in itself.

Re:You're Right, Of Course

[theaveng]

>>>The other is, for lack of better words, a "load balancing" part that requires using multiple free accounts instead of purchasing space and CPU time for less than $2,000 USD per month. The boss sees it as "distributed" computing when in reality it's "parasitic".
>>>

Can someone explain what this means? Multiple free accounts of what? Gmail? I'm confused.

Since scraping is detectable, I would follow this course of action:
- tell the boss you think "we'll get caught"
- if boss appears to want to fire you, then go ahead and do the action, but ask for him to put it in writing
- note on the order you think it's a bad idea; keep original for yourself and hand copy to boss
- write the program
-
- (optional)
- from your home computer (using an anonymous account), tell the website what your program does, and explain you would have been fired if you had not complied with your bosses' wishes, but feel it's unethical to scrap data.
- watch as Boss looks like fool when website with stolen bandwidth decides to bar his company's access
- if fired, hire lawyer and sue the company for unjustified dismissal
$ profit

Re:You're Right, Of Course

[SatanicPuppy]

Yea, but unless you're running that list across a botnet, the IP addresses are a give away.

Even if you are running it across a botnet it's pretty easy to pick out the patterns using some pretty trivial statistical hacks...If you graph bot traffic it looks like a heartbeat; even if you randomize the access times they don't match "human" numbers (unless you add so much random that it ceases to be an efficient scraper...If you could hire a guy to browse the site and write down the data faster than you can scrape it, they beat you.)

I've never actually been banned for it either, but it's all a crapshoot. I used to work for a company that did GIS data and we smote scrapers on a near-hourly basis, and that one turned freak-nasty because when we found a really good scraper, we'd feed them 60% crap data, and with GIS it's not easy to tell good data from bad.

Things like posted schedules, imho, are the real legitimate use for scrapers. Those people want their data to get out, but they may lack the tools to put it out there.

Re:You're Right, Of Course

[foniksonik]

Don't forget to use a generic email account from your work for any online activity.... don't use your own work email address (definitely not a personal address) as they will try to use that to put the burden on you as an individual....

So use things like it@company.com then use it2@company.com or something similar.

Re:You're Right, Of Course

[Anarke_Incarnate]

This is news?? HR has always been "A part of the company."

This is like when your parents tell you to tell the truth and you won't be punished. You get punished anyhow. HR is the business' way of covering its own ass.

Re:You're Right, Of Course

[Jane_Dozey]

Wait, so if a companies business model is broken and they need something that they can't afford it's OK to rip someone else off now? Come on, stealing bread for human survival is a whole lot different from scraping information because it's cheaper than buying it.

Re:You're Right, Of Course

[Chapter80]

The proper way to document this in email is something like this:

Boss-
I'm able to do the data scraping and should have it up and running by the end of the day.
- Your faithful employee

In case you are wondering about the technical details, here they are:

The scraping is implemented with a perl script which is activated using cron.

We scrape the site twenty times per minute, which is a violation of their terms of service. By doing this, of course, we risk that they may shut us off at any time, or even provide us with fake data.

The typical PHB will read the first two lines on his blackberry, and you're golden. Worst case he or she will scroll down - but the managerial brain is set to shut down at the word "perl". The word "cron" is a failsafe - in case the PHB also has ADD.

Later when s/he comes back and says "why didn't you warn me", you can point to the text "beneath the fold" of your email.

Re:You're Right, Of Course

[SatanicPuppy]

The example that would leap to my mind is a number of services that allow you to "map" an ip address to a geographic location...I use one of those for my job search homepage, and it only allows ~200 queries a day for the "free" account...It would be plenty useful to have as a free service (targeted advertising), and if you set up enough "free" accounts, you could use it that way.

Since I'm doing all my job searching away from where I'm currently living, I use mine to make sure that my job searching page always looks "under construction" for people who live where I live. My boss actually checks it occasionally, I guess to make sure I'm not trying to leave.

Re:You're Right, Of Course

[moderatorrater]

"Of course we must fear evil men, but there is another evil that we must fear more... and that is the indifference of good men."

What's more evil, misusing an online service or letting your family starve? Would you steal a loaf of bread to feed a starving child? You're getting up in arms about something that's definitely unethical but isn't really all that evil in the long run. If the person asking the question has a family to take care of, however, then getting himself into a shitstorm that hurts his career is, IMHO, more evil than just letting his shitty boss make a shitty website that will probably not last a year.

Re:You're Right, Of Course

[pixelpusher220]

My understanding is something like "personal use accounts are free with a capped amount of data, but if you want it all and/or to use it for business, that costs money"

And the PHB here wants to just use multiple 'personal' accounts to get around that stipulation.

Re:You're Right, Of Course

[Anonymous Coward]

I'm an admin for a pretty big site and we've had to deal with bots before. It's a bad business decision to rely on the the misappropriation of another company's work for your commodity. A lot of these times, these bot writers are competing with us in the marketplace, so we have every reason to screw them goatse style.

First, it can easily expose you to copyright liability. As a p2p filesharing kid, you probably find it easy to evade liability, but when you're running a company, you're a sitting target, possibly with deep pockets.

Second, you're depending on another company to provide you with all your data and all they have to do is blacklist you, swap the content with gibberish for you, or merely update their page format. All of this would substantially set you back, and you have no legal recourse.

As for the free services bit, doing stuff like writing workaround bots to host an image stash across photobucket accounts instead of buying your own hosting is only asking to get screwed.
1) free services are not even likely to have 90% uptime, nevermind 5-9s. As an admin, I wouldn't even consider anything less than 5-9s.
2) free services typically prohibit commercial use, and to access them, you have to actually click "accept" on their ToS. Many cases have found those are legally enforceable contracts. If you're intentionally writing code to abuse their expensive contracts and the cost difference is $2k/mo, you're talking at least $2k/mo in liability if they sue you, and possibly punitive damages for being a dick about it.

Re:You're Right, Of Course

[nabsltd]

Unless it's something about you, personally, then an HR employee has no requirement to keep it "confidential".

In other words, if you are talking about your health insurance, your personal information, etc., that's not general "company business" and shouldn't be spread around. But, if you bring them some information about someone doing something that could be detrimental to the business, they really do have an ethical requirement to pass that along. A good HR team would know not to bring your name into it if you are "snitching" until it was absolutely necessary, but sometimes that happens sooner than you would like.

What you were thinking of is a company ombudsman. These people are somewhat like your "lawyer" within the company, and are there for the employees. What they would do is explain to you your options (go to management yourself, let them do it and respect your anonymity as far as possible, never divulge your name even if that means the complaint can't proceed, etc.), and then help you implement them.

Re:You're Right, Of Course

[AndersOSU]

You're absolutely right. The problem is being right, like being ethical, doesn't put food on the table.

The reason crises happen is three fold, first people with power see a competitive advantage in acting unethically, second people in charge or monitoring unethical/illegal behavior aren't up to the task, and third people tasked to do the work don't raise bloody hell when asked to do anything unethical.

In order to solve the problem you only need to fix one of those. The problem is, the first two options involve convincing people to act against their personal interests. People contain a remarkable survival mechanism, the ability to justify and rationalize difficult actions. Going after people who stand to gain by acting unethically is the business equivalent of abstinence only education.

Re:You're Right, Of Course

[Anonymous Coward]

You forgot the part in which the company's more expensive lawyers kick his ass.

Re:You're Right, Of Course

[HikingStick]

Look, he needed to ask the question. Maybe he was both a Boy Scout and someone reared to respect authority (rare in these days, I know). S/he likely felt morally conflicted regarding two competing standards: obey the boss (respect authority) and do not steal (moral code). Add to it the fact that a majority of people in tech [Warning: shirt-cuff statistic ("majority") made up by the author on the spot based on the author's previous experiences.] have views on data piracy (e.g., copying software, media file sharing) that vary from black and white (right and wrong) to varying shades of grey (it's not okay in most cases, but in your cicumstance it is justified). Given that environment, it is logical and expected that the submitter would ask the question. We must not forget that our own views on many topics were not always concrete. They are formed (and sometimes re-formed) by our experiences and the things that happen that reinforce or contradict our views.

Re:You're Right, Of Course

[interiot]

The thing is, you can get away with a lot more low-level scraping than you think. If it's something where you don't need to load significantly more pages than an average surfer (you just need to repeat it several times a day), it isn't necessarily going to stick out in the logs that much. And a lot of admins just don't have the time to analyze their logs (Wikipedia allows hotlinking of their images, for instance... combined with the fact that anyone can upload any picture, this is rife for abuse. But there are better things for them to spend their time on). Also, some admins don't have the tools/skills to drill down and hilight the entries that would make it clear someone is scraping.

If you're relying on data for commercial use, putting yourself in a position where you need that data is a risky thing...

Now that I agree with. Scraping is a gamble. It's possible that an admin could spot you on your very first run, because on close inspection, your requests do look different (you don't immediately load images or CSS/JS subpages like a browser does, for one).

Since you have no idea if the site will block you on the very first fetch, or the billionth, it's not something you should rely on for business.

For personal use though, it can be very educational. There's a lot of data out there, and if you can find a novel way of analyzing it, it can be very rewarding intellectually.

Re:You're Right, Of Course

[Greg_D]

You clearly have never worked for a PHB.

This is what a PHB would to with the information you provided him:

(1) He will fly into an angry purple rage that you even attempted to detail the possible consequences of your actions
(2) When he calms down, he will explain away any possible problems regarding the scraping, completely clueless about the technology you have to develop, complete with cheesy grin and thumbs up before going to lunch on the company dime
(3) When you do indeed get caught, he will say that the problem is obviously your lack of programming skills and experience, since he tried to counsel you on the best way to handle the situation

Re:You're Right, Of Course

[darkmeridian]

I hope you looked at your own EULA to see if this would result in liability for you. Most EULAs are bulletproof but you don't want to be taken to court for deliberately sabotaging members of your web site--the harvesters, after all, did sign up and click on the EULA you propounded so they do have rights despite their violation of the TOS.

Re:You're Right, Of Course

[postbigbang]

"When there are no negative consequences for doing the right thing, ethics is mostly a curiosity. Ethics exist to guide you when the right path isn't easy. And yes, you are personally responsible for your own ethical behavior, regardless of whether someone with a bigger paycheck -- or even someone who signs your paycheck -- says otherwise."

No, just because you don't get spanked doesn't mean that an ethical obligation can be ignored. Were that the case, civility would evaporate. The OP is in a tenuous position, and clearly feels the ethical breach at hand. Sleeping at night, and staying with one's own moral and ethical code takes courage. I'm hoping he/she finds a work around. Thieves are everywhere on the Internet, and scraping is just over the 'line' of cross-linking, which is nominally fair-use.

It's my opinion that if you don't want it linked, then say so or don't post/write the page. Scraping involves more issues related to copyright, which the OP says aren't involved. If they are, then it's a different legal story. Asking an employee to commit an illegal act is conspiracy. If the act has dubious or unclear ethical implications, then it needs to be documented (see above posts about throwing the boss under the wheels) and executed presuming the recourse is the corporation's, not the employee. If the employee is a contractor, then I'm guessing the contractor probably needs liability indemnification to proceed. IANAL.

Re:You're Right, Of Course

[interiot]

I really agree with this. If someone is already going to the effort of writing a lot of scraping code, it's already worth it to them to buy one of those $10-15/month shell accounts online that have SSH access. SSH gives them the ability to forward local TCP requests to that remote IP, their scraping app just has to have the ability to use a SOCKS proxy. This means scrapers have a proxy IP that 1) doesn't show up on any of the open-proxy DNSBLs, and 2) is fast and reliable enough for them to get real work done. And if you block them, they just pay another $10-15 to get another reliable IP.

Re:You're Right, Of Course

[FilterMapReduce]

from your home computer (using an anonymous account)

And an anonymous IP address through Tor or the like, just to be safe.

Re:You're Right, Of Course

[Anonymous Coward]

Believe me, there are *many* people around who think it is just the height of comedy to fuck with people who are basically stealing their stuff anyway.

But it's fun!

Re:You're Right, Of Course

[TheLink]

And could prove to be very amusing for a future slashdot submission if they encounter a BOFH.

There are just so many things that could be done.

They're planning on taking data from some site and pumping it to others and they have _ZERO_ assurance that it's going to be good data and continue to be good data.

When you do stupid stuff like this, if you're not careful very bad things could happen (SQL injection, maybe even malware slipped in) and they could just go "nope not us", and while you could try to sue them it's pretty darn hard to prove since you requested the "bomb", and it only appears once and never appears again.

If you're lucky it's just going to be goatse/tubgirl.

If you're not, it could be a lot worse. Just imagine the BOFH thinking "What should I do today to them and their users" and rubbing his hands with glee.

Just slightly tampered data will be bad enough.

Re:You're Right, Of Course

[Anonymous Coward]

Do it, and anonymously tip-off the site too once its going. They'll block your access, making you look good for bringing up the concerns, and also letting you continue working and move to an honest solution to the problem.

Re:You're Right, Of Course

[aadvancedGIR]

In my country's law, all HR personnels are explicitely considered members of the direction staff, not regular one. It makes things clearer.

Re:You're Right, Of Course

[plague3106]

The problem is that you can end up on the hook legally, if there are legal ramifications. You clearly knew it was wrong, and did it anyway.

Re:You're Right, Of Course

[Anonymous Coward]

I say do it. but EMAIL your boss with your concerns and then continue.

when the shit hits the fan you have documentation to throw him under the bus hard and watch the wheels crush him.

What you have is documentation that you knew it was wrong and went ahead and did it anyway. You're implicating both him and yourself. It would be better to send an email explaining that the proposal is against the TOS and if that doesn't work, explain the penalty for breaking the TOS is [x] number of dollars for each violation. Weigh that cost against the cost of doing it legally.

Re:You're Right, Of Course

[mea37]

"No, just because you don't get spanked doesn't mean that an ethical obligation can be ignored"

I can only guess at what you think you're replying to, because the text you quoted from my post pretty much says: ethical obligations can't be ignored.

Perhaps where I wrote "when there are no negative consequences for doing the right thing", you read it as "no negative consequences for doing the wrong thing"; which I suppose would be a different philosophy altogether (yet one sufficiently at odds with the rest of my reply that I'm a bit surprised if that's what you thought you read).

The point is, most people prefer to talk about how ethical they are when being ethical has no cost. It's all well and good to say 'I prefer to act ethically all else being equal' -- which is essentially what such a person is saying -- but that is the least interesting scenario for talking about ethics. It's pretty much a curiosity.

When ethics actually matter, is when there is a cost to being ethical -- yet in those cases many people will use that cost to justify unethical behavior. Unfortunately, while it seems intuitive that a person with greater means would have more flexibility to take a hit for the sake of ethics, the reality seems to be that a person who's used to having more, feels entitled to more -- and so will sacrifice less in the name of ethics.

Re:You're Right, Of Course

[level4]

Did I say I didn't approve?

I am definitely in the "bastards who find actual physical pleasure in fucking with my enemies" camp ; )

Re:You're Right, Of Course

[Surt]

I'm not sure what world you live in, civility vanished decades before I was born. My guess is that the population crossed a critical threshold where civility became unsustainable. Now life is all look out for #1.

Re:You're Right, Of Course

[Surt]

Blacklisted? Seriously, on what blacklist. I would love to be able to know who has been blacklisted by a previous employer. I interview tons of people, and have never come across a blacklist.

Re:You're Right, Of Course

[orclevegam]

Depends a lot on how they're doing the scraping. It's not terribly hard to write a scrapper that fairly realistically duplicates human behavior, although as you pointed out if they're using it to feed their own processes it does put some demands on how often etc. it's forced to run which could make it stand out from normal activity. Of course, given 20 or so of these bots all scraping from different IPs, so long as you balanced their duty cycles so they were all offset from each other you could have scraping going on for 24 hours without ever deviating from normal browsing patterns. The downside to something like that though is that it requires a certain amount of insight into the layout of the site, you can't just randomly follow every link on a page as that's waaaay too obvious, so if the site layout changes it can break your scrapper(s) until a dev can sit down and update them.

In other words, it's totally doable, and even in a "undetectable" way, but it's fragile, a total pain in the butt, and overall just not worth the headache. Just pay the damn company for proper access to the data and be done with it, it'll be a whole hell of a lot simpler. If your boss doesn't understand that then he's a moron, get out now before you have to do something that will end up on thedailywtf.com, much to your eternal shame I'm sure. Better yet, get out now, and then submit the details of the project you were asked to make to thedailywtf.com, I'd love to read exactly what "genius" ideas this guy has come up with to save a bit of money in one area by paying a bunch of money in another.

Re:You're Right, Of Course

[Clover_Kicker]

Yeah, subtly wrong data is a million times worse then goatse, the scraper might not notice for weeks or months...

Re:You're Right, Of Course

[orclevegam]

Believe me, there are *many* people around who think it is just the height of comedy to fuck with people who are basically stealing their stuff anyway.

You say that like it's a bad thing. Now where did I put my cattlepro... I mean cable tester.

Scraping data is a last resort, not the first thing you try. Forget the ethics - the fact he's working for a company willing to be that insanely cheap and stupid in the first place should be a signal to run far, far away in itself.

Seconded. I used to think my managers were daft, then I started reading thedailywtf.com and I gained a much greater appreciation of exactly how bad things can actually be. From the description this guy gives, he's definitely dealing with someone well on his way to ending up on that site.

Re:You're Right, Of Course

[postbigbang]

We strenuously agree. Ethics aren't cheap; but they have to be abstracted from their motivational cost.

Re:You're Right, Of Course

[postbigbang]

You can reject that, and start out immediately as a civil person at no penalty to you, and as an improvement to the rest of us. Yes, esprit de corps is at a terrible low, but if you join those that are trying to ally efforts to civility, you become an active, rather than accidental beneficiary.

It's not vanished. It's less easy to find unless you're expecting it. Expecting it seems to discipline giving it, too. It's kind of like love that way.

Re:You're Right, Of Course

[onepoint]

I have a site which I get paid very well to manage under contract. I have traps all over the place.
one of my traps is for email scrapers ( php script that loads about 5000 names randomly over 50 pages )
another page I love is my "crap" page, when an ip hit's 20 times in 2 minutes, it loads up and ask if you are a human or a computer, humans end up on the recaptcha page and on there merry surfing way, computers end up in crap section, which is 100% non real data at bargain prices and very specific key word phrases that I can track down. causes more harm, since it also loads porn pictures and goat se and a few other ugly images. it works rather evilly well.

got a few others, but that's the basics.

if you want to read more about bot blocking go to incredibill page, that will help you alot

Re:You're Right, Of Course

[Lumpy]

(4) you mention the next morning "what the hell happened to your car? did you run off the road? the side is all scraped up."

(5) days later, loosen the cores of all the tires scrader valves, replace the caps loosely as well.

(6) subscribe the PHB's email address to every porn site you can find.

(7) put ad in local alternative magazine for boss looking for homosexual urban tantric partners, give the main switchboard and home numbers.

PHB's are very easy to deal with.

Re:You're Right, Of Course

[bickerdyke]

Perhaps it's not different at all, but I think it's a bit unfair to blame someone else for "just following orders" instead of getting fired (or in front of a firing squad). Would you live up to your own expectations - if you were in the SAME SITUATION?

And then there are different kinds of "orders". IANL, so can I be sure that my interpretation of the TOS is the right interpretation at all? So.. why not get the legal department CYA?

(Wow.. pretty much mixing things up here... So here's my CMY: I know that understanding legalese and testifing at the Nürnberg Trials are NOT comparable at all. I was just trying to get back to the main topic.)

Re:You're Right, Of Course

[Surt]

I wish that were true, however, we've now reached the point where I could not buy food for my family if I was not willing to act without civility, because I would never exit the supermarket, having waited in line eternally for people to stop cutting in front of me. And I'm only slightly exaggerating. It would literally cost me hours at the supermarket to do this (I would have to wait for a line to be empty to not have people trying to cut in front of me).

Re:You're Right, Of Course

[Anonymous Coward]

Never attribute to malice what can be adequately explained by stupidity.

Re:You're Right, Of Course

[Misch]

It's happened. ESPN connived a way to get to another sites private database and reported the data as its own. The website injected some fake data which ESPN picked up and reported and were caught.

Re:You're Right, Of Course

[element-o.p.]

Things like posted schedules, imho, are the real legitimate use for scrapers.

I agree. That's the one and only time I actually scripted a scraper for work. One of the senior execs was having trouble downloading an airline schedule (we frequently had to fly techs to our remote locations), so I added a task to cron to wget the schedule from the airline's web page. Downloading the schedule and reposting it on our internal web server was, IMHO, a lot easier and faster than trying to figure out why an exec couldn't download the schedule from his desktop.

In the interests of playing nicely, I set the task to download the schedule once a week so that it wouldn't waste the airline's bandwidth.

Re:You're Right, Of Course

[cranky_slacker]

Ok, so obviously what he was asked to do is unethical. And the consensus seems to be that he should either A) refuse to do it (politely or otherwise) or B) do it, but document objections for CYA purposes.

But I see a lot of comments like this one:

An anonymous email to the scrapee's web admin, noting that they might watch traffic from IP thus an such, might also elicit a fun little "I told you so" opportunity to the boss.

And I'm afraid I have an issue with that advice. I think we can all agree that if the OP is being asked to do this, it is because his employer stands to benefit from it. So, even though a little "I told you so" to a clueless boss would be fun, wouldn't this anonymous email be an act of sabotage? The last time I checked, it isn't ethical (or smart) to sabotage one's employer. I mean seriously, unless the scrapee is inept or indifferent, it will fall apart eventually and the boss will get his. In the meantime, the OP can sleep at night with the knowledge that he didn't intentionally damage his employer's business or reputation.

And before someone points out that the scraper itself damages his company, let's not forget that he was instructed to write it, but he'd be sending the anonymous email of his own volition.

Re:You're Right, Of Course

[descil]

The 'load balancing' part is not really unethical but more importantly stupid. Why? Because this guy who is leaving his free shell accounts open for you to access without limit, is eventually going to notice (in his pocketbook, if he doesn't have any decent monitoring solutions - but really that's his problem.)

Now, the other part, the site scraping, that's pretty rude. It's one thing to access freely available shells. It's another to offer someone else's services as your own without any notification. You can always do this legally just by notifying the person, and they'll even thank you if you do it for the right people - like affiliate schemes, neh?

Re:You're Right, Of Course

[Jack9]

People can say whatever they want. This does not effect me in any way as I consider my salary just compensation for any verbal abuse they want to lobby at me. I know what I know and unless they can make a logical argument that I respect and consider, I'm not really listening. Email exists (as long as you remember to CC correctly) and all you've suffered "when you get caught" is a delay in what would have happened if you refused. Except they paid you in the interim. (1),(2),(3) translate into "nothing" for me.

Re:You're Right, Of Course

[element-o.p.]

Are you serious?

You have no control over what anyone else does or does not do. All that you can control is what you yourself do. If he explains why this is a bad idea, explains the ethical considerations, and his boss says to do it anyway, then what more would you have the employee do? At that point, walking away from the job is the only ethical option you have, since you can't compel anyone else to forgo the job. All you can do once you leave is hope that anyone coming after you will refuse to do the job as well.

Re:You're Right, Of Course

[Fulcrum of Evil]

Who says stopping a linecutter is uncivil? Who says that allowing a linecutter is civil?

Re:You're Right, Of Course

[cartman94501]

That should be obvious. You're only responsible for your own ethical decisions. "If I don't do it, someone else will, so I might as well reap the rewards" is ethically indefensible.

Re:You're Right, Of Course

[GarryFre]

I remember when my boss demanded that I lie to customers. I said if I were willing to lie to the customers, I would have to be willing to lie to him and asked him if he really wanted to have a dishonest employee? I would bring up concerns about the reaction of folks if folks recognize the content as being from another site, and what might happen if an opportunistic lawyer noticed this, how this might ruin the reputation of the company and all that work there when things start hitting the fan. I bet it's more than a small chance that those who would be buying his software, would also be researching other sites and would find the duplication and become aware of what was happening here. PS: Keeping your integrity is a good job getter.

Re:You're Right, Of Course

[LunaticTippy]

Seriously? I want to know where you live and never ever go there.

People around here can be rude, especially when driving, but there seems to be a base of civility. I recently underwent surgery and was walking around with a cast. I was surprised by how many people offered to let me cut in front of them, open doors for me, help me with bags, etc.

Re:You're Right, Of Course

[element-o.p.]

What happens when you quit a job in protest, and then your next prospective employer calls for a reference and is told that you weren't a team player, balked at management direction, etc.? It's not a written "do not hire" list per se, but I would imagine it can be rather difficult to land a new job if your previous employer is giving bad recommendations.

Even if your previous employer is smart enough to be very discrete when answering questions, companies that have been in business for very long know how to ask and answer the important questions without exposing themselves to liability:

Prospective Employer: "Would you consider re-hiring this employee?"
Previous Employer: "No."

or Previous Employer: "This employee was let go for cause." (if applicable).

Re:You're Right, Of Course

[Anonymous Coward]

I've come to the conclusion that HR is just a staff department and owes allegiance to, you guessed it, the management team. Not you.

HR never owe any allegiance to you. The one and only role of HR is to protect the *Company* from the *Employees*.

Always view it that way and you won't go far wrong.

It's not cynical. It's just true.

Re:You're Right, Of Course

[element-o.p.]

That's just rationalizing unethical behaviour. You seem to misunderstand what you quoted at the beginning of your post, and, in fact, use it to prove the opposite of what it is really saying. "...the indifference of good men" is someone saying, "It's not my problem, so I am just going to ignore it." It is working for an unethical company -- even if you are not required to do anything unethical yourself -- rather than leaving in protest, blowing the whistle, etc. It is most definitely not accepting unethical tasks so you can feed your family. As the old saying goes, you were looking for a job when you found this one, so you know you can find another one; it is highly unlikely that your family will starve if you quit the crappy job you have right now, even in our current economy.

Re:You're Right, Of Course

[red_one]

Because you're not killingpeople?

Re:You're Right, Of Course

[billcopc]

The real question is: how much money will be spent developing and maintaining this workaround ?

If the cost of the developer's time approaches or exceeds the cost of getting a license, then you just pay the goddamned license and get on with your life.

You're not working for one of those Wikipedia scrapers, are you ? :P

Re:You're Right, Of Course

[profplump]

Even if it's as slow or slower than a human the bot is still often more efficient -- you don't have to hire a human.

Plus the bot can work 24/7/365, and never makes any transcriptions errors -- humans typically fail both those tests.

/ I write bots for a living

// Scraping information at its owner's request

/// Still have to apply work-arounds because information owner doesn't tell (often third-part) webmaster that I'm allowed to scrape

Re:You're Right, Of Course

[level4]

Troll?! What part of that comment constitutes any kind of troll?

Please save your mod points for my real trolls! You won't have to wait long, I've just opened my 4th Asahi Super Dry ...

Re:You're Right, Of Course

[Anonymous Coward]

Only thing is that as fun as it might be, and even just paybacks to the scraper for stealing the information, it could also open the scrapee up to legal ramifications from the scraper.

More than likely there is no click-through that says if we detect scraping we will feed you bad information or redirect you to porn sites which could damage your reputation with your customers.

And the kind of boss that thinks it is good business to piggyback and steal would possibly also think he had a right to piggyback and steal good information and not bad.

Just sayin'...

Re:You're Right, Of Course

[Anonymous Coward]

My question is am I wrong about the ethics?

My question is this: Is this how your parents raised you? Did they tell you: "Son, if you ever happen to wonder 'Are my deeds ethical?', ask Slashdot! They will weight-in on the issue, their vote is supreme!" A moral advice on Slashdot? Really? Dude, what is wrong with you? You are an autonomous subject, therefore you have to decide.

Re:You're Right, Of Course

[Fulcrum of Evil]

and then your next prospective employer calls for a reference and is told that you weren't a team player, balked at management direction, etc.?

They get sued for defamation. Seriously, most places won't ever do that for fear of lawyers.

Re:You're Right, Of Course

[Tubal-Cain]

...the managerial brain is set to shut down at the word "perl".

Also known as Dummy Mode

Re:You're Right, Of Course

[Anonymous Coward]

Why is everyone so serious? These are TOSes that we're talking about. If you violate, they terminate your access. I don't see anything ethically wrong, let alone any legal exposure, unless you plan to cause them real damages. The only problem is that your program will inevitably fail when it's discovered, probably sooner rather than later.

Re:You're Right, Of Course

[Anonymous Coward]

ooOOOOOooo.... loose customers... sounds kinky!

Re:You're Right, Of Course

[sumdumass]

I'm not sure that being ordered by your boss to do something would exonerate someone from any liability. It would likely bring them down with you but it doesn't Necessarily bring you out of the troubled water and the documentation could hamper any defense later if you indicate that you knew something was potentially illegal before hand. It's still a good Idea but it needs to be crafted carefully.

My suggestion would be to play stupid and email your boss or have him put what he want done in writing but make sure your concerns are in the form of a question to where the boss has to answer with authority. Saying something like "you want me to do X when the site says I'm not allowed to" could be rephrased as " Do I understand correctly that I am supposed to do X, The website has a terms of service that seem to not allow this in certain conditions, could you get a legal opinion about our rights on this action, attached are the relevant TOS agreements to those sites" or maybe even something softer but still crafted to make the boss affirm to you that your not operating illegally or anything and that you asked the question. This way you don't lose your ability for a potential defense later.

I know that playing stupid isn't a valid defense either. But sometimes, it can be the difference between the minimum penalty or slap on the wrist and the maximum penalty allowed by law or perhaps no charges (civil or criminal) being filed against you personally at all. Knowing that your doing something wrong is worse then not knowing because if ignorance which is still worse then being mislead by someone in authority over you.

If someone was willing to go into a position like that, they need an escape route that doesn't chain them to a sinking ship. I'm not going to get into what I think the ethics of something like this carries, but if the decision is made, be careful in how you CYOA so you don't fry your own ass in the process.

Re:You're Right, Of Course

[billcopc]

You could easily replace that with a local Geo-IP database. You can find free databases on the web, use your Google.

Re:You're Right, Of Course

[TooMuchToDo]

There in lies the trick. If you're the one providing the data, you don't block on IP, you kick off in realtime those who are exceeding soft query limits only known to you (and not made public).

Re:You're Right, Of Course

[Anonymous Coward]

Your solution only works for longer update times. The more they have to update the easier it will be to catch. Plus, your use sounds like it was more benign than his bosses plan.

Re:You're Right, Of Course

[level4]

Yeah, I stupidly came across sounding like I disapproved of it. Couldn't be further from the truth, I am exactly the kind of immature bastard who will put 6x as much work into something like this just to cackle manically when some poor fool walks into the trap.

For example ...

I may or may not know a company who were/are creating a "netbank" service, and part of that was publishing near-realtime currency exchange rate data, for which they paid dearly btw. The service hadn't even left beta (still hasn't) when lo and behold, an obviously automated visitor started appearing with extreme regularity on the logs.

I can't remember the exact algorithm but it was something like: if someone from the same IP, who has visited more than 10 times in the last 5 hours, with a standard deviation of less than 1 minute in the interval between visits .. they get served this "special" page.

On that special page, the loop that printed out all the currency rates had a special addition, which looked (from memory) something like this:

if rand(1000) 2
    rate *= (1 + (rand(100) - 50).to_f / 100)
end

ie. once every thousand values, you get a rate that's adjusted by a random value between +/- 50%.

I hope that the guy (or his clients) didn't make any currency trades based upon the info he got from that page. Actually, I lie, I hope he did ;)

Now I think about it, I'm gonna ask how many hits that "special" page gets these days. Heh.

I feel your pain about the bosses, I've had some bad ones myself. But these guys were awesome - they considered that kind of scraping as just another espionage attempt which should be handled with subtlety and maximum "interference" powers. I love that kind of attitude.

Re:You're Right, Of Course

[interiot]

So a dedicated scraper would change IPs, write some code to detect and avoid the potholes, and then resume scraping.

There are lots of decent ways to detect scrapers or hotlinkers. But I haven't seen any idea yet from either side (web admins, or scrapers/hotlinkers) that can't be bypassed with enough work. It really seems like it's something of an arms race.

But the arms race hasn't progressed very far, even for attractive targets with some amount of money (porn sites), because it's just a lot of work for both sides.

Re:You're Right, Of Course

[DeadPixels]

I would suggest you propose the $2k/month route and if your boss balks at it, start interviewing with other companies.

Better yet, why not bring some more numbers into it? Break it down into the language of management.

He will have to pay you $(insert your salary here) to code the scraping and managing the free accounts. He also stands to lose money from a lawsuit from any of the companies, not to mention possibly having to pay you overtime or take you off of other projects to maintain the scraping system if you start getting poisoned data or the sites shut you off. In the end, the cost-benefit just isn't worth it, and if you can put that into dollars, he'll be a lot more likely to listen than if you complained about "ethics."

$2k/month is a whole lot cheaper than the money he'd be losing if he took you off of other projects just to maintain this one.

Re:You're Right, Of Course

[soycap]

"I say do it. but EMAIL your boss with your concerns and then continue." This is dead on. Unfortunately you are paid to do what your boss tells you to, not to save the world. If you have serious personal ethical concerns with your company you need to quit, but if you want to stay an employee all you can do is voice your concerns and proceed as instructed.

Re:You're Right, Of Course

[laughingcoyote]

Only thing is that as fun as it might be, and even just paybacks to the scraper for stealing the information, it could also open the scrapee up to legal ramifications from the scraper.

More than likely there is no click-through that says if we detect scraping we will feed you bad information or redirect you to porn sites which could damage your reputation with your customers.

And the kind of boss that thinks it is good business to piggyback and steal would possibly also think he had a right to piggyback and steal good information and not bad.

Just sayin'...

What legal ramifications? I doubt there's anything in the site agreement which does guarantee any given type or quality of data, and indeed site agreements often specifically state that there is no such guarantee. When you send a request to a webserver, you're going to get back whatever the server is set to send you. If you're sending actively malicious code designed to damage the requester's computer, that may certainly be illegal, but sending back something other than what they are expecting is entirely legal.

That's like saying if Slashdot decided to change its front page to RIAA and Microsoft press releases, Slashdot readers would have grounds for a lawsuit since that's not what they expect to see here. That's simply not the case and would get laughed out of court.

It certainly doesn't matter what our hypothetical PHB thinks he has a right to here. He might think he has a right to some certain type of information from someone else's server. I might think I have a right to have you pay me a million dollars. Neither of these imagined "rights" have any basis in law.

Re:You're Right, Of Course

[retchdog]

And now we understand why those EULAs need to be as draconian as they are. :-/

Re:You're Right, Of Course

[Crudely_Indecent]

I have a special page on my website with this very purpose. Anyone who ends up on my $#!+list quickly finds that all pages alias a page full of nothing but a "don't do this" statement.

The largest abuser is (was..hehe) Cyveilence searching for copyright infringement, followed by the russian search engine yandex.

To your statement, yes it is possible, yes it is easily done (with .htaccess), yes it is effective. They don't come back, of course, I don't remove them from the .htaccess list, so why would they?

Re:You're Right, Of Course

[DrgnDancer]

Well, while a few people have answered your questions or poked holes in your argument, I'd like to point out that this is on of the few:

"1. ...
2. ... ...
Profit!"

Posts that has an actual chance of profiting. I think you must be doing something wrong :-)

Re:You're Right, Of Course

[SatanicPuppy]

It all depends on the data they're after. If it's something small that doesn't change often, they never get caught unless they're wildly stupid.

But if they need to get a LOT of data on even a weekly basis, you can almost always spot them and knock them offline, or feed them a pile of crap, or some other piece of wickedness. Even a piddly 10 dollar account becomes expensive if you have to get new ones all the time, and I am not above knocking out an entire IP block if the vast majority of the traffic is crap.

Re:You're Right, Of Course

[m50d]

How is splitting and allowing the work to be done by someone else to do any more ethically sound than doing it yourself?

Because you're making it costly for the company to do something unethical.

Re:You're Right, Of Course

[mccoma]

truthfully, this sounds more like a web cache then a scraper. heck, you probably actually save the airline bandwidth costs (and maybe a support e-mail)

Re:You're Right, Of Course

[Abcd1234]

Perhaps it's not different at all, but I think it's a bit unfair to blame someone else for "just following orders" instead of getting fired (or in front of a firing squad). Would you live up to your own expectations - if you were in the SAME SITUATION?

Actually... yeah, I would. It's simple: any organization that acts so immorally is not one that I want to work for. God only knows what they'll do to *me* if it'll turn a buck. Meanwhile, I have my self-respect, damnit, and I'm skilled enough, and have a sufficiently untarnished reputation, to know that if I need to find another job because I'm being asked to do something that cuts against my sense of decency, I can do it.

Put simply: a job is just a job. But self-respect isn't something I'm willing to so blithely sacrifice.

Re:You're Right, Of Course

[quanticle]

It depends. I'm not a lawyer, of course, but, generally the legal system looks rather favorably on you if you're warning your boss about the possible legal ramifications. If you warned the boss, the boss can't toss the legal stuff down onto you by saying you went ahead and did this without his authorization.

Of course, in a criminal case its different. Its illegal for you to do something that's criminally wrong even if you're ordered to by your boss. But, in general, things like this aren't handled by the criminal justice system, but are usually handled by the civil courts.

Re:You're Right, Of Course

[mccoma]

well, I guess when the "authority" tells you to violate your "moral code" they really cease to be an "authority" worth obeying.

Re:You're Right, Of Course

[Anonymous Coward]

Does this situation change at all if you are a client of a specific website? For example, my company contracted with "Other Company" to do some order fulfillment. Orders are placed on Other Company's site. Originally my company wanted to iframe Other Company's page into our site or give them our header and edit the style sheet to fit our look. Other Company cannot produce a clean page to iframe with the required dimensions or to allow a custom header or style sheet. Is it ethical to scrape Other Company's site to built a stylized "bridge" between the two? The scrape would be pulling over specific data regarding inventory and other related content if that matters. All of the data exists on a particular page built specifically for my company.

Re:You're Right, Of Course

[svank]

- watch as Boss looks like fool when website with stolen bandwidth decides to bar his company's access
- if fired, hire lawyer and sue the company for unjustified dismissal
$ profit

[svank@localhost ~]$ profit
bash: profit: command not found
[svank@localhost ~]$

Re:You're Right, Of Course

[Thaelon]

That is the most insidious and horrible thing to do to your (PHB) boss I've ever seen.

I love it.

Disclaimer: Lucky for me, I don't have a PHB. \o/

Re:You're Right, Of Course

[pluther]

...we've now reached the point where I could not buy food for my family if I was not willing to act without civility, because I would never exit the supermarket, having waited in line eternally for people to stop cutting in front of me. And I'm only slightly exaggerating. It would literally cost me hours at the supermarket to do this...

Fascinating. A post just a few above this one points out that people have a remarkable ability to justify their lack of civility and here we have a perfect example. It's a fairly minor aspect of life, but this person has actually managed to somehow convince themselves that it's not only OK, but actually necessary for them to cut in lines at supermarkets.

And "Everybody else does it" is probably just about the most common justification people use for their lack of ethics in the first place.

Re:You're Right, Of Course

[mollymoo]

I've come to the conclusion that HR is just a staff department and owes allegiance to, you guessed it, the management team. Not you.

The clue is in the name. You are not a person, you are a resource.

Re:You're Right, Of Course

[Anonymous Coward]

I run a site that posts data that another site likes to steal and post as his own. I regularly note his IPs in realtime and start sending fake data that loosk like it might be real, and occassionally just for fun, I send "STOLEN FROM xxxxxxxx.com" messages intermixed.

Re:You're Right, Of Course

[MK_CSGuy]

wow... devious... you truly live up to your (nick)name!

Re:You're Right, Of Course

[Surt]

I didn't say I cut in lines. I said it was necessary for me to act uncivilly to prevent other people's cutting from keeping me in the store indefinitely.

Re:You're Right, Of Course

[jotaeleemeese]

"the employer is completely liable for the acts of the employee."

This is in general not true. Just imagine a world in which people acted with impunity and then they just blamed the company for any unethical, illegal or immoral actions the committed.

In most instances such excuse will not pass any legal mustard (and as a matter of fact, there are industries in which people become personally liable in similar cirmcunstances).

Re:You're Right, Of Course

[KDR_11k]

Noone sane would guarantee any accuracy of the data to unpaying customers.

Re:You're Right, Of Course

[DrgnDancer]

Ethics is a multifaceted issue though. Let's say the poster has a wife and two kids. If he quits his job over an ethics issue, they will suffer. Depending on how quickly he can get a new job, they may suffer a little or a lot. So his actions can cause some significant suffering to those he love. This would be unethical.

Which is more unethical? The issue being discussed is unlikely to bring significant suffering to anyone, but may a very low grade of suffering to possibly a lot of people in the company being targeted. On the other hand quitting is likely to bring a moderate degree of suffering and perhaps a good deal more to those who may depend on the poster for support (Maybe he's a single parent of two small children, it's an unlikely but possible scenario). Now if he was covering up for his company's dumping of chemical waste that was potentially killing people, there would be little ethical comparison. But if it's a choice between keeping the house and doing something that's, at worst, likely to annoy an admin in the other company, a responsible parent or spouse should probably look for a job WHILE continuing the current one and covering their ass.

Ethics and morals are not the same thing. It is clearly immoral (at least by the standards of most users of this site) to scrape content and resell it as your own. But ethics is not about an absolute decision of right or wrong, it's about weighing choices to determine which choice is best given a set of moral priorities. It is a moral dictate that you not steal someone's work and sell it as your own, it is a moral dictate that you should take care of those who depend on you. Which dictate has priority in this case? That's ethics, and every person will have their own answer, and may even change that answer depending on the exact circumstances.

Maybe my wife has a really good job and I know I can quit without repercussions for my family, that's a weighting factor. Maybe the company we're talking about stealing from is a one man operation barely making it, that might tend to increase the amount of suffering the scraping could do. Maybe my one dependent is a sick little girl who HAS to have medical coverage and chunk of money every month to live... Ethics are determined by circumstances, and aren't always clear.

Re:You're Right, Of Course

[Gyga]

If they break the TOS doesn't that mean the contract is broken and you don't have to uphold your part?

Re:You're Right, Of Course

[Alpha830RulZ]

I don't think it's as bad as you are making it out to be. This is basically whistleblowing, and whistleblowing is somewhat protected in the states, and is usually regarded as reasonable ethical. If the OP informs an injured party where to watch for the injurying party, I don't think many courts are going to stand up for the injurying party.

That said, there is no sense taking chances, so a little gmail activity would be warranted.

Re:You're Right, Of Course

[Alpha830RulZ]

I wasn't clear enough. I'm referring to acts where the employee is operating under instruction, or on approved behalf of the employer. If the employer is committing injury through approved acts of the employees, the employer is going to be the one catching the grief from the courts. If for no other reason than that they have deeper pockets.

Now, for criminal acts, such as assault, fraud, or theft, where the employee is a knowing participant, I agree, the employee is at risk of prosecution as well. But I don't see any criminality here, just a potential contract violation.

Re:You're Right, Of Course

[taucross]

Too true.Reminds me of a story.

Two men are in a boat in the middle of the ocean. The first man starts drilling a hole underneath him, while the second man looks on in astonishment. The second man says, "Why are you drilling a hole in the boat? Can't you see you'll make us sink?" And the first man says,

"The hole is underneath ME. What business of it is yours?"

Re:You're Right, Of Course

[shentino]

Indeed. How civil are you being letting line cutters embolden themselves cutting in front of you only to wind up cutting in front of others?

Nip them in the bud and you do the next guy the favor of not dealing with line cutting cretins.

Tolerating wrongness is itself uncivil if you aren't the only victim.

Similiarly, OP, consider that you are being watched. Your coworkers may have heard of this ethical dilemma you are having. Inspire them, or corrupt them. Just remember, they're probably watching you.

Stand up for your morals and you may inspire your peers to put pressure on your boss to rethink his plan of action.

Re:You're Right, Of Course

[The Cisco Kid]

Answer: "rate limiting queries-per-IP to something just slightly more than a legitimate user is likely to use, which is almost certainly far less than a scraper would have to make to be of use."

Re:You're Right, Of Course

[The Cisco Kid]

Simon? Is that you?

Re:You're Right, Of Course

[cbhacking]

An alternative (derived from the webmasters who objected to AVG's scan-every-search-result thing) is to redirect the traffic back at the company doing the scraping. It's a very simple rule to set up in any web server - when you see a request from a given IP range, instead of serving them the requested page you serve them a redirect header that causes the client to go scan some URL of your choosing.

It would be immediately obvious from the client's end, but it would also be a reasonably clear way of telling them the fsck off without (to the best of my knowledge, IANAL) doing anything that could result in legal trouble (especially since, due to voilating the ToS, the server is perfectly within their rights to refuse them the requested data).

Of course, this doesn't help the guy being asked to write this client, except possibly to point out that the other company could easily A) block this, and B) increase your own bandwidth usage in the meanwhile.

Re:You're Right, Of Course

[Anonymous Coward]

No need to invoke Godwin here. Fragmentation of responsibility has been around for a while.

If you want to see how this runs its course, I suggest you read Jonathan Glover's book Humanity: A Moral History of the Twentieth Century.

One of the chapters describes Allied bombings in WWII. At first, we were targeting factories in hopes of a swift victory by attrition. The initial flight routes were planned to minimize civilian casualties. As the war drew on, though, we started plotting the routes to maximize civilian casualties and eventually began outright bombing campaigns against civilian targets.

The transition was a gradual one, with everyone just âoefollowing ordersâ from their superiors. No one person was directly responsible for the unnecessary deaths of thousands of innocent civilians.

Disclaimer: It's been some time since I've read this book, so feel free to correct me.

Re:You're Right, Of Course

[Anonymous Coward]

Right, except that it's a waste of time and karma trying to get your boss to put an instruction like that in writing.

Write it down yourself. Send him an e-mail saying "So this is what I'm planning to do, okay?" Easier and less painful than trying to make him create evidence against himself.

Re:You're Right, Of Course

[kunwon1]

Any journeyman Perl programmer could easily write a scraper that would be practically impossible to blacklist from a production server. You just change user agents and slow it down. Put in some random time delays, and don't access things in sequential order. The difficulty would go up or down depending on the structure of the website (Is there a page that links to every other page on the site? Is there a huge amount of dynamically generated content that changes from moment to moment? Do people rarely stay for longer than a few clicks/pageviews?) but unless you're willing to blacklist other legitimate users for things that look like scraping but actually aren't, you're boned.

Re:You're Right, Of Course

[postbigbang]

Situational ethics can be difficult, we'll agree. Yet the obligations really don't waver. Anecdotally, the ethics appear to be pretty clear in this case. Common practices are sometimes useful to understand in such cases, and it would appear that the legality is also pretty clear as well.

I'm reminded of the old aphorism that says that wrestling with pigs will get you dirty, and the pig likes it.

One's first duty is familial protection, as you cite in your example, tilted as it is. How does one get out of that entrapment? The first viable road out is the best way. Documenting what's going on is useful, too, should litigation ensue. There's a bit of shield in doing this if he/she's a W2 employee, and not a 1099 contractor. If a contractor, then there's an assumption of liability, however IANAL.

While ethics seems to be aside from moral action, instead it's a derivative of morality. We use common/best practices to decide these, just as we use juries of peers in other areas. The standards used have to be comprised of all of the parties in a common denominator against these industry common/best practices to derive meaning. Otherwise, as you say, everyone has their own answer, and a million people will have as many answers; that's why ethical standards need apply here.

There are many answers, but the ones that fit best are the ones he/she has to live with. It would appear that the citation gives merit to the individual concern, and one shouldn't be enslaved to do what one considers evil. Does that mean that a CA pharmacist should sell an abortion pill if they believe that abortion is wrong? This has been tested in the courts, along with selling contraceptives, and so on. Clearly, a person has to live with themselves, but there may be a legal and seemingly moral reason for the contention. Ethics would hopefully supply the answer. Sometimes, it does not.

Re:You're Right, Of Course

[Anonymous Coward]

I was wondering that too.

Re:You're Right, Of Course

[Falconhell]

This take on HR is from platothefish's blog at www.multiversity.blogspot.com. A hillarious blog.

"Can I just mention that your ideas to rename HR whilst amusing are unlikely to be take up by the university"

In light of the realisation that the term Human Resources is antithical to our diversity policies (which by the way were crafted by HR) I hereby suggest some alternatives:

* 'The department of paper, rules and dogma'
* 'Directorate (in an equal opportunities non directive type of way of course) of Valued Employees'
* 'The communist party' - I think party has a nice focus to it, different to directorate comrade
* 'The group of really nice people trying to get other really nice people to comply with the rules and policies we design to make it look as if we are really useful department.'
* 'Slowing things down and stopping untidy innovation team'
* A bit radical this - 'Administration.'
* 'Directorate of creative mavericks and ideas control'
* 'The Empire' - why is the only part of the university that appears to be in full blown growth, 'Human Resources'?"

Re:You're Right, Of Course

[Anonymous Coward]

I used to be a high-school sysadmin, and I had a principal like the boss in _The Office_: 100% heart, 0% brain, and he thought that our school ran independently of school board policy.

Unable to say "no" to the overly spoiled teachers, he decided that teachers had the "right" to check out school-owned equipment for personal use--even a video projector screen for a teacher's daughter's wedding.

School district policy not only prohibited this, but required employees to submit proof of homeowner or renter insurance to check out equipment, even if for work-related purposes. The principal wasn't willing to even require this of teachers, so I amended the school district's equipment check-out form:

"As the principal of ____ High School, I understand that this equipment checkout is against ____ County School Board policy (1) to check out equipment without proof of insurance, and (2) to check out equipment for non-work-related use." (sign)

Of course, he signed it without even reading it. Fortunately, I had no incidents, but I had my documentation. He said that if there ever was an incident, he'd pay for the lost equipment himself. I'm just glad I didn't have to actually put this promise to the test.

Because of intolerable work conditions, I was actively seeking another job whilst trying to keep mine in case I couldn't find another one. When the principal's end-of-year stunt was making me check out $2000 of audio equipment to the band playing at a student's private graduation party, I smiled at the thought of resignation. Which I submitted soon after that school year ended.

Re:You're Right, Of Course

[DirtyShaman]

The person who wrote the parent is obviously a prude. Do it. Do you really think someone's going to come in after the fact and go 'yes, this developer here, its all his fault. He conceived and created the whole thing'? Get a grip on reality, thats why your company has a legal department. Thats why you HAVE a boss. It isn't your place to second guess every decision that is made. Do your job. Don't ask questions. If you don't like the work you're being asked to do, quit. End of story.

Re:You're Right, Of Course

[Hal_Porter]

You're one of those people who wants the bad guys to win in movies, aren't you?

Re:You're Right, Of Course

[douglaid]

If you are an employee, you should be safe from legal action. A book on casuistry (ethics as a "science") put out by the Catholics said that you aren't morally to blame either. If you are an independent contractor, then you have a problem.

Treat this advice with caution. I am not in IT; I am an Australian lawyer.

Re:You're Right, Of Course

[Anonymous Coward]

Is the e-mail at your work reliable for this purpose? Can you e-mail it to several others without getting in trouble? Is he friends with someone who can have a "glitch" erase the records of that e-mail? If you're really concerned about liability, consult a lawyer before starting.

Re:You're Right, Of Course

[Anonymous Coward]

Is someone scraping your posts? Cuz you spelled "lose" as "loose"! Very clever but I'm on to you!

Re:You're Right, Of Course

[allgoodnamesaretaken]

is goatse royalty free?

Re:You're Right, Of Course

[meringuoid]

The typical PHB will read the first two lines on his blackberry, and you're golden. Worst case he or she will scroll down - but the managerial brain is set to shut down at the word "perl". The word "cron" is a failsafe - in case the PHB also has ADD. Later when s/he comes back and says "why didn't you warn me", you can point to the text "beneath the fold" of your email.

Sir Humphrey Appleby? Is that you?

Re:You're Right, Of Course

[DrgnDancer]

My view is that ethics is the practical application of morals. Much like engineering is the practical application of science, but without the clean line of either :-). Moral dictates are absolute. This is not to say that they are not different between people, cultures or even an individual's lifetime; but within a specific time, place and person, they are firm rules. Killing people is bad, helping others is good, stealing is bad, taking care of your family is good. Most people rate morals on a hierarchy, they'd consider killing worse that stealing, and stealing a lot worse than stealing a little. My personal moral code, and I suspect the codes of most people at the most basic level, are based on the level of harm or suffering that an action causes to others. Even most religious based codes have this are their core (The Golden Rule and The Sermon on the Mount both argue for this in Christianity, several Prophet argue for it in Judaism, similar ideas exist inside of Buddhism, and most forms of Hinduism, etc), but may be modified by specific requirements of a holy writing, an interpretation, or a desire to please God.

Ethics is the practical application of these absolute moral strictures. In the real world there is seldom a pure moral choice, it's always a choice of situational ethics. Which choice, based on available information, will cause less suffering? Less harm? Of these two or three, or four moral imperatives, which is more important? Because you can only please one (or maybe two, or maybe all but not perfectly, whatever). Is it better to imperfectly follow two moral strictures than perfectly follow one while ignoring another? Beyond that comes facts that are not influenced by your morals. Maybe you know that your boss won't fire you if you refuse to do something against your moral code, because he's a good guy. That makes the ethical choice a lot easier.

In most cases the outcome of a choice cannot be predicted perfectly. Maybe the clear moral choice will lead to greater suffering down the road. Ethically we should consider that. An ideal example of this IMHO is sex ed. Many Christians feel sex out of wedlock is immoral, the clear moral choice for them is therefore to limit or ignore sex ed. Often this choice leads to teen-pregnancy, STDs, abortions, etc. Might the better ethical choice have been to have strong sex ed, despite that it seems the less moral choice, to prevent greater moral problems down the road?

Moral questions are easy. Clearly it is immoral to do the screen scraping program. He shouldn't do it, I agree. Ethical questions are much more complicated. They require a lot more background than is given in this post. How likely is the poster to be fired if he refuses to do the work (it may be that he can refuse with little risk if he is otherwise very well thought of)? Does he have a family? How much does his family depend on his income? Does he have a savings to tide him over? Even without a family I consider that I have a moral obligation to myself. Somewhere on my hierarchy of things that are good and bad, I place helping myself as good and hurting myself as bad. Depending on how much I am going hurt myself by quiting or being fired, and how much the screen scrap is going to hurt someone, I may make the ethical decision that helping myself is more important than helping some nameless person who is unlikely to even notice the small harm being done to them.

Ethics is always about situations. Often those situations are clear, I should not take loads of money from other people so that I can have a really big boat. I'm doing a lot of harm to a lot of people and only benefiting myself and maybe my family. This is a lot grayer, and we don't know all the details. All things being equal, I agree that the poster should not do the work. Things are rarely equal. There are a ton of factors that might make it a lot better for almost everyone if he did it. There are an equal number of factors that might make it even more obvious that he shouldn't do it.

Re:You're Right, Of Course

[Anonymous Coward]

If you have to leave and you're worried about being blacklisted as a 'whistleblower'

Why would he be worried about that? He didn't say that he has paranoid delusions.

Re:You're Right, Of Course

[Anonymous Coward]

Most of the internet is an arms race of some sort. Probably any time you have competition that doesn't have a finish line, you have an arms race, or a Red Queen's race. Having identified that, can we move on to say something more meaningful? Why it is difficult, say, or relate this particular arms race to other similar, higher-profile arms races.

I'm not going to downmod, but I really wish that your post were a bit more worthy of the 'insightful' mod.

Re:You're Right, Of Course

[TheoMurpse]

sue the company for unjustified dismissal

A company can dismiss an employee for good cause, bad cause, or even no cause at allthis is black letter law. Ignoring a contract defining a specific work timeline, the only exceptions I know of are

1. civil rights violations (sexism/racism/ageism/...); or
2. if you
   1. say you won't do act X;
   2. are told if you don't do act X you will be terminated;
   3. are terminated based on your continued refusal; and
   4. the act violates some constitutionally- or legislatively-demonstrated public policy.

If you did the act and then got fired for something else, you likely will not prevail on a wrongful termination suit.

Re:You're Right, Of Course

[Anonymous Coward]

I think the upper two posts reflect the two aspects there are to this issue; a moral one and a practical one. Looking from the practical side, I would agree with Lumpy. From a moral point of view, I guess no-one can tell you how bad it would make you feel if you did it, and whether that's worse than quitting your job. I guess, in the end it's a moral decision. Good luck.

If you want legal advice...

[Anonymous Coward]

...ask a lawyer.

Re:If you want legal advice...

[dintech]

No, no, no. If you want legal advice, ask Slashdot! Given enough time, you'll get an answer that is exactly what you hoped for and you can ignore or mod down the ones you don't like. It couldn't be simpler and it's a whole lot cheaper!

Re:If you want legal advice...

[realisticradical]

You must be new here.

Re:If you want legal advice...

[Anonymous Coward]

In fact, ask your company's lawyer. They'll know what contracts your company may or may not have with these other companies, which is basically the question being asked.

Basically: If your company does not have permission to do this, they need to get it. Your company's lawyer is probably the right person start the process of getting that (legal) permission.

Re:If you want legal advice...

[Zwicky]

OK Ray... that's your cue ...

Re:If you want legal advice...

[Anonymous Coward]

IANAL but that sounds dumb...

Re:If you want legal advice...

[dintech]

You forgot to install your sarcasm chip today.

Re:If you want legal advice...

[Hognoxious]

What the hell are you on about, I've never heard of such a thing.

Re:If you want legal advice...

[dintech]

I see what you did there, very good. :)

Just ask yourself this:

[Anonymous Coward]

Did the contractors on the Death Star deserve to die?

Re:Just ask yourself this:

[Intron]

"Did the contractors on the Death Star deserve to die?"

Depends on whether it was the ones that did the weapons array or the ones that did the low-flush toilets. Oh wait, Halliburton did both.

Re:Just ask yourself this:

[Fred_A]

So you're saying we should kill the OP ? Or do you mean his boss ?
Isn't this a little harsh ?

Re:Just ask yourself this:

[Shakrai]

Did the Strom Troopers also get electrocuted in the showers on the Death Star?

Re:Just ask yourself this:

[Quince alPillan]

Because there have been a couple people who don't understand the reference from Clerks...

Randal: So they build another Death Star, right?
Dante: Yeah.
Randal: Now the first one they built was completed and fully operational before the Rebels destroyed it.
Dante: Luke blew it up. Give credit where it's due.
Randal: And the second one was still being built when they blew it up.
Dante: Compliments of Lando Calrissian.
Randal: Something just never sat right with me the second time they destroyed it. I could never put my finger on it-something just wasn't right.
Dante: And you figured it out?
Randal: Well, the thing is, the first Death Star was manned by the Imperial army-storm troopers, dignitaries- the only people onboard were Imperials.
Dante: Basically.
Randal: So when they blew it up, no prob. Evil is punished.
Dante: And the second time around...?
Randal: The second time around, it wasn't even finished yet. They were still under construction.
Dante: So?
Randal: A construction job of that magnitude would require a helluva lot more manpower than the Imperial army had to offer. I'll bet there were independent contractors working on that thing: plumbers, aluminum siders, roofers.
Dante: Not just Imperials, is what you're getting at.
Randal: Exactly. In order to get it built quickly and quietly they'd hire anybody who could do the job. Do you think the average storm trooper knows how to install a toilet main? All they know is killing and white uniforms.
Dante: All right, so even if independent contractors are working on the Death Star, why are you uneasy with its destruction?
Randal: All those innocent contractors hired to do a job were killed- casualties of a war they had nothing to do with. (notices Dante's confusion) All right, look-you're a roofer, and some juicy government contract comes your way; you got the wife and kids and the two-story in suburbia-this is a government contract, which means all sorts of benefits. All of a sudden these left-wing militants blast you with lasers and wipe out everyone within a three-mile radius. You didn't ask for that. You have no personal politics. You're just trying to scrape out a living.

(The Blue-Collar Man (Thomas Burke) joins them.)

Blue-Collar Man: Excuse me. I don't mean to interrupt, but what were you talking about?
Randal: The ending of Return of the Jedi.
Dante: My friend is trying to convince me that any contractors working on the uncompleted Death Star were innocent victims when the space station was destroyed by the rebels.
Blue-Collar Man: Well, I'm a contractor myself. I'm a roofer... (digs into pocket and produces business card) Dunn and Reddy Home Improvements. And speaking as a roofer, I can say that a roofer's personal politics come heavily into play when choosing jobs.
Randal: Like when?
Blue-Collar Man: Three months ago I was offered a job up in the hills. A beautiful house with tons of property. It was a simple reshingling job, but I was told that if it was finished within a day, my price would be doubled. Then I realized whose house it was.
Dante: Whose house was it?
Blue-Collar Man: Dominick Bambino's.
Randal: "Babyface" Bambino? The gangster?
Blue-Collar Man: The same. The money was right, but the risk was too big. I knew who he was, and based on that, I passed the job on to a friend of mine.
Dante: Based on personal politics.
Blue-Collar Man: Right. And that week, the Foresci family put a hit on Babyface's house. My friend was shot and killed. He wasn't even finished shingling.
Randal: No way!
Blue-Collar Man: (paying for coffee) I'm alive because I knew there were risks involved taking on that particular client. My friend wasn't so lucky. (pauses to reflect) You know, any contractor willing to work on that Death Star knew the risks. If they were killed, it was their own fault. A roofer listens to this... (taps his heart) not his wallet.

Re:Just ask yourself this:

[lisany]

They knew the risks when they took a job working for The Empire.

Re:Just ask yourself this:

[Rennt]

Haven't you seen "Clerks"

Dante: My friend is trying to convince me that any contractors working on the uncompleted Death Star were innocent victims when the space station was destroyed by the rebels.
Blue-Collar Man: Well, I'm a contractor myself. I'm a roofer... (digs into pocket and produces business card) Dunn and Reddy Home Improvements. And speaking as a roofer, I can say that a roofer's personal politics come heavily into play when choosing jobs.
Randal: Like when?
Blue-Collar Man: Three months ago I was offered a job up in the hills. A beautiful house with tons of property. It was a simple reshingling job, but I was told that if it was finished within a day, my price would be doubled. Then I realized whose house it was.
Dante: Whose house was it?
Blue-Collar Man: Dominick Bambino's.
Randal: "Babyface" Bambino? The gangster?
Blue-Collar Man: The same. The money was right, but the risk was too big. I knew who he was, and based on that, I passed the job on to a friend of mine.
Dante: Based on personal politics.
Blue-Collar Man: Right. And that week, the Foresci family put a hit on Babyface's house. My friend was shot and killed. He wasn't even finished shingling.
Randal: No way!
Blue-Collar Man: (paying for coffee) I'm alive because I knew there were risks involved taking on that particular client. My friend wasn't so lucky. (pauses to reflect) You know, any contractor willing to work on that Death Star knew the risks. If they were killed, it was their own fault. A roofer listens to this... (taps his heart) not his wallet.

Kevin Smith knows his stuff.

Re:Just ask yourself this:

[ari_j]

Which Death Star? I can see Halliburton doing the first one, but after the Empire stopped making payments at the end of A New Hope, I just don't see it being characteristic of their business strategy to build the second one.

Re:Just ask yourself this:

[KylePflug]

Notably, they didn't finish the second one. When you stop paying, you get a Death Star with a big-ass hole in the side.

Re:Just ask yourself this:

[Surt]

Yes, they helped to build a giant weapon designed to kill innocent people, they deserved what they got.

Re:Just ask yourself this:

[dr_dank]

I'll bet they weren't even supposed to be there that day.

Re:Just ask yourself this:

[WaXHeLL]

That's sloppy electrical work.

That's the first thing that should be done when you wire up new equipment -- check for ground faults.

Re:Just ask yourself this:

[Anonymous Coward]

Don't you think the empire would DELIBERATELY keep SOMETHING broken all the time, just to keep a few "innocents" there? If one is ever hurt, it's great media in their favor. If you can afford a Death Star you can afford a half dozen contractors at all times. It not contractors, keep various public works groups there. Offer PETA an office.

What's more, the empire was THE empire. Those fighting them were the REBELS. The empire was evil, but often lawful evil. (It's easy to be on the right side of the law once you're the one making it.)

Were the contractors doing wrong by supporting the only "legit" government around? Morality can go out the window when the other guy blows up whole planets.

As for babyface, what legitimacy does he have?

Hilarity ensues when...

[Anonymous Coward]

...you build a system that closely relies on this nonstandard (and unsupported) method of getting information, they change it and it breaks.

Either by accident, or because they spot a load of particular access patterns from your address, figure out what's going on and intentionally break it.

Re:Hilarity ensues when...

[Anonymous Coward]

...you build a system that closely relies on this nonstandard (and unsupported) method of getting information, they change it and it breaks.

Either by accident, or because they spot a load of particular access patterns from your address, figure out what's going on and intentionally break it.

Or, better, they blackhole your IP.

Re:Hilarity ensues when...

[Paeva]

I would think this would be a good way to address the issue with your boss. He wants to save some money to get, as he thinks, the same thing for free. But in fact, there are potential downsides to playing that game. He may be disregarding potential legal issues, but he should be less willing to disregard practical issues. If this other company discovers what you're doing, they could make it a little harder to access, or they could ban your company's entire subnet and send a letter indicating that if you'd like to get access again, then you'll have to start paying them for the service you've been stealing.

The key is that, in the meantime, your boss' plan will seem like a dramatic failure that should have been foreseen.

Re:Hilarity ensues when...

[Sockatume]

Workaround: write a program which generates random data using a small amount of harvested data as a guideline. If the boss is too lazy to generate the data he's meant to be generating, then he's probably going to be too lazy to check that the data you're "harvesting" is actually accurate.

Re:Hilarity ensues when...

[geminidomino]

I remember reading once (grain of salt time) about a guy in this position. Someone was deep-linking his images, so he actually redid his old site, changed all the image names, and made replaced the files being linked with goatses. The scammer actually threatened to sue him.

Hilarity indeed. :)

Re:Hilarity ensues when...

[Anonymous Coward]

if you'd like to get access again, then you'll have to pay a (very large) activation fee and start paying them for the service you've been stealing.

Re:Hilarity ensues when...

[SatanicPuppy]

That's pretty common. John McCain had an issue with that earlier in his campaign when his MySpace page got hit. The guy who did the original template wasn't keen on having his images hotlinked from such a high volume site and made a hilarious substitution (which was widely misreported as a "hacking" incident in the media).

The AC is dead on. If you depend on someone elses data, they are going to notice, and they are going to remove your access, or, worse, start feeding you crap.

Re:Hilarity ensues when...

[Martian_Kyo]

This is so true, I worked for a company that did scraping of this sorts. And it's chasing ghosts. You get banned, or even slightest search engine change screws-up everything.

Few things I learned from that job:
-There are vast number of big name sites with unreliable search engine *caugh*virgin*caugh*holidays*caugh*
-ASP and .net in general are evil, I hate hate hate HATE viewstate. It's black magic.
-Live httpheaders is one of best firefox plugins ever!

Re:Hilarity ensues when...

[Sockatume]

Daniel Rutter of Dan's Data fame has done something like this on a few occasions. It's a classic for a reason.

Re:Hilarity ensues when...

[Anonymous Coward]

If you work as a freelancer, then you just charge them more to fix it (and it's usually a pretty simple fix for a scraper anyways)

Re:Hilarity ensues when...

[sjames]

If this other company discovers what you're doing, they could make it a little harder to access, or they could ban your company's entire subnet and send a letter indicating that if you'd like to get access again, then you'll have to start paying them for the service you've been stealing.

If they want to be nice, that's what they'll do. If they aren't feeling very nice, they'll silently redirect requests from the offending subnet to the raunchiest porn they can find that's still somewhat legal, all formatted to look like the correct text to the scraper.

The company doing the scraping will not likely take legal action since they would have to first admit wrongdoing in court, and then face the argument that such a re-direction couldn't have materially harmed them had they complied with the TOS.

Get yourself caught, get him blamed

[Anonymous Coward]

Wait until he takes credit, then create some problems. Put in a user agent string identifying your company... have the distribution not quite equal... Just do a few things to tip off others and sabotage his brilli

Re:Get yourself caught, get him blamed

[dword]

I'm throwing away mod points because of this so you better read it, mr AC.
The guys is trying to keep his job. How would sabotaging this help him keep his job if his boss ends up behind bars or, worse, his company goes bankrupt. Did you ever consider his boss is probably one of those guys that always finds someone to blame for anything that goes wrong? If the system fails in any way, the guy is clearly fired. The whole point of this discussion is to help him keep his job and show some compassion for the morally wrong things he's about to do - he'll do it anyway, he just hopes that some of us will give him a good excuse.
I know, because I've been there!

It's your job...

[sshuber]

and it doesn't sound like anything illegal, just a bit questionable maybe. I say count your blessings you even have a job in this economy and do it unless you feel you are breaking the law. I don't have my copy of the ACM code of ethics, but this seems pretty borderline to me.

Re:It's your job...

[I cant believe its n]

I don't have my copy of the ACM code of ethics

Well, look no further: The ACM Code of Ethics

Some sections relating to this issue would be:
1.1 Contribute to society (and human well-being.)
1.2 Avoid harm to others.
1.3 Be honest and trustworthy.
1.5 Honor property rights (including copyrights and patent.)
1.6 Give proper credit for intellectual property.
1.8 Honor confidentiality.

Re:It's your job...

[Anonymous Coward]

Hey, you scraped that!

Re:It's your job...

[Mr. Slippery]

I don't have my copy of the ACM code of ethics, but this seems pretty borderline to me.

Depending on the details, I don't really see an ethical issue here.

If "scraping" is done so as to not put a heavy load on the servers, it's no more unethical to do it automatically than to hire a bunch of drones to compile the data manually. Copyright does not apply to the raw data. Site's "Terms of Service" are baloney (by reading this post this far you agree to send me $20); when you publish information you cannot put restrictions on its use. I see more of an ethical problem with putting up such bogus "Terms of Service" than with using the data.

As for using free accounts, it might make for an unprofessional and shoddy product - which would be grounds to object - but there's nothing inherently unethical about using a free service for business purposes. A lot of small businesses use a Gmail or Yahoo! e-mail account, for example.

Re:It's your job...

[CKW]

Nice, the ACM regards copyrights and patents as a subset of "property". Good thing 1.1 and 1.2 form gaping holes the size of trucks :)

The solution

[VincenzoRomano]

Resign now!

Anything on the web is available for access

[BadAnalogyGuy]

If you can access it, it was designed to be accessed.

What your boss is trying to do is not unethical, just cheap. And to be honest, doing something at a lower cost and achieving the same results is actually good business.

So you can either be on board with your boss, or you can acquaint yourself with the unemployment line.

Re:Anything on the web is available for access

[Anonymous Coward]

You really live up to your name... I don't think that even hackers(1) agree with you. They just access stuff because they can. And shit their pants hoping nobody ever notices.

(1) as coined by the media, really should be crackers

To me it's obvious it would be a crime. But hey, who am I to tell you not to commit it?

Re:Anything on the web is available for access

[jthill]

By those rules, taking candy from a baby isn't unethical.

Re:Anything on the web is available for access

[BadAnalogyGuy]

Babies really shouldn't be given candy in the first place.

Re:Anything on the web is available for access

[Pig Hogger]

What your boss is trying to do is not unethical, just cheap. And to be honest, doing something at a lower cost and achieving the same results is actually good business.

Actually, in $COMMONWEALTH_COUTRY, it is even possible to get tax credits for writing a solution! I know, I've done it...

Re:Anything on the web is available for access

[gnud]

Well, the problem arises when they notice you scraping and block you from accessing anything at all.

Re:Anything on the web is available for access

[networkBoy]

touche...

You know you are one of the more "noticeable" nicks on /. and normally I love your comments.
but today made me wonder if you might have a touch of aspergers?
And yet I still love the commentary.

Re:Anything on the web is available for access

[megamerican]

Babies really shouldn't be given candy in the first place.

Then taking candy from a baby IS the ethical thing to do!

Re:Anything on the web is available for access

[Anonymous Coward]

There is a reason the post is modded Insightful. Keep in mind that he is BadAnalogyGuy.

Re:Anything on the web is available for access

[Anonymous Coward]

Do you mind if a bunch of brothers pop your ass cherry? After all, it can be accessed.

Re:Anything on the web is available for access

[mcgrew]

If you can access it, it was designed to be accessed

Accessing data is not the same as infringing copyright, which is what it sounds like he's doing.

Re:Anything on the web is available for access

[dbrutus]

I like your sig.

Funny enough, I got surveyed by "the motion picture industry" this weekend. My response was "the motion picture industry? You mean the MPAA people? Nothing personal but screw them [click!]"

Re:Anything on the web is available for access

[MightyYar]

Mmmmmmm... ass burgers.

Re:Anything on the web is available for access

[BadAnalogyGuy]

To quote the submitter:

we're safe on copyright issues

So it actually doesn't sound like that's what he's worried about.

Re:Anything on the web is available for access

[SatanicPuppy]

This is marked troll, but frankly this is the prevailing wisdom regarding scrapers and I don't see it as trollish to point that out.

Coming from a guy who spends a non-trivial amount of time dealing with scraper-related problems, I've never met someone who felt that our hard-compiled content was anything other than their god-given right. I've dealt with complaints from people running scrapers that got blocked who literally could not understand what the problem was, even really abusive scrapers that looked more like a DDOS than a scraping.

Frankly, and I'm saying this as the guy who could be blocking the article posters scraper in a week or two, this problem needs to be dealt with by the content providers. No law is going to stop abusive scraping...Likely they'd just offshore themselves if you bothered to try. You need to treat it the same way you'd treat any other security problem.

Re:Anything on the web is available for access

[cnoocy]

Ghost:
        So what you're saying is, when you take candy from a little bitty baby -
Zorak:
        I'm doing him a favor, for cryin' out loud!
http://www.c4vct.com/kym/sg/scripts/njcandy.htm

Re:Anything on the web is available for access

[joto]

Yeah, I bet that told them off!

Re:Anything on the web is available for access

[dbrutus]

It was immensely satisfying actually. What worries the survey people is that collectively fewer and fewer people are answering surveys at all. They do tally the reasons.

Re:Anything on the web is available for access

[Anonymous Coward]

>What your boss is trying to do is not unethical, just cheap.

So if there's a sample platter of lunchmeat in the grocery store I'm allowed to take a whole ham home for free? The ToS is a click-through agreement, which except in egregious circumstances is binding. While "lower cost" is generally good, don't expect a "citizen of the year" award for organizing raids on local gas stations food marts to haul off as much as you can.

Short answer... "no".

[argent]

If your boss asks you to do something illegal, don't. If he doesn't agree, you should probably be looking for a new job, already. If he's willing to play these kinds of games with another company, what makes you think he won't do the same to you?

Re:Short answer... "no".

[Anonymous Coward]

well, depends if you need the salary to live on;
if so, keep the job, while you find a new one

Re:Short answer... "no".

[Anonymous Coward]

If your boss asks you to do something illegal, don't.

Illegal? Running a screen-scraper isn't illegal.

If he doesn't agree, you should probably be looking for a new job, already.

Because you've got his request in writing to break the law. That is quite a bit of leverage...

Re:Short answer... "no".

[eleuthero]

I believe Dante called it a "circle" - but hey, we are in the modern age--is the 6th cycle before or after

1) my computer executes the instruction

2) I put the load in the dryer

3) a Harley or a Suzuki (or maybe quality... in which case... 6th one down might be... BMW?)

4) ...
?

Re:Short answer... "no".

[91degrees]

Illegal? Running a screen-scraper isn't illegal.

Copyright violation and breach of contract. Not criminal, but not exactly legal either. It might be possible to apply some anti-hacking laws. Legislation tends to be quite vague here which, although unlikely, could make it possible to charge them with something and the OP would be directly liable.

Re:Short answer... "no".

[Alpha830RulZ]

Violating a web site's TOS is unlikely to be illegal. At most, this is a violation of a contract between the OP's company and the site operator. More likely, it's just use of the site which is declared to be abusive by the site operator, and the operator's only recourse is to black the scraper.

Re:Short answer... "no".

[clone53421]

Copyright violation

OP already clarified that they wouldn't be violating copyright.

breach of contract

The TOS isn't a contract. Unless, perhaps, they forced you to click "Yes, I agree to these Terms of Service" before they served you any pages. And in any event, all breach of contract means is that the scrapee is no longer obligated to uphold their end, which is (probably) to provide the content to legitimate users. If you're not a legitimate user according to their TOS, they can give you goatse instead and you really can't complain...

Re:Short answer... "no".

[91degrees]

Would you bet your house on that? That was a question from our legal guys when I questioned their overly conservative opinion on the legality of something.

Re:Short answer... "no".

[argent]

Violating a web site's TOS is unlikely to be illegal.

It has already been treated as unauthorized access to a computer system in existing cases.

Re:Short answer... "no".

[kubajz]

I like this argument, and it also has a cute flipside. Try asking your boss this: if he expects you to break some rules now, does he expect you to keep the rules he is setting? I'd say it is definitely worth standing up and saying - I will not do this because I feel it's bad. But you can also count on me not to do anything bad to you. That's a fair deal isn't it?

Re:Short answer... "no".

[Anonymous Coward]

if your boss asked you to do something illegal, tell him to do it and that you will have no part...If you get fired then you are in the money. They cant fire you because you wouldnt help them steal. But always cover your as$. send emails explaining the risks to your boss and the HR department. This will look better on a resume than i did what i was told.

Re:Short answer... "no".

[Anonymous Coward]

It's not illegal, but it is against the TOS. There's a difference.

Re:Short answer... "no".

[shentino]

5) Profit!

Re:Short answer... "no".

[RomulusNR]

you should probably be looking for a new job,

Good fucking luck with that.

Look, a business's job in a capitalist system is to make money. When ethics, rules, laws, etc. get in the way of making money, either weigh the risk and ignore them, or (if you have the means) try to change them. The goal is not to follow rules. The goal is to maximize ROI.

So there's two problems with the empty, unhelpful "go get a new job" argument. One is that there is no reason at all to expect that your new job will be any less likely to ignore/break ethics/rules/etc... particularly as some industries find it hard to maintain income through normal means these days. (See previous paragraph.) And the second problem is that, in regards to "these days", finding another job -- particularly a stable job -- is going to become more difficult. If you have a stable job, it's smart to keep it.

In other words, if you're ready to be a hermit, go for it.

Re:Short answer... "no".

[argent]

When ethics, rules, laws, etc. get in the way of making money, either weigh the risk and ignore them, or (if you have the means) try to change them. The goal is not to follow rules. The goal is to maximize ROI.

Yes, we're perfectly well aware that fines for breaking laws is often treated as a cost of doing business. That applies to laws about paying employees what they're owed, honoring agreements with employees, and so on. Oddly enough, these kinds of things go along well with each other.

And in any case, I think you're a lot more cynical than you should be. Companies are run by people, not programs, and the kinds of people you really want to work with tend to have an aversion, when it comes right down to it, to breaking the law. If they don't, how stable could ANY job with them possibly be?

Whether you go along with them or not, you're already on quicksand, and you should be looking for another gig.

I have in the past had this kind of discussion with managers. I've never yet had to go beyond "you know, that's probably illegal, and I don't feel right about doing it even if it is legal"... what I found out was that they didn't actually realize that they were pushing the envelope.

If I had to go beyond that, it'd be time to get that resume out on the street again.

Really?

[Anonymous Coward]

You can quit, whiner! If my boss asked me to rob a liqueur store, I wouldn't conduct a poll on the police fraternity league website first. I would quit and then report him.

Re:Really?

[j-pimp]

You can quit, whiner! If my boss asked me to rob a liqueur store, I wouldn't conduct a poll on the police fraternity league website first. I would quit and then report him.

I would report him and then ask the police if I should quit. They might want a mole.

Re:Really?

[dword]

And your children would starve if you couldn't find another job in time.

Uh...

[Anonymous Coward]

No. By your own admission you think its wrong. Next?

Sigh

[MyLongNickName]

Okay, this one is simple. You know what is right and what is wrong. The reality is that 99% of the folks will do what the boss asks without even raising a fuss. The reality is that you will be damaging your career if you don't go ahead.

Now, the other reality is that shit flows downhill. That is, if this project gets questioned, the boss will claim ignorance, and put the blame on you. Your job is to cover your ass.

Email is a good documentation tool. "Clarify" the request, asking if this is what he intends for you to do. Remove the emotion. Put in only facts. Put in a piece about your not being sure, but this may be a violation of terms of service. Ask if he wants you to proceed. Forward your sent email to a personal account.

By the book. This one is so simple that it should be in the FAQ.

Re:Sigh

[VShael]

Email is a good documentation tool. "Clarify" the request, asking if this is what he intends for you to do. Remove the emotion. Put in only facts. Put in a piece about your not being sure, but this may be a violation of terms of service. Ask if he wants you to proceed. Forward your sent email to a personal account.

I've worked in a large company, who's name I can't reveal for fear of litigation but essentially, using email in a CYA fashion would get you fired. (Terms of contract, they can end the contract at any time for any reason, but the money was good)

This was company policy. Not just for security (sensitive things at that company) but also to protect against future legal action. No traceability.

Sad to say but in this day and age, such paranoia is commonplace. (Just look at all the elected Republicans who used private email accounts to conduct official business, just to avoid the FOIA.)

Re:Sigh

[MyLongNickName]

I've worked in a large company, who's name I can't reveal for fear of litigation but essentially, using email in a CYA fashion would get you fired. (Terms of contract, they can end the contract at any time for any reason, but the money was good)

Bullshit. What do you put in the subject line "This is a cover my ass email"? You are only clarifying what the boss is asking for. It is basically to be used if ever there any question about what you were told to do. There is not such thing as a "you can't cover your ass in an email policy". The only thing that could be prohibited is forwarding to a personal email address. If that is the case, print the sucker out.

Re:Sigh

[bwalling]

You're right in that it is simple, but you're wrong about the solution. The solution is to find another job. It's important to stick to your morals and ethics and not just "do what you have to have to do to make a living".

Re:Sigh

[discord5]

This one is so simple that it should be in the FAQ.

Actually, it's in "Asscovering 101 : A guide to sleeping comfortably at night".

Re:Sigh

[sydb]

It's only any good if the other party co-operates. The boss can easily phone you or walk up to you and say "Yes I want you to do it." and you have no record, and for many people this is their default mode of operation because that way no-one can pin anything on them. Unless they're singing their own praises, when everyone gets cc'd in.

I used to find it infuriating but fury gets you nowhere in the workplace.

Re:Sigh

[realisticradical]

Really? Were you working for the mafia?

That kind of policy makes me think of companies like Enron, Arthur Anderson, or the cigarette companies. Any company that would fire people for trying to protect themselves from company sponsored illegal or unethical activity must be engaging in lots and lots of it. The best part it it sounds like the policy is simply a CYA policy to protect those at the top.

The part I don't understand though is how did they differentiate between CYA emails and actual questions about projects? If, "Sorry boss, I was a bit confused at the meeting we concluded that I should go about the project by [illegal activity]" gets you fired how did they ever get anything done?

Re:Sigh

[Chrisje]

So wait, when someone asks you to invade Iraq and kill anywhere between 500.000 and 1.000.000 natives your answer would be "I'll e-mail you that this is a rather bad idea, but if you clarify the request, I'll do it"?

Great thinking there, Einstein.

If your boss asks you to do something morally questionable, tell him you can't and take the consequences.

By the book my ass. Who wrote the book anyway? Depending on the book you're reading, the consequences of such a line of thinking could be disastrous.

Re:Sigh

[mollymoo]

You must have some pretty weak employment laws where you live.

Re:Sigh

[MyLongNickName]

You bring up a good point which leads to lesson #2: Written trumps verbal. If shit hits the fan, you halve your email. if your boss then says that he verbally told you not to proceed, you only have to say that you have no recollection of any such conversation. He is on the defensive as he has nothing to back it up. If he was "appalled" at the thought of breaking the TOS, then he would have written back and clarified.

Now, if you want to double cover your ass, give him status reports via email. Ask questions. You are covered.

Now to answer some other questions about whether to quit or not. You have to make that decision on your own. For screen scraping, I wouldn't quit over something so mundane. Sorry. Especially if you are a grunt. You voice your concerns, and go on. The reality is that 4 times out of 5 if you voice your concerns like this in a written manner, that the boss will back down. I have faced it twice in a grunt position with two different managers, and both times I got thanked for bringing it to their attention. It is all in how you deliver it. If it comes across as "I am ethical and you are a piece of shit", then your career is hurt. If it comes across sa "I am trying to look out for your well being and that of the company", it can be a positive. Wording is everything.

Re:Sigh

[jefu]

I'd suggest actually getting it on paper - calmly and rationally lay out what he wants you to do and your objections so that it looks like a memorandum of instructions to you and get him to sign and date it. Make sure that he understands that if he does not sign it, you won't do what he has requested. Then make a couple of copies (and one for him) and keep the original in a safe place. You might also make it clear that you think the company might well be sued over these actions.

Of course, this is likely to get you fired (perhaps not immediately, but eventually). But it is also likely that he'll decide not to sign it in which case you're off the hook. Keep the paper copy anyway with a note to the effect that he would not sign it -- it may well happen that he finds someone else to do the dirty work.

Re:Sigh

[MyLongNickName]

Thanks for the strawman argument.

There is a point where you have to draw the line. If you are morally equating screen scraping which is, at best, ambiguous with the Bush regime's actions, then I think you have got an issue with perspective. I would be willing to bet if we looked at your life, we'd find actions you have taken that others would view as more unethical than this.

Re:Sigh

[jalefkowit]

Screen-scraping some Web sites is hardly morally equivalent to launching an unjustified war that results in the deaths of hundreds of thousands of people.

Re:Sigh

[eleuthero]

In which case, maybe he should just print the email with headers?

Re:Sigh

[nine-times]

Email is a good documentation tool. "Clarify" the request, asking if this is what he intends for you to do. Remove the emotion. Put in only facts. Put in a piece about your not being sure, but this may be a violation of terms of service. Ask if he wants you to proceed. Forward your sent email to a personal account.

Email is good also because you can CC people. You have to be careful because politically you might not be able to do something quite as bold as copying your boss's boss, but you might be able to bring someone else into the conversation. Think about whether there's someone who might have influence over your boss, would probably be on your side, and who you might plausibly CC innocently.

Also, it helps if you can make a business case for why you shouldn't be doing these things, beyond the ethical issues. Bring up possible legal ramifications (not just "it's illegal" but "what if we get sued?"). Raise the issue that, since you're just screen scraping, you might have to redevelop if they restructure their page. See if you can find some way in which using multiple accounts will cost more in development or maintenance that it would to get a single pay account.

I know that we'd all like to think that being on the right/legal side of the argument is enough, but in reality it helps to have money and politics on your side.

Re:Sigh

[houghi]

Email is indeed the best option to cover yourself. See who you can include as CC without making it too obvious that this is just to cover yourself.

When he mails the response to go ahead, mail him and all others a reply (in case he did not do a reply-to-all) to confirm that you are going ahead as HE asked you to do.

If you feel seriously bad, ask a friend to mail said company to pay attention to IP addresses of the range W.X.Y.Z. If you told in your first email to your boss that they might block the whole company, let them send the email the day after you are finished with the project. Do not send the mail yourself.

Then you can even send a "na na na, told you so" kind of mail once you are blocked. This will make them think twice about doing it a second time AND your feedback will be higher valued. Management is generally not impressed if they waste money on a protect where it was said that it would blow up in their face. Obviously only if there are more people involved besides your boss.

Re:Sigh

[dbrutus]

That's when you write down their verbal approval in an email back to them.

"Thank you for stopping by today to give me the verbal go ahead on project x. The legal analysis needed to go through the TOS on site y and z was really beyond me so your clarification that we weren't violating their TOS was very helpful. As we discussed, I'll get right on it."

You're 'praising' your boss. You're documenting the conversation. If you've 'gotten it wrong' he had a clear opportunity to fix things before money was wasted/laws were broken. You've established that you're not a lawyer but that you assumed that your boss was procuring competent legal advice. In short, it's the best CYA you can get.

Re:Sigh

[Anonymous Coward]

Most states are at-will employment states, unless the OP is on a contract you can be fired at any time for any (legal) reason.

Re:Sigh

[MyLongNickName]

I am going to disagree with you here. CC is not a good idea. It puts the recipient in a defensive mindset. A simply email will most times get the project canceled (at least that approach to it). If he is the only recipient, in his mind he has dodged a bullet. He can simply let the matter drop, and no one is the wiser. If the sender's point is to get in a pissing match, then the sender will lose.

I have been in this position and it has worked for me. The key point is humility. If you are truly looking out for your boss and the company, it will show through. If you are looking to prove yourself as the better person, that too will show through.

Re:Sigh

[SanityInAnarchy]

If your boss asks you to do something morally questionable, tell him you can't and take the consequences.

Depends how morally questionable. What's the impact?

Your example is borderline Godwin:

So wait, when someone asks you to invade Iraq and kill anywhere between 500.000 and 1.000.000 natives your answer would be "I'll e-mail you that this is a rather bad idea, but if you clarify the request, I'll do it"?

We're not talking about killing hundreds of thousands of people. We're talking about scraping a fucking website. At the end of the day, worst case, this starts costing said website some money, and they either sue your company (and your CYA will protect you), or they'll start blocking you, or they'll feed you bad data.

In either of the above cases, you've cost someone some money -- not hundreds of thousands of lives, but some increased bandwidth bills and some time -- and you've also taught your boss a lesson he'll never forget.

Of course, if you're going to be idealistic about it, why would anyone be working for this tool in the first place? If you can answer that question, you might also be able to answer why someone might want to continue working with this tool.

Re:Sigh

[houghi]

Then send a mail to confirm what was said. I do this all the time with other things. First I request e.g. a change in procedure. I get no reply. Then after several mails, I send then the proposal. Still no reply.

Then I send a mail stating that unless they tell me otherwise, procedure XYZ will be in effect starting Monday. I have had people very upset and the yelled at by THEIR manager that they did not read their mails and now they should stop whining and deal with it. :-D

So even if they do say nothing, you can still state your intentions and to why you are doing it. e.g. "Hello, just to inform you that I will be working on breaking and entering http://slashdot.org/ next Wednesday as you requested. Time it will take is probably 60 seconds. I will keep you updated if I succeed in stealing the content as you asked me to do." OK, perhaps a little less obvious, but still.

It works even better by people who never reply. (It is also the reason why I do not mind being the person making the minutes from a meeting. I get to decide what was important and what not)

Re:Sigh

[chrysrobyn]

I've been working for an American... er... Globalized company for 9 years now. Sometimes you have to do things you don't like. Sometimes you have to do things that don't seem right or obvious, but procedural e-mails like this one are not strictly CYA, they're also for documenting procedures. If anybody has a question as to what you did or on whose authority, it's good to have that documentation. This can be used for deadline tracking or promotion fodder. It's far easier to forward a procedural document along with "It's online now" to some line managers than a phone call, and that forwarded document can help careers all around.

It's only any good if the other party co-operates. The boss can easily phone you or walk up to you and say "Yes I want you to do it." and you have no record, and for many people this is their default mode of operation because that way no-one can pin anything on them. Unless they're singing their own praises, when everyone gets cc'd in.

"Thanks for the phone call. I understand that I am to amend the below procedure by inserting a 20 second delay with a 5 second standard deviation in order to reduce visibility in the logs, but otherwise the description is as intended. I'll have it in place by the end of the week."

Re:Sigh

[Rennt]

I don't think Chrisje was attempting suggest that warcrimes and screen-scraping are morally the same thing, but that there is a similar mental process (or a fundamental human weakness) involved that makes people do things they normally wouldn't do if they are told to by their boss. See the Milgram experiment for more detail.

It might have been a little overstated, but strawman argument it is not.

Re:Sigh

[Alpha830RulZ]

I also work for a large company, whose name I can't reveal. The lawyers here recently turned on an email journaling application that archives -every- piece of email, due to recent court decisions that they interpret to require this.

You don't need to send copies to your private address. Just note the time, date, subject, and adressees of the email. If it comes to court, your lawyer just subpoena's the archives.

Now, I'm a belt and suspenders guy, so I'd probably have a printed copy or two.

Re:Sigh

[MyLongNickName]

On a similar note, email is great for lifting others up. A few years ago, I got promoted to a mid-level grunt, and worked a lot more with other business units. In big organizations, units don't play nice with other units. On the few occasions, someone went out of their way to provide good service and actually be helpful, their boss got an email letting them know. Maybe two or three emails a year, but those folks are the ones who are going to get the bigger raise and have chances at promotions.

Unfortunately, 95% of all emails about people are complaints. Do your job well, and the best you can hope for is to be ignored. I have personally tried to reverse that now that I am a low-level manager. I can't change the world, but I can influence my dusty corner of it.

Re:Sigh

[N1AK]

I've had this happen in the past, and it really isn't the get out that they want it to be if you are willing to not let them steam roll you. My general response is to send them another email, documenting one additional unimportant aspect of the job "I expect work on this project to be completed in around 2 weeks" but basically just as a cover for saying "Thank you for explaining to me in person why you do not believe the concerns I have aired are an issue". Hell they are gonna know you are pinning this shit on them, but if it really is such a dubious idea that they daren't be blamed for it then you have even more reason to cover your own ass.

Re:Sigh

[Anonymous Coward]

I've worked in a large company, who's name I can't reveal for fear of litigation but essentially, using email in a CYA fashion would get you fired.

You don't have to tell us, but if the company's initials are P and G, don't reply with just a <cough>.

Re:Sigh

[steelfood]

Follow up with a confirmation e-mail.

Re:Sigh

[poot_rootbeer]

The boss can easily phone you or walk up to you and say "Yes I want you to do it." and you have no record, and for many people this is their default mode of operation because that way no-one can pin anything on them.

When this happens, I simply send another email that says "Because you gave approval for this approach in our conversation this morning, I am now proceeding with it." If the shit hits the fan and the muckety-mucks are trying to decide who the scapegoat should be, a second-person account of the approval is better than nothing.

Re:Sigh

[VShael]

If I was dumb enough to try a CYA email, I would receive an email along the lines of

"Of course not! Where did you get that idea?! That would be illegal, and a gross violation of our company's ethics policy!

Have you been doing something like this already without our knowledge? Come by my office immediately."

And no, it wasn't the mafia.

Re:Sigh

[VShael]

No, not bullshit.
This is explained to all new hires. Clarifications are to be done face to face. Not over the phone, not over email.
Email is considered a completely permanent record (even if it's not actually permanent in practice) and would be the equivalent of having a politician speak to a journalist on the record.

ie. you don't ever do it unless you have to, and then, you clear it with the press office first.

Well, the company I'm talking about was exactly like that. (But not politicians and journalists.)

Re:Sigh

[VShael]

You must have some pretty weak employment laws where you live.

You must have never worked as a contractor in your life.

It's not illegal (or even uncommon) to be let go on no notice with no warning for no reason.

Re:Sigh

[Anonymous Coward]

Heh, you haven't worked in a company like this. You send an email to your boss that mentions a couple normal things and slips in "And as you directed, I'm scraping the site that says don't scrape them." Your boss sends back an email that says "Gee, you have a lot of incorrect assumptions in this mail, I'm scheduling a meeting for us to clear this up." Then he tells you to your face that you're doing it or you're not a team player -- it's OK if have any more communications problems, he'll just note it in your file for the next review.

If they don't want to leave a paper trail, they won't. It'll just be your word against theirs, and they're the experienced manager and you're the employee with the poor work history. Shit rolls downhill.

The only thing you can do with a workplace like this is leave. Tell your friends to avoid it. Give the BSA or whoever an anonymous call if you think it'll help.

Re:Sigh

[Digital Vomit]

lesson #2: Written trumps verbal. If shit hits the fan, you halve your email.

How does splitting your email in two help?

Does it, like, give you twice as much protection or something?

Re:Sigh

[MyLongNickName]

How does splitting your email in two help?

Does it, like, give you twice as much protection or something?

Yes. Just like condoms.

Re:Sigh

[sheldon]

Clearly this isn't a financial company as SOX compliance requires a written document trail of why decisions effecting profit/loss were made.

Fascinating, and while I actually disagree with the contention over this particular story, i wouldn't work for a company that questioned me writing a document trail for a requirement.

(In this story, I'd go ahead and do the screen scrape pointing out the caveats. A TOS violation won't get you in legal trouble, it'll just get your companies IP address blocked from the website.)

Re:Sigh

[Anonymous Coward]

I had a boss that would verbally respond to my emails ... until I started emailing him a summary of every conversation that we had, bcc'ing HIS boss, (INCLUDING non-work conversations!) asking him to verify that my understanding of the conversation were the same as his.

Re:Sigh

[Kjella]

A lot of the United States have at-will employment, but many contractors don't have any job security anywhere. They might if you've got a fixed percentage contract, but if you're only billing hours based on work assigned to you they can simply stop assigning work to you and in any case they can always find a reason to not renew your contract much easier than terminating an employee. Even in at-will states his boss will question terminating people without warning, but rarely if ever not renewing a contractor. The only thing keeping this together is that professional contractors also typically have a professional reputation to keep which is the only thing keeping them employed. If you're doing something illegal you're asking me to fuck with my career, and I wouldn't have that.

Re:Sigh

[cparker15]

All very good points. CYA.

In addition to what the parent said, does your boss have a boss? Does your company have a compliance or HR department? (Although watch out, because someone in the HR department could be friendly with your boss! Happened to me. All communication with HR was supposed to be confidential, and the HR manager just went back to my manager with my concerns, naming me as the source.)

Re:Sigh

[mollymoo]

I worked as a contractor for a good few years. No reason at all is fine, but some reasons are illegal. They can't get rid of you for being black, for example. Even if you were a contractor and your contact said "you agree to be sacked for being black" it would still be illegal. On a practical level it would be really hard for a contractor to prove that there was a reason and that reason was illegal, but that doesn't change the principle. Even contractors have some rights.

Re:Sigh

[Anonymous Coward]

I KNEW it was them!

Re:Sigh

[Anonymous Coward]

We did it for oil, so the US and the rest of the world wouldn't grind to a halt.

Death due to starvation and lack of needed medicine, clean water, etc, all depends on oil.

The dead Iraqis are just a side benefit.

Re:Sigh

[rtfa-troll]

SOX complliance doesn't have anything to say, however, on the question of whether that written record should match with the reality. All you need is some reasonable legal resason for each decision. A record of slight irrational madness in your past decisions will help make almost anything "reasonable" for some lawyers definition of reasonable. For example, if you can demonstrate that you bought game consoles for all your employees to improve productivity, then buying a personal fighter jet for the same reason will be perfectly acceptable.

Re:Sigh

[Anonymous Coward]

That's when you follow up with an email to confirm what was stated on the phone.

"Per our earlier conversation, we agreed to do..."

If he/she doesn't email back a clarification/denial, then it's reasonable evidence of agreement.

Re:Sigh

[Anonymous Coward]

The way to get around that is to send a confirmation email back to your boss saying per your verbal instructions you will be going ahead with that project and that if this is in error please email you back immediately.

Sure you won't have an email from your boss but you'll have the next best thing. The implication that the choice was not yours and at the very least that your boss had a chance to stop it and didn't.

Re:Sigh

[Anonymous Coward]

Document the other party's offline response, too. Send an email like, "Boss, confirming that we are going ahead as you ordered, specifically, (blah blah). Let me know if there is any change of plans." Then, documentation-wise, the ball is in their court if they dispute the agreement later.

Re:Sigh

[DrgnDancer]

Ok, I wrote a long post about this earlier, but I'll try again. Moral and ethics are different. Morals are absolutes "killing people is bad," ethics are situational "Killing people is bad, but if I don't kill this guy he'll kill the man he's hitting with a bat." Morals are your "code;" ethics are how you apply your code to real life. If my sick five year old needs health insurance and $200 a month in drugs to survive and I quit my job over screen scrapping, I've either have a really screwed up sense of priorities or I made the wrong ethical choice. Even though screen scrapping may be against my morals their may be other moral priorities that prevent me from quiting over it.

In the real world ethical people must often do things that are morally wrong to prevent other things that are morally wrong. In this case the screen scrapping is morally wrong, but at a much lower level of priority than many other things that are morally wrong. It's also morally wrong to let your dependents become homeless while you look for a job.

Re:Sigh

[Aristos Mazer]

> It's only any good if the other party co-operates.

Not a problem. After they walk away from your desk, just send them an e-mail thanking them.

"I appreciate you stopping by my desk today to approve the project. Just so we are both on the same page, here's what I heard you want me to do..."

You may quickly get a "that's not what I said" e-mail in response. Then just ask them to clarify, logging every reply.

Re:Sigh

[vldmr_krn]

Okay, this one is simple. You know what is right and what is wrong. The reality is that 99% of the folks will do what the boss asks without even raising a fuss.

The reality is that you will be damaging your career if you don't go ahead.

Now, the other reality is that shit flows downhill. That is, if this project gets questioned, the boss will claim ignorance, and put the blame on you. Your job is to cover your ass.

Email is a good documentation tool. "Clarify" the request, asking if this is what he intends for you to do. Remove the emotion. Put in only facts. Put in a piece about your not being sure, but this may be a violation of terms of service. Ask if he wants you to proceed. Forward your sent email to a personal account.

By the book. This one is so simple that it should be in the FAQ.

If that's by the book, I want to know what book you're reading.

You tell your boss that you won't participate in this project, or if you feel strongly enough, that you will have to resign if the company goes ahead with it.

The poster has ethical issues with the project, he's not asking how to make sure his ass is covered after he does it.

Re:Sigh

[MyLongNickName]

If the personal has ethical problems with something and is coming here to ask "should I do it or not", then he has some bigger issues.

The fact is, this is a very borderline case at best. Not industrial espionage, not theft. Screen scraping is what is being brought up. Questionable practice? Maybe. Stupid idea. Definitely.

The other fact is that if the original poster brings up his objections in writing, that the manager is very likely to back down. You can go in with your refusal, and feel all good about yourself, but there are better ways to go about it.

Re:Sigh

[Anonymous Coward]

Absolutely right. I had this problem at a previous job where I had a verbal agreement with my boss that I would get time in lieu for any overtime worked on a particular project. Several months later I resigned and had a lot of time owing to me. Every time I spoke to him, he agreed that I should get the cash equivalent as they weren't able to give me the time off. I would email him a summary of our conversation and cc the HR manager. It took several weeks before he would commit to it in writing.

Avoid doing that if possible

[basiles]

I am definitely not at your place, but I would avoid doing such unethical & unfriendly stuff. Rememver, that such kind of behavior would follow you much more that your boss. In a few couple of weeks, your boss's name would have been forgotten, but your name would be remembered and associated with such kind of behavior. A few years ago, I did a web crawler, above Larbin, to measure statistics on web pages. I tried to have it be web friendly (i.e. robots.txt aware, and not to much load on a single site). However, I suppose that you are in the USA and I guess the employment situation is bad there, so I understand you want to keep your job. But try to also keep your reputation (which lasts longer than a single job). Good luck. Regards.

Simple Progammer, No Consequences

[Anonymous Coward]

"My question is am I wrong about the ethics? If I do need to walk how best can I handle it without damaging my reputation and future employment opportunities?"

Well, you can escape most of the "consequences" of doing something like this [basically none] by virtue of being a simple programmer.

I was just following orders is the good way to go. Make sure you document everything, including your objections.

Real engineers on the other hand, do not have such a simple recourse.

Re:Simple Progammer, No Consequences

[RMH101]

With the benefit of my Futurascope, I call Godwin.

Re:Simple Progammer, No Consequences

[Anonymous Coward]

Premature Godwin disallowed.
It may be inevitable, but you still have to wait until it actually happens.

Re:Simple Progammer, No Consequences

[Anonymous Coward]

Nazi fascist :-P

Re:Simple Progammer, No Consequences

[RMH101]

{high fives}

Does your boss understand...?

[entgod]

Does your boss understand that what he is asking is unethical and possibly illegal? If not, try to explain it to him.
If he won't listen to you, you should probably start looking for another job.

Re:Does your boss understand...?

[Andy Dodd]

Depending on this guy's company (specifically what resources he has available), he may also wish to talk with his company's ethics office or legal department (if they have an internal ethics office or legal department - larger companies always do, smaller companies may not). In fact unless he can convince the boss himself very easily that this is a VERY BAD IDEA (it is), he should immediately walk away from Slashdot and to wherever his company's ethics or legal department sits.

Spammer logic.

[argent]

If you can access it, it was designed to be accessed.

So you're totally behind email spam, you don't think spam should be considered unethical, let alone made illegal?

Re:Spammer logic.

[BadAnalogyGuy]

Should I be against spam for any other reason than I am annoyed by them?

I don't think spam should be any more illegal than billboards, flyers, or direct mailings.

Re:Spammer logic.

[SatanicPuppy]

That's a Bad Analogy(tm xD), since no one sneaks into your house and installs billboards, and flyers, billboards, and direct mailings both have a significant cost attached for the initiator which prohibits abuse on the same level as spam.

Re:Spammer logic.

[d3ac0n]

I don't think spam should be any more illegal than billboards, flyers, or direct mailings.

The flaw in this argument is that your three counter-examples (Billboards, Flyers and Direct mailings) are paid for entirely by the SENDER. IE: Billboards are paid for up-front before they are mounted, Flyers and direct mailings have printing costs paid up-front and delivery costs (either the local govt. mail service or paid people to manually give it to you) paid up-front as well.

Spam, on the other hand, is largely delivered on the backs of OTHER payers. Both through the incredibly high bandwidth costs (HOW much of the total Internet traffic is Spam now?) and through ancillary costs such as costs for software and hardware to filter Spam out, and human costs in terms of work-hours wasted manually going through spam. Not to mention the costs to people and networks infected with Spam botnets.

This is what makes Spamming SO profitable, and why it won't go away. Because the costs for Spam are decentralized to millions of people otherwise not directly involved, even a return as little as .01% will turn a HUGE profit. This just doesn't work in the regular advertising world. It's also why it's Illegal is several countries now. It is essentially stealing service from millions of other people and generating millions of dollars of expense for hundreds of companies around the globe, for what are largely scamming and phishing operations.

Get it?

Good.

Re:Spammer logic.

[BadAnalogyGuy]

I suppose that the visual damage caused by billboards is a non-cost. Or that my arthritic hands are not aggravated taking junk mail from the mailbox.

No, I do not "get it", if "getting it" means singling out a one form of an accepted practice simply because it differs in scope to its relatives.

Re:Spammer logic.

[DodgeRules]

Companies that use billboards don't illegally come onto your property and erect a billboard. Junk mail advertisers don't sit in the back seat of your car while you drive around and "make" you deliver their ads. Spammers illegally access computers that they have no right to access to deliver those ads.

Re:Spammer logic.

[SQLGuru]

SPAM has a cost to it as well. That cost could include any or all of the following:
- Cost of maintaining mail server to send from
- Cost of finding an open mail server to send from
- Time spent composing a SPAM-filter-passing message
- Time spent learning new SPAM-filter-fighting technique
- Bad speling and gramer lessuns
etc.

Granted, the cost per contact is low for spam, but legitimate businesses also use this same low cost approach....they are just better about obeying the opt-out clause.

Layne

Re:Spammer logic.

[eleuthero]

In Spain, billboards are by and large illegal because they are considered to pose traffic hazards. Flyers and direct mailings lend to environmental problems depending on how they are printed (since they invariably find themselves in the dump). Spam by email wastes energy and bandwidth that might be used otherwise. Let's make them all illegal and force advertising to come up with something new.

Re:Spammer logic.

[Anonymous Coward]

Official Heretic from the "Church of Global Warming".

Spam doesn't exist. Despite the incontrovertible evidence to the contrary, you're lying.

Re:Spammer logic.

[MightyYar]

What about phone solicitations? Personally, I think that the do not call list is the best thing since sliced bread, even if a few shady characters still call.

Re:Spammer logic.

[Timothy Brownawell]

Should I be against spam for any other reason than I am annoyed by them?

I don't think spam should be any more illegal than billboards, flyers, or direct mailings.

Sure, as long as it has some consistent header field that lets me filter it all out without dropping any legit mail. But somehow, I don't think the people spamming from botnets are going to care about little details like that...

Re:Spammer logic.

[dogmatixpsych]

Ok, I'll feed the troll, even though it's unrelated to the topic at hand.

There is a difference between the science of global "warming" ("climate change") and the religion of global warming. The first is good science. The second is not science or is based on poor science. Right now the religion of global warming (think, Al Gore or paying to offset carbon credits) is much larger than the science of it. You can believe the science of global warming without believing the religion of it.

Re:Spammer logic.

[Anonymous Coward]

What? There's nothing inherent in spam that says, "Hey, I'm accessing a computer illegally!"

Don't confuse botnet originated spam with spam in general. If your complaint is with botnets, say so. If your complaint is about spam, then you need to modify your analogy.

I recommend using cars.

Re:Spammer logic.

[michaelhood]

I don't think spam should be any more illegal than billboards, flyers, or direct mailings.

While this makes you ignorant,

No, I do not "get it"

this makes you stupid.

This isn't complicated.

How much does it cost you to get ads in the mail? $0.

How much does it cost your ISP, and you, to receive spam? >$0.

Get it, now?

Re:Spammer logic.

[domatic]

Well over 98% of our incoming mail traffic is spam. If I did nothing about this, email here wouldn't be useful at all. Trying to bury our legitimate business in a sea of penis enlargement ads is reason enough for me to want to these jerks shut down in the most obnoxious way possible. And since as you mention in another post you "don't get it" then I'll cheerfully get out the Cluebat and explain it:

1. We bear a bandwidth cost even for mails tagged or rejected.
2. I expend significant effort keeping the s/n ratio high for my users.
3. The bulk of that traffic comes from botnetted machines. I eagerly await your explanation of what makes that ethical or legal.
4. Much of the spam attempts to make use of system, mail client, or browser vulnerabilities. The potential consequences range from minor breaches of privacy, compromised clients, to defrauded recipients.

5. Even the spams that don't carry some sort of technically malicious payload are out and out scams. They flog products that don't work, pirated software, and other such shenanigans.

So we don't have a mere annoyance here. Criminal means are used to flood mailservers with scams and even the (very) small residue that are actual businesses with actual products and the politicos are abusing the network resources of others.

Re:Spammer logic.

[jareth-0205]

BadAnalogyGuy:

I don't think spam should be any more illegal than billboards, flyers, or direct mailings.

Fantastically appropriate username...

Re:Spammer logic.

[Anonymous Coward]

But your fuel tax pays for the road that goes in front of the billboard. So yes, you ARE paying to see that billboard.

The electric company is using the taxes and profit from YOUR electric bill to run poles and wires to that billboard, so you are paying for that billboard.

Your 'i pay for internet' argument no longer holds water. Buy a good spam filter and get on with life.

Re:Spammer logic.

[Anonymous Coward]

Just curious, what about telemarketers. They parallel spam much more closely.

Re:Spammer logic.

[TrekkieGod]

So you're totally behind email spam, you don't think spam should be considered unethical, let alone made illegal?

Uh...email spam by itself most definitely should not be illegal, and it is not unethical.

That doesn't mean that the some types of spam, or the most common methods for spamming shouldn't be illegal. Nigerian Prince-type spam is fraud (it's not the e-mail sending that should be--and actually is--illegal, it's the fucking fraud). Using spam bots to send spam should be--and most likely is--illegal because it's hijacking people's computers.

Not allowing for an unsubscription method is unethical. Providing an unsubscription method that doesn't actually unsubscribe the person is most definitely unethical.

When best-buy sends you an e-mail about their latest deals, that's spam. However, they didn't use spam bots, they're not defrauding you, and when you click on the unsubscribe link, it actually works. Therefore, that should not be illegal OR considered unethical.

Re:Spammer logic.

[argent]

The cost of a telemarketing operation is high enough that it doesn't have the scaling problems of spam.

Re:Spammer logic.

[Fulcrum of Evil]

Uh...email spam by itself most definitely should not be illegal, and it is not unethical.

It is absolutely unethical - you're imposing a cost on people without any sort of consultation. It should be illegal, but first you have to come up with a legal test and some way to enforce penalties. Good luck on either.

Re:Spammer logic.

[Anonymous Coward]

Spam exists largely because ISP's accept payment from known spammers, spammers often pay for their bandwidth. Just like cell phone spam is prevelant because it is a source of income to the providers.
Many large "legitimate" companies use mass emailing. I never want their email. Is that spam?

"Costs for Spam decentralized"? Not always. Local US spammers acquire accounts for 1-30k per month with large numbers of IP's and change the source of IP's often. It's a centralized payment, the ISP knows it's a spam customer, they still make the sale.
The incremental costs associated by the "end user" are often trivial. White lists make spam bsolete for standard business email. Out of the millions being spent on antivirus and antispam, many vendors and implementers are getting rich.

The reason spam is around is because it's push advertising. It's effective for selling useless items and ideas, and useful.
Barkers, criers and snake oil salesmen have been around for the ages. Spam isn't new, it's just a alternative delivery method.

Re:Spammer logic.

[Anonymous Coward]

Parent is correct about the distributed cost structure of spam, but fails to understand that traditional advertising has many of the same cost features; spam's cost is just easier to monetize and more voluminous.

Bandwidth is easy to monetize in cyberspace, but imagine a road where billboards can be placed, all that real estate has some social value beyond the cost of erecting another billboard. If you make the billboard density high enough, people will complain as they start to feel this 'cost'.

Filtering costs? ever watch TV and wonder if you're watching an ad or real tv? ever take time to seperate the ads from the newspaper? these are all costs to recipients, just harder to measure and (probably) smaller in scale.

I'll grant the botnet issue is relatively unique to spam, but otherwise the difference is one of degree, not kind.

Re:Spammer logic.

[Anonymous Coward]

Spam has a non-zero cost associated with sending it, and billboards cost me valuable time to ignore them, too. The only difference is in degree.

I'm against spam, sure - but I'd like to see almost all advertising vanish from the face of the earth, too.

Reputation...with who?

[steveb964]

"What other people think of me is not my business." IOW, is how others see you more important than how you regard your own integrity? Do the due diligence yourself. If the website owner's TOS do not allow what you are trying to do and you do it anyway, what does that say about your 'reputation' as a whole, to yourself, and the people whom most care about you? - stevieb

Check with Compliance Officer/Department

[Ohmaar]

I work in health care, so maybe it's different in your industry, but every hospital I've worked for has had a compliance officer with an anonymous 800-number for compliance questions. This is DEFINITELY the kind of stuff they want to know about.

Re:Check with Compliance Officer/Department

[jabithew]

My dad had something similar in our nationalised health system. The hospital administrator wanted to move some patients. What she really wanted was for him to refuse so she could blame unco-operative consultants for her problems. Instead he agreed to do so with the proviso that she wrote a letter declaring that she'd commanded the action against his clinical advice. She backed down pretty quickly.

IT guys have less power than an NHS consultant though. I still think a cover-your-arse paper trail is the way around this one. It's not really even arse-covering, just making sure responsibility falls where it's due. After all, bosses are paid more to take responsibility, right?

Re:Check with Compliance Officer/Department

[Anonymous Coward]

FYI, there is no such thing as an anonymous 800 number. The 800 number gets billed for incoming calls. Therefore they want to have a list of those calls to make sure they are not being over billed. That proof? The caller's number and time.

If you're going to call "anonymously" - do so from a payphone and make sure it's not the one near your home and that there's no camera.

Re:Check with Compliance Officer/Department

[Anonymous Coward]

There is no such thing as an "anonymous 800 number."

The company that has an 800 number pays PER CALL. The phone company bills by the call, and provides the number of every caller at the end of the month, in their bill.

Consider it a Prototype

[BlueBoxSW.com]

And make sure you have a clear chain of emails showing this was your boss's idea.

Things like this always fall apart in the end. People who are greedy and short sighted in one respect are always greedy and short sighted in other ways that eventually bite them in the ass.

Someone will come along, see the idea, and will invest a small amount of cash in their own, and end up with something much better in the end.

That person may even be you.

Re:Consider it a Prototype

[hesiod]

> That person may even be you.

If you mean writing something for someone other than the current employer, that might not be such a good idea, depending on the scope of non-compete clauses and other such things on which I am no expert.

Grey Area, But Probably Still Wrong

[segedunum]

Defining and checking for screen scraping is a very grey area because browsers are effectively screen scrapers. If you make a web site publicly available then you must accept public access, unless the load is excessive.

What I worry about is this:

Part requires scraping most of the site, parsing the data and presenting it as our own without human intervention.

I can't see you being able to do that without getting into hot water later. Even if copyright is OK it is still dodgy.

The other is, for lack of better words, a "load balancing" part that requires using multiple free accounts instead of purchasing space and CPU time for less than $2,000 USD per month. The boss sees it as "distributed" computing when in reality it's "parasitic".

Your boss is a fucking idiot. Pay the money. It will all be blamed on you when it is inevitably unavailable one of these days.

My question is am I wrong about the ethics? If I do need to walk how best can I handle it without damaging my reputation and future employment opportunities?

The fact that you are phrasing that in terms of stepping non eggshells tells me that you need to be looking for another job and reducing your stress. These ideas that you have presented from your boss are s-t-u-p-i-d, and they will almost certainly come back and bite you and not him. I'm afraid you'll have to be honest and blunt.

Why tell them?

[east coast]

Regardless of your decision in this matter it has the possibility of changing your opportunities elsewhere... that is, if you tell your future would-be employer. My question is, why would you tell them either way? Even if you "do the right thing" do you really want to wear that on your sleeve in your next interview? I think that kind of thing is just asking for trouble.

Have a prospective employer questioning your ethics or lack there of in the interviewing process is going to throw up red flags. It's a question best avoided all together.

If it's legal, you do it, if it's not, you don't.

[Anonymous Coward]

Nothing ruins your reputation more than working against your boss (as long as he demands nothing illegal from you.) You can discuss other options with your boss, and if your arguments are solid, you might convince him. What you have to realize though is that he's the boss and as such he makes the decision. Do not keep on about it.

If you can't convince him and you're having moral qualms, quit. If you stay, do the job.

Do what geeks do best

[GreyyGuy]

Fix it. He wants to do something on the cheap and look good. But the way he wants to do it is going to fail spectacularly. And when it fails, so will you. If this puts any amount of load on the services it is using, it will get picked up by the service provider. Maybe not today, but it will. And then the accounts will get turned off and possibly your IP addresses blacklisted, and then it all goes away. So give him a better solution. If he is balking at the $2k/month find a cheaper service. There is almost always one. Compare the cheaper solution to the time spent fixing it when the free service cuts you off. Provide examples of free service cutting people off.

And unless you are looking for some very specific information, I would expect someone to provide an RSS feed with something similar that is supposed to be used for this sort of thing.

Complain, then just do it.

[Anonymous Coward]

Every time I'm told to do something stupid (which is probably about 10 times a day at this point), I carefully outline in writing why I think what they want me to do is stupid and/or illegal, present that to management, and then do it anyway. If you are a programmer, you are not employed to determine whether something is legal or ethical or not. However, if you see your company doing stuff like this, it might be a good idea to look for a new job. Bad management does not limit itself to any one area of idiocy, and you don't want to tie your survival to a business that behaves that way.

Subject

[Legion303]

Sure thing. Your boss sounds like the kind of ethical guy who would accept the consequences of any illegal or unethical actions, and he totally wouldn't let you take the fall for it instead.

Oh, and can I interest you in a timeshare? A bridge, perhaps? How about some prime Louisiana real estate?

Reputation?

[Anonymous Coward]

Why would your reputation get "damaged" if you walk off a job because of this if employers claim to "only check date ranges" for references?

Why are you asking *US*?

[elrous0]

Only YOU can decide how far you're willing to go for your job. You're essentially asking us what your own ethical limits are.

Re:Why are you asking *US*?

[Anonymous Coward]

Only YOU can decide how far you're willing to go for your job. You're essentially asking us what your own ethical limits are.

By telling me how I can and cannot define who I ask to define my own ethical limits, you have defined how I define who defines them.

... Yes, I think that's right.

Business sense

[ThePyro]

Even if your boss doesn't care about the ethics of this scheme, he probably does care about ramifications to the business. What happens when you get caught? All your development work will have been wasted because they'll shut you down at the very least. There's potential for a lawsuit, which is an expensive proposition even if you win. Damage to your company's reputation may make it harder to do business. And as another poster already mentioned, this isn't exactly a gem of a project to put on your resume.

Employee Handbook?

[Anonymous Coward]

Our employee handbook covers this very nicely in the "Standards of Conduct" and "Professional Responsibility" sections. To wit:

CORPORATE INTEGRITY
To maintain the confidence of the public and customers we serve, it is essential that all
employees of (company) adhere to the highest standards of business ethics and
employee conduct. We must all strive to create a work environment based on trust,
respect, integrity, responsibility and honesty.
We must all remain alert to possible violations of the law, regulations, or business ethics
anywhere at (company). Any questions or doubts that may arise regarding a
particular situation should immediately be brought to the attention of a supervisor,
Senior Management or the President.
In general, when conducting business on behalf of (company) you should use
common sense and apply the following "S.L.I.P." test. Remember, think before you act.
- Scrutiny: Will my actions withstand public, media, legal and/or organizational scrutiny?
- Legal: Will my actions violate applicable laws, regulations and/or (company) rules or policies?
- Interests: Are my actions in the best interest of (company) and the public?
      Will any party, including myself, stand to gain an undue advantage?
- Performance: Are my actions consistent with the proper performance of my duties?

I think that failing 3 out of 4 probably makes this a no-no.

A character check?

[juuri]

Having been put in a position once before that an employer asked me to do something I found to be frankly quite lacking in a moral nature here's what I ultimately decided to do.

After considering the work for a while, both why I didn't feel like performing the work personally and why the company desired this functionality I finally decided to do the work, but inform my boss and his boss that I was uncomfortable creating this before hand and giving them clear notice of the whys.

Firstly I did the work because it was simply my job and I had signed onto the job. It's something a *lot* of people might not have given a second thought to creating, obviously as they both had no problems with the work since they asked me to continue even after raising my concerns. Secondly because it wasn't really "that bad" and having steady income of cash dolladolla bills allows me to have nice things like somewhere to live and food I wanted to see if it was something I was over-reacting to.

After completion? Yep, I still felt like shit. So I gave them my notice and told them in the my resignation letter why I was leaving and referred them to the early notification of my objections. So, for me, it was a good learning experience about myself and having done it in this manner I have no problem explaining it to future employers as my reason for leaving this particular job.

Re:A character check?

[91degrees]

There is a matter of magnitude here. Abusing peoples trust and killing 6 million Jews are slightly different crimes.

Re:A character check?

[Anonymous Coward]

I was in this position in my first job out of college. The job was for a credit bureau and I was asked to do something that wasn't illegal but was definitely unethical.

However my experience was the complete opposite of what most people have posted. After bringing up my concerns not only did the project not get implemented but I also discovered that there were a lot of other questionable activities that a credit bureau could do that we were either not doing or we were doing in a way that protected peoples privacy.

      It may have been that since the company was turning a huge profit that allowed it to be more ethical. However, I would like to think that the reason the company was doing so well was because of its employees and it had better employees than its competition because it was a more ethical company.
(our software and IT was about 1/4 our competitions and our database was significantly more accurate)

Re:A character check?

[Anonymous Coward]

Yeh, absolutley, abusing trust is disgusting.

Who cares?

[1u3hr]

would require me to break multiple web sites' Terms of Service (TOS).

A website's "terms of service" are not the Ten Commandments. They're not laws, or even moral rules. They're just what one company wants you to do. You don't work for them, why do you care? If they notice and complain, it's your boss's problem, legally; and morally, I wouldn't lose any sleep.

Only thing to do is cover your ass and get your boss to put his instructions in a memo so he can't blame you should problems arise.

Really "scraping a website" is not a moral question on the scale of collaborating with Nazis. It's a business. Other businesses are your rivals, not your friends. They'd fuck you over in a minute.

Re:Who cares?

[MightyYar]

I wrote a little script to search multiple cities in Craigslist, simply because they don't offer the function at any price. People can say I'm a jerk, but I really don't care because it saves me a lot of time.

Re:Who cares?

[Courageous]

Planning ahead of time to breach a contract, with malice aforethought, may not be as free of moral constraint as your letting on.

C//

Re:Who cares?

[HikingStick]

Who's to say that there are grades of evil? If the standard is perfection, then any failure disqualifies. If the passing score on an exam is 100%, and one student gets a 99% while another gets only 3%, does it really matter?--they both fail.

Re:Who cares?

[Anonymous Coward]

Would you *REALLY* build a business that relies on no one else noticing that you are scrapping their site?

Like you said 'theyd fuck you over in a minute'. What would stop them from doing that? Just filter out that site and load problem gone.

I would personally go back to the boss and say 'do you really want to really on the good will of other web sites?' Sounds like the dude is a bit shady and will see someone messing him over much easier.

I personally wouldn't build a system where it relies on the 'niceness' of others. Not because I am a nice guy. But because it is a fragile system where the data you depend on might go away.

Also would you REALLY want to test those TOS in a court when their IT guy shows how you are stealing the data from them. They made a good faith effort to 'wall off' the data. They probably would have a good case. But more than likely they would just cut you off and ban your ip.

Re:Who cares?

[Hatta]

If you violate a web pages Terms of Service, does that access not become unauthorized access? Unauthorized access of networked computer systems is against the law.

I doubt anyone has litigated this exact point, but it seems like an argument with a good chance of success in court. It's certainly not any more of a stretch than blizzard's contention that a copy of their game in RAM is an unauthorized copy in violation of copyright law, unless you follow their EULA.

Re:Who cares?

[tnk1]

The site provides data on free accounts. Presumably, they don't care so much that the data could be used by people for free.

Although the provider is no doubt going to say that you need to pay for the data, what you are really paying for is a paid account. This account presumably has superior characteristics which allow you to rely on this data in a business critical manner.

No provider of public accounts is going to get in the business of legal action against people who use those accounts, because that will have a chilling effect on the people who they *want* to use the accounts. Also, they will have a harder time in court proving that harm was done to them. You're just going to get your company's network banned.

Of course, if you have a compliance officer, you will want to get guidance. That is always the best action.

If you only have your boss, I'd skip the ethical posturing, you already know he's acting sleazy, and sleazes don't like to be called on it. If your job is otherwise a decent one, it may not be the time to stand on principle. Prophets and martyrs tend to be viewed favorably in retrospect, but they tend to find themselves very uncomfortable in the present.

Inform the boss that you know that the plan put forward will work, but the application will likely completely fail unexpectedly at some point in the future, possibly with absolutely no ability to re-work it to get around the failure (i.e. they IP ban your hosts).

When you have hard facts to rely on to get your desired result, drop the subjective arguments.

Now, if your boss then tries to have you get around their initial restrictions... you need to get a new job, because then your hat has changed shades from a charcoal grey to ultra-black. Its one thing to try and rip someone off using publicly available data, its another thing entirely to circumvent access controls. Then, you will be well and truly fucked.

Re:Who cares?

[The Living Fractal]

And here is a good example of how Slashdot's moderation system is ineffective.

We have multiple +5 Insightful posts on this thread. Some of them tell this person that what he is doing is wrong, and then there's this one which goes the other direction completely. Yet both are treated equally as +5 Insightful.

Had Slashdot's system allowed scores above 5 then it would be more like a dynamic poll (and be more representative of what the populace thinks about the issue) than this hard-capped false ceiling that we have now.

As far as Karma (a useless stat anyway imho) you could still cap how much one can gain on a single post, but allow the score of the post to be unlimited.

Seriously, coding this in would take, what, ten seconds?

Re:Who cares?

[Overzeetop]

Probably not. Generally, all the terms of service can provide for is termination of service. Now, if there are copyright or other IP violations, that's a separate matter, but this sounds like data which is not copyrightable. Unauthorized access usually means that you have _accessed_ a computer without authorization (such as through someone else's account). Here, they're getting legitimate accounts but _using_ them in a way that was not permitted or authorized. Note that the difference is in unauthorized _use_ versus unauthorized _access_. The Blizzard case was, iirc, a civil case, not a criminal one. While a civil case could be brought in this instance, it would be the corporation on the filing, not the individual.

Re:Who cares?

[Just Some Guy]

What contract? Care to point me to the "TOS" header in HTTP that asserts that you've agreed to it?

Re:Who cares?

[Anonymous Coward]

Thank you so much for being the first person to recognize that a website's TOS doesn't mean anything. They're not really legally enforceable and if they didn't want their data on the net then it shouldn't be there in the first place.

Its like that lady that wanted to sue google for caching her site. She should use a robots.txt or password protect her site if she's that anal about.

Re:Who cares?

[1u3hr]

Planning ahead of time to breach a contract,

What "contract"? No contract was mentioned.

Re:Who cares?

[1u3hr]

Who's to say that there are grades of evil?

Anyone with a brain. Dropping litter in the street is not as evil as theft, and rather a lot less evil than murder, for instance. If not, we;re all going to hell regardless.

Bulletin: there are shades of grey between black and white. At least 254.

Re:Who cares?

[1u3hr]

the application will likely completely fail unexpectedly at some point in the future, possibly with absolutely no ability to re-work it to get around the failure (i.e. they IP ban your hosts).

"Absolutely no ability"? Proxies?

Re:Who cares?

[1u3hr]

representative of what the populace thinks about the issue

If you read the instructions to moderators, that is not at all what moderation is supposed to do. It's supposed to highlight posts worth reading, and push ones not so out of sight. It's not meant to be a "poll" on "what the populace thinks". Otherwise every "me too" on a subject you agreed with, or "fuck you" on one you didn't would be modded up.

And if a poll was wanted, why limit it to the few moderators?

Re:Who cares?

[Anonymous Coward]

Actually, this brings up the case of Lori Drew. She is currently being charged by the US Attorney for precisely this - breaking the MySpace terms of service though 18 USC 1030(a)(2)(C).

So in actuality, if the government's theory is valid, breaking a terms of service could be criminal, beyond the breach of contract claims that could be brought in civil court.

Re:Who cares?

[HikingStick]

In the realm of Christian theology, your final statement is accurate: that "we;re [sic] going to hell regardless." That is the foundation of understanding the Christian teaching of sin and redemption.

The belief may be summarized thus: God is without sin and requires mankind to be without sin to enjoy fellowship with him. Mankind has partaken in sin--of varying degrees, yet all falling short of God's perfection--requiring God's justice (e.g., "the wages of sin is death"). God's justice requires sacrifice (i.e., "without the shedding of blood there is no forgiveness of sins"), but no mere man could offer a sacrifice great enough to appease God's wrath. Christian teaching is that messiah, Yeshua (Jesus) or Nazareth, was God himself come to earth in human form (i.e., "In the beginning was the Word. The Word was with God and God was the Word...and the Word became flesh and dwelt among us."), to take the punnishment for sin (i.e., "For what the Law was powerless to do in that it was weakened by the sin nature, God did, by sending his son, in the likeness of sinful man, to be a sin offering."). Having lived as a man without sin, death could not keep him, as it only has authority over those under the power of sin. Thus, crucified, he became accursed (i.e., "Cursed is everyone who hangs on a tree."), died, was burried, but was raised to life (since he had no sin to keep him under death's authority). The teaching continues that his resurrection is the hope of all who believe on him (i.e., trust God's means of salvation through the cross of Christ), that they, too, will one day be raised from the dead because they trust that Christ's death was sufficient to pay for their sins (i.e., making them positionally "not guilty" or having a debt [to death] "paid in full" by the death of Christ).

Therefore, to the Christian (not merely the de novo Christian), all "sin" is failure to meet God's perfect standard, no matter how gross the offense, and all are equally condemned. The Christian realizes that God is more interested in the attitude of the heart (e.g., Yeshua taught that hatred is the same as murder and that lust is the same as adultery), than in the perceived severity of the sin. That being the fact, I apologize for any times that others, claiming the name of Christ, may have belittled you or given you a hard time in the name of proclaiming their faith or in claiming that they are somehow better than are you. Your comment, "we;re [sic] going to hell regardless", applies as much to them as it does to you or did to me. As for me, I'm all in now. I trust this seemingly strange salvation that God has offered, and I hope you will consider the same. It is the only thing that has brought me peace.

Re:Who cares?

[Courageous]

The OP is somewhat loose on details, but he mentioned using free online accounts. I would assume the TOS was there...

Re:Who cares?

[1u3hr]

That being the fact, I apologize for any times that others, claiming the name of Christ, may have belittled you

I don't ask you to apologise for others. But I will just point out how hypocritical it is for you to say that while at the same time sneering at me.

Whether you are sincerely a religious nutcase or just a troll pretending to be one, I can't tell and don't really care.

Re:Who cares?

[clone53421]

If you violate a web pages Terms of Service, does that access not become unauthorized access?

No. If they send you the page on the assumption that you're going to keep the TOS, they can't claim unauthorized access if you break the TOS. You asked for the page and their server freely gave it to you, and in many cases you didn't even have to click something "agreeing" to abide by the TOS.

a copy of their game in RAM is an unauthorized copy in violation of copyright law, unless you follow their EULA.

If they can pin you for it, unauthorized copy is punishable under copyright law. If they can pin you for breach of contract, on the other hand, their only recourse is probably to not provide the services their TOS said they'd provide. In other words, they could block you or send you garbage.

Re:Who cares?

[Creepy Crawler]

Which is why Christianity is broken.

There are deeds that are worse than others. Those class of deeds are ones that one cannot undo. One can return or replace stolen goods. One can fix what was broken.

However, one cannot erase the memory of rape, or undo murder.

To claim that somehow sin is all equal is just broken, as there are certain actions that can never be undone.

Re:Who cares?

[Anonymous Coward]

Oh please. I didn't read the post the first time around, seeing as it was completely offtopic, but reading your reply prompted me to go back and find out exactly what was so sneering about the tone. There was nothing. Get over it and ignore that post like I was originally planning on doing...

Re:Who cares?

[nine-times]

A website's "terms of service" are not the Ten Commandments. They're not laws, or even moral rules.

This raises the question in my mind, "What are 'terms of service'?" Is it something akin to a contract, or is it merely a statement about what you should or should not expect from someone providing service? Because I think you have a decent point, here. If a site says something to the effect of, "we don't want you to open multiple accounts and we reserve the right to close your accounts if we believe you've been opening multiple accounts," then it's not necessarily unethical to open multiple accounts. It may just be that the company providing the service is covering its own ass in cases where there's gross abuse.

Re:Who cares?

[HikingStick]

It was not my intention to sneer at you. If that is how you perceived it, then I apologize. I, simply, desired to present my beliefs in the context of the discussion.

Re:Who cares?

[HikingStick]

While I understand what you are saying, and it makes sense in human terms, simply returning a stolen object does not "undo" a theft any more than resucitating a body (whether believed possible or not) would undo a murder. The violation occurred. The point I'm trying to make is that true Christianity is not a system whereby deeds, good and bad, are weighed to determine if someone is "good enough" to gain eternal reward (though that is an error to which many de novo Christians adhere). To the Christian, in the presence of a holy God, even the most trivial sin is damning because it is a reflection of a desire to put ourselves first, both before others and before our creator. In my eyes, do I believe that the man who raped my six-year-old neice deserves greater punishment than someone who omits reporting $40 of income on their taxes? Absolutely!, and in this world there are institutions of justice that should work to preserve the social order and punish wrongdoers, but in God's economy, both offenses are enough to keep us from fellowship with him.

If you have no desire to understand the Holy, I cannot fault you for finding fault in my argument. The teaching of the Cross (messiah's sacrificial death for our sins) has long been considered folly to those who do not believe. If you consider me a fool, then I wear that badge with honor--not because it wins me any special favor with God, but because it demonstrates that I am finally starting to care less about my own reputation (as measured by those who live in this world) and more about being faithful to proclaiming the good news that I have received.

May you know peace, and may God himself breath upon you and bless you.

Re:Who cares?

[Frank T. Lofaro Jr.]

There are deeds that are worse than others. Those class of deeds are ones that one cannot undo. One can return or replace stolen goods. One can fix what was broken.

Bill Gates should go to hell for Windows.

There ain't no fixing that!

Re:Who cares?

[tknd]

You don't have to mention a contract for a contract to exist. Verbal contracts are made all the time and can be enforced by law. But as you might guess, proving that the verbal contract existed in the first place may be much harder than had you had a written contract.

There are exceptions like for the sale of real estate. But to my knowledge (I am not a lawyer but I am taking a law class) this is not one of them. That doesn't mean that you can't find a way out of it--just that it might be reason enough for the other side to take you to court and win or at least sink you in lawyer expenses.

Re:Who cares?

[Anonymous Coward]

I believe in treating crimes according to severity.

I care about my standing in the world.

Me first, only then can I worry about others. No one's got my back but me. Got to look out for me because no one else will. God didn't save the September 11 victims.

Are some of them burning in Hell because they were going to get saved soon, but never got the chance.

Will I feel pain that will make my last burn seem like a picnic in comparison?

IS THAT F*CKING JUST?!

Oh, by definition it is, nevermind...

Those verses in the Bible about "many stripes" and "few stripes" and "until you pay the last penny" don't actually mean what they say, either.

The Mormons understand better than the average Protestant.

The Catholics screwed up early Christianity, the Protestants replaced bugs with other bugs. Simplify to the point of falseness and absurdity.

Re:Who cares?

[Frank T. Lofaro Jr.]

Then when the proxies ban you, what next, botnets?

Then comes Federal prison. Federal PMITA Prison.

Re:Who cares?

[1u3hr]

It was not my intention to sneer at you.

How many times do you have to draw attention to a typo ("we;re [sic] going to hell regardless") before it counts as a sneer? Do you think I don't know the difference between a semicolon and an apostrophe?

Re:Who cares?

[1u3hr]

You don't have to mention a contract for a contract to exist.

And yet there has to be SOME evidence that both parties knowlngly entered into it.

Re:Who cares?

[1u3hr]

Then comes Federal prison. Federal PMITA Prison.

Right. And how many people have been sent to prison (in the US) for using a proxy to scrape a website?

Can you name one?

Re:Who cares?

[HikingStick]

I meant no offense. I simply do my best to follow the rules of writing that I was taught. I presented an exact quotation with "[sic]" to designate that I was presenting the quote exactly as it appeared, and that the non-standard usage was the original writer's and not my own. I guess you can take a man out of academics, but you can't take the academics out of the man.

Humblest apologies,
Andrew
a.k.a. HikingStick

Re:Who cares?

[1u3hr]

Dictionary of Modern English Usage by Henry Watson Fowler, P 532
"a neat & compendious form of sneer".

Re:Who cares?

[Peeteriz]

Well, the original point was exactly that it does not relate to the evil deed that is done, but to the sinful intent of one's mind.

The perception or the loss or fixing of the deed is irrelevant, if you have thought evil and acted evil, then the question is not about fixing the consequences, but about fixing and saving yourself, your soul.

Re:Who cares?

[meringuoid]

Wow. Lot o' parentheses there. You're not a Lisp programmer by any chance?

At any rate this was the part I never liked about Christianity. God demands blood for sin, but doesn't mind whose - the sinner himself is fine, a sacrificial animal is fine too, his own son is best of all? I mean, that's downright stone age. It's about as far removed from justice as you can get, and it makes God a bloodthirsty monster.

Re:Who cares?

[HikingStick]

It would seem that the commentary provided with the definition provides a literary point of view rather than simply the definition, though I will concede that some may intend it as such. I have always known the definition to be, as stated earlier, that the presented text was an accurate representation of the original. Both the Mirriam-Webster and American Heritage dictionaries make refernce to that meaning, without inclusion of any suggestion of a virtual sneer. From http://www.merriam-webster.com/dictionary/sic%5B3%5D: " intentionally so written --used after a printed word or passage to indicate that it is intended exactly as printed or to indicate that it exactly reproduces an original ".

Re:Who cares?

[Anonymous Coward]

We Christians understand that some crimes/evils/sins have worse effects than others, but we believe that all sin is equally damning. As a result, we do not protest different punishments for different crimes.

Re:Who cares?

[1u3hr]

I don't want to insult you, but I you must be a bit autistic if you can't see how you can offend people by making such statements, regardless of the literal meaning.

Whatever you intend, it comes off as snide, condescending and sneering. Save your [sic]s for academic treatises, or be prepared to deal with a lot of ill will.

Re:Who cares?

[The Living Fractal]

"Otherwise every "me too" on a subject you agreed with, or "fuck you" on one you didn't would be modded up."

I see your point. I just disagree that it would happen like that. If I get moderator points, which I do regularly, and I click on a story to find a post has +34 Insightful, then I am most likely not going to moderate it up further. As a good moderator I will go looking further into the posts for other insightful posts and moderate those.

What I am saying is the situation would be more dynamic than the "everyone sticks on one post like glue" that you seem to think would happen. Slashdotters are more inquisitive and intelligent than herd animals.

Re:Who cares?

[HikingStick]

I apologize for offending you with my use of "sic". No, I did not see it as something that could come across that way.

You are not the first to suggest a touch of autism--I had a doctor once who thought I might lean that way except for the fact that my language skills never regressed (it's not something for which they tested when I was a kid). I have one son in the autistic spectrum, however, and I can see much of myself in him. Even if I am a touch autistic, it would not excuse me from poor behavior. In the sense that my comment offended you or lacked sensitivity, then I bear fault and accept it.

Hey, anonymous! This is your boss.

[jollyreaper]

I told you to scrape Slashdot, not read it. Now get back to work!

one approach

[buddyglass]

1. Tell your boss it's a bad idea to break these websites' terms of service. He'll probably override you and tell you to do the project anyway.
2. Code up the project just like he asks. Demonstrate that it works.
3. Shortly afterwards, email the sites in question from a non-work friend's account and let them know (with specific information) the accounts and IP addresses that are violating their terms of service. Hopefully the accounts will be disabled, and/or your employer's IP range will be blocked.
4. Throw up your hands and tell your boss, "Well, I guess they figured out what we were doing!"

Re:one approach

[uneek]

1. Tell your boss it's a bad idea to break these websites' terms of service. He'll probably override you and tell you to do the project anyway.
2. Code up the project just like he asks. Demonstrate that it works.
3. Shortly afterwards, email the sites in question from a non-work friend's account and let them know (with specific information) the accounts and IP addresses that are violating their terms of service. Hopefully the accounts will be disabled, and/or your employer's IP range will be blocked.
4. Throw up your hands and tell your boss, "Well, I guess they figured out what we were doing!"

Thats pretty stupid.

It doesn't solve the ethical or technical problem.

It worsens his / her relationship with the boss.

Re:one approach

[gEvil (beta)]

The boss will just say "You're a smart guy. Find a way to get around their protections."

Re:one approach

[Jazz-Masta]

you forgot #5

5. Boss comes into your office and says you didn't program it correctly - that you should have incorporated some sort of IP spoofing techniques. He fires you for incompetence.

Re:one approach

[Rob Kaper]

And you do. And so does the tech guy at the other end. And so on.

End result: not just one, but at least two people in ICT with job security!

Re:one approach

[DingerX]

No need to rat them out. Just give the boss the 411 on the "Hidden Costs" of doing things that way. Ethical arguments are well and good, but when you're asked to do something like this, it's clear that the ethical arguments mean nothing compared to economic ones. Guaranteed system-wide outages and worse catastrophic failures (=poisoned data) are going to cost a lot. Since you cannot predict when they will happen (only that they will happen), or what they will look like, you can't give an estimate for the downtime while you develop counter-countermeasures; nor can you guarantee that those counter-countermeasures will succeed for more than a brief period (especially once they're "on to you").

The fact is, what he's proposing will be more expensive and disruptive than doing it the legit way. And it is your job to point that out. You can also then point out that while you personally consider it unethical, the industry-standard reaction does not usually involve a lawsuit, but rather to deploy simple countermeasures that disrupt and embarass the amateurs stupid enough to perpetrate it.

Re:one approach

[buddyglass]

That's one possibility. I had assumed a certain level of technical understanding on the boss's part. Honestly, if the boss is such that he's going to fire you for something like that, then you were already going to get fired for something. It's just a matter of "when" and "what".

Re:one approach

[buddyglass]

So repeat the process a couple of times. Eventually, as long as the boss doesn't come to view you as incompetent, he may come to view the "task" of getting around these other companies' blocking mechanisms so difficult that it's worth respecting their TOS and actually paying for the content. And, of course, while all this is going on, be looking for another job.

Re:one approach

[buddyglass]

It does solve the ethical problem in that you're no longer enabling your employer to violate these other companies' terms of service. At least, not for any longer than it takes for those companies to block the fake accounts your employer had you create.

It doesn't overly stress your relationship with the boss, unless that relationship is such that he can't stomach any feedback whatsoever. You did your "due diligence" and raised your objections to the task. Then you sucked it up and did exactly what he asked. Ostensibly, it isn't your fault that this other company actually decided to enforce their terms of service.

As for the technical problem...wait, what technical problem?

Re:one approach

[Vladus2000]

I wouldn't ever rat out a place I worked for unless the shit hit the fan and I was in court defending my ass. That will not look good on your resume if you get caught. While your odds of getting caught are low, they are not zero. Just let the site figure out what you are doing and block you. If you want to code it in such a way that they find it faster that is fine, but no future employer wants a whistle blower since just about every place I've worked for has broken a law or 100 (mostly small and harmless ones) and really don't want that kind of person working for them.

Re:one approach

[syousef]

Shortly afterwards, email the sites in question from a non-work friend's account and let them know (with specific information) the accounts and IP addresses that are violating their terms of service. Hopefully the accounts will be disabled, and/or your employer's IP range will be blocked.

If your contract includes a non-disclosure statement, and this is ever tracked back to you (and people do blab!) you're in deep doo doo. You've already said to go ahead and violate terms of service, now you're adding breaking the conditions of your employment contract to the mix??? Talk about falling on your sword for no good reason!

Re:one approach

[buddyglass]

The advice was contingent on someone being able to send email that couldn't be traced back to them by their employer. While email obviously isn't that anonymous, in order for the guy's employer to find out the company who's terms of service are being violated would have to rat him out to the company violating their terms of service. That seems pretty unlikely.

If you're truly paranoid:

1. Don a disguise: wear a hat, glasses, dye your hair, grow a beard, etc. (for the interior cameras).
2. Wear gloves (no fingerprints).
3. Walk to a library, restaurant, etc. with public terminals (walk = no license plate caught by parking lot cameras).
4. Install a SMTP client.
5. Send your email.

Re:one approach

[syousef]

You obviously missed my point (and deflected with an elaborate straw man accusation and ridicule that I'm paranoid) so let me put it more simply: If you have a moral dilemma that's in a legal grey area and you resolve that by outright breaking the law with no doubt about it being unethical, that makes you a bafoon, not a good guy. The idea is to keep your integrity in tact so you can continue to work with an untarnished reputation, and to sleep at night knowing you've done no wrong.

Re:one approach

[Anonymous Coward]

1. Same.
1a. List concerns in email form, take a backup.
2. Same.
2a. Wait long enough that enough people in your organisation have seen it, and that it works. Make it known to everyone that you are concerned about the ramifications of doing so.
3. Same.
4. Same.
5. Mention the presence of the email to anyone who asks, forward it on to anyone who is interested.
6. Boss can't pin it on you easily becuase it'll make him look bad if your email gets around.

Re:one approach

[Anonymous Coward]

Apologies, I messed up my other post. It was supposed to be:

I was going to post to say almost the exact same thing. I'd personally add a few steps:

1. Same.
1a. List concerns in email form, take a backup.
2. Same.
2a. Wait long enough that enough people in your organisation have seen it, and that it works. Make it known to everyone that you are concerned about the ramifications of doing so.
3. Same.
4. Same.
5. Mention the presence of the email to anyone who asks, forward it on to anyone who is interested.
6. Boss can't pin it on you easily because it'll make him look bad if your email gets around.

--

(EXTRA TEXT SO THAT I CAN ACTUALLY POST THIS MESSAGE)
(MORE TEXT TO GET AROUND THE FILTER)
(WHAT EXACTLY DO I NEED TO DO TO POST THIS MESSAGE?)

Posting to Slashdot
Filter to stop bots posting
Is making this hard

Re:one approach

[Anonymous Coward]

That's when you just screw around for a day, and get back to the Boss, saying that it doesn't look possible. But you could try it via a proxy, but to get the traffic you'd need, it'll cost. Do all the work, run it for a little bit, then let the target know exactly what you're doing again. Repeat. It'll rapidly become a huge time and money sink. Keep a paper trail of your concerns if your boss won't back off.

Re:one approach

[buddyglass]

I wasn't ridiculing you by using the word "paranoid". Paranoia is often completely warranted, as it might be in this case.

Now, to the rest. First of all, I don't necessarily equate "breaking the law" with "unethical". In this particular case, violating the terms of an NDA probably isn't be "breaking the law" per se, though it would give my employer excellent grounds to sue me. Depending on how specific the email is, it might not even violate the NDA. "Hey, you might want to examine the usage patterns of accounts originating from company.com's IP range" isn't exactly divulging trade secrets.

I wouldn't lose sleep over implementing this plan because, in effect, it prevents one party from acting unethically and taking advantage of another. In all honesty, I probably wouldn't implement it because no job is important enough to me to go through all that cloak and dagger crap. I'd probably immediately start looking for another job, and in the mean time go over my boss's head and point out his unethical demands to someone who might actually care. If that person sides with the boss, well then I quit.

Before the courts

[pete-wilko]

Not sure of your jurisdiction, but Ryanair (Irish low-fares carrier) is currently suing two companies that use 'scrapping'/crawling for travel aggregation comparison services. Details in the wikipedia page: http://en.wikipedia.org/wiki/Screen_scraping - see section on 'Legality'. Could be something to keep an eye on. The load balancing part however sounds dodgy as.

"we're safe on copyright issues"?

[Sockatume]

I don't think you are, actually. Harvesting somebody's data and presenting it as your own (which I'm sure is what you're planning to do), presumably expensive data like stock prices, research data, or whatever, is still plagiarism, and would probably not be looked upon well in court. They could argue, for example, that you're counterfeiting their "product". Or it could be seen as a form of industrial espionage, I guess. You're not terribly likely to get caught, mind you. At least you weren't until you posted this on Slashdot and told the internet "O HAI, if we, your competitor, mysteriously publishes terabytes of useful data in a week, we probably nicked it from your website."

Re:"we're safe on copyright issues"?

[bigsteve@dstc]

In addition, if he violates the TOS, this may actually invalidate any implied permission he has to download a literal copy of the web pages ... making him liable for breach of copyright.

He / his boss needs to talk to an IP lawyer!!!

Re:"we're safe on copyright issues"?

[Anonymous Coward]

I don't think you are, actually. Harvesting somebody's data and presenting it as your own (which I'm sure is what you're planning to do), presumably expensive data like stock prices, research data, or whatever, is still plagiarism, and would probably not be looked upon well in court. They could argue, for example, that you're counterfeiting their "product". Or it could be seen as a form of industrial espionage, I guess. You're not terribly likely to get caught, mind you. At least you weren't until you posted this on Slashdot and told the internet "O HAI, if we, your competitor, mysteriously publishes terabytes of useful data in a week, we probably nicked it from your website."

You may be thinking of the UK's data protection laws. These do not exist in the USA. If the submitter is in the USA, they are free to scrape websites for data.

The legal basis of this was established years ago by companies that wanted to produce telephone directories - using the same directory data as the phone company-issued directories. In the USA, facts such as directory information or pricing cannot be copyrighted. You are free to transcribe facts from any source.

I remember years ago being approached by someone from the UK asking if we could send them a copy of some data from our website - I told them to just copy it from the website, and they told me that the data protection laws in the UK prevented them from doing that. My reply to that was that if we wanted to make sure nobody copied our data, we wouldn't have put it on the web. I had to waste time making a copy of the data and sending it to them.

Re:"we're safe on copyright issues"?

[Anonymous Coward]

In addition, if he violates the TOS, this may actually invalidate any implied permission he has to download a literal copy of the web pages ... making him liable for breach of copyright.

He / his boss needs to talk to an IP lawyer!!!

Contract law 101 - If no money changes hands, there is no contract. Simply clicking "I agree" does not a contract make.

Some types of data are not subject to copyright - any copyright asserted on a list of poodle owners in your town, for example, cannot be enforced.

The worst the web site can do for violating the terms of service is block them from further access to their information. Unless we're talking about actual copyrighted stuff like works of fiction or art or some such - then copyright law comes into play and civil damages can be gone after.

Re:"we're safe on copyright issues"?

[Sockatume]

I wasn't thinking data protection, actually, which AFAIK only controls what a company can do with information which has been submitted to them, not what companies can do with other companies' published data (although by implication it forbids secondary use of the data without the explicit permission of those who provided the raw data in the first place). I was thinking about simple plagiarism of data, which is what they're aiming for. Data, particularly data which was generated, compiled, processed or assessed at great cost, is legally protected from plagiarism. For example medical trial data can be reproduced in meta-analyses etc. with proper attribution, but if I attempted to pass off said data as my own work the original journal publisher would sue me out of existence.

Re:"we're safe on copyright issues"?

[Anonymous Coward]

Wow. This is a perfect example of why NOT to trust internet "experts".

Money trading hands has nothing to do with the validity of a contract. EULAs have been held up in court. Besides, it could easily be argued that the TOS gives the person the right to read and use the website. If you assume that the TOS is not a contract allowing the person to use the site, then they don't have any right to it at all.

And while database type information has not been copyrightable in the past, in the US and a few other countries laws have been passed to allowed data aggregators to copyright their database products. They would have to be able to prove that it was their data that was taken and repackaged, but that would be trivial if they interjected a few bad points of data.

Re:"we're safe on copyright issues"?

[Dragoness Eclectic]

In what country? Not in the U.S.A. The Supreme Court ruled a while back that un-original arrangements of data are not copyrightable; the case in question was a white pages phonebook. The layout (fonts, decorations, etc) or other *original* elements of a compilation of data can be copyrighted, but not the data itself.

The U.S. Supreme Court very specifically struck down the argument that "sweat of the brow"/cost of assembling data entitles one to copyright protection. In the U.S., originality is required for copyright.

Re:"we're safe on copyright issues"?

[bigsteve@dstc]

Did I mention contract? I don't think so!

This situation is more akin to the way that an open source license grants permission to use copyrighted material subject to certain restrictions on your behaviour. If you don't meet the restrictions, your permission to use the material is cancelled.

CYA by Asking!

[cliffiecee]

The whole idea sounds pretty scummy, based on your description. Multiple free accounts? yeesh.

So why don't you just ask the webmasters of the sites you're about to scrape? I'd bet the site owners would settle for a few hundred per month to provide you with data in whatever form you require. And it's cheaper than the $2000/mo. for a server, etc. (If these sites are "bigger" than what a few hundred a month would buy, then you damn well better ask (see below).

Ask your Legal department about this as well. They can be extremely helpful in stopping hare-brained ideas like this. If the websites in question are big enough to take action against this, YOU'RE the one left holding the bag, not Mr. Bright Idea Guy.

WARNING: All of this assumes your boss is partially sane and reasonable!! If he's a jerk, you are hosed. I'm sorry.

Re:CYA by Asking!

[gEvil (beta)]

So why don't you just ask the webmasters of the sites you're about to scrape? I'd bet the site owners would settle for a few hundred per month to provide you with data in whatever form you require. And it's cheaper than the $2000/mo. for a server, etc. (If these sites are "bigger" than what a few hundred a month would buy, then you damn well better ask (see below).

This also assumes that the other site isn't a competitor. If they are, then merely suggesting the idea to them will probably screw your company over.

CYA

[audacity242]

Have a little chat with your in-house counsel. Not in the "OMG my boss is making me break the law!" but in the "Look, I want to make sure we protect ourselves" way. Better yet, do it via email so it's documented. Then your counsel will go have a little chat with your boss telling him what an idiot he is.

Time to leave

[OhHellWithIt]

If I were you, I'd try to dissuade my boss on the legal and ethical grounds, and then I'd start trying to find a new job. Violating ethics is one thing, but your firm's violating another firm's terms of service could cost your firm much more than they're saving, in legal damages and lost reputation. If your boss has higher-ups, he could be sacked, or he could try to shift the blame onto you, so you could be sacked. Some companies have a lawyer on staff, or on retainer, and this is the guy/gal who really ought to be pulled into the question.

Beyond the legal and ethical considerations, there is the pragmatic consideration that if you are scraping someone's site and the load is noticeable, the "host" (to use your parasitism analogy) is going to take technical measures to stop it. This means your stuff is going to break and be unreliable. That's not going to be good for the company's bottom line.

For yourself, besides the question of your reputation and future employment, you have also to consider that if your boss is willing to do this to others, he is probably willing to do something similar to you. It's a question of when.

Why have an ethics dilemma at all?

[man_ls]

You're getting paid to do a job, and you're not going to be personally liable should anything go wrong anyway. Why bother having an ethics dilemma? Are you really going to walk out of your job over violating the terms of service of a few web sites?

It's not your job to worry about the ethics of the situation, that's probably not even your boss's job -- it's somewhere in your corporate legal department, the Board, or an Ethics or HR department perhaps. But not the programming department. Unless you're a member of ACM, in which case you're encouraged to voluntarily abide by their code of professional ethics, just do what you're being paid to do and ask fewer questions.

Given that nobody will be injured by your actions (you can't use 'just following orders' to justify war crimes, etc.) there's no problem. And if they sue you, the boss and the company are going to be at fault, but not you personally.

Re:Why have an ethics dilemma at all?

[man_ls]

I noticed a bunch of replies below mine about how if you really have to make a fuss, you should phrase it positively about 'protecting the company' and that with your in-house counsel via e-mail. I agree with that suggestion. Ultimately, the decision whether it's acceptable or not is off your head, but you'll have raised your concerns.

Re:Why have an ethics dilemma at all?

[Daimanta]

"You're getting paid to do a job, and you're not going to be personally liable should anything go wrong anyway."

"I was just following orders!"

Where have we heard that golden oldie before?

Re:Why have an ethics dilemma at all?

[man_ls]

I think I addressed that point in my original post....

Re:Why have an ethics dilemma at all?

[ledow]

This is a shortsighted view of the problem here.

"You're getting paid to do a job, and you're not going to be personally liable should anything go wrong anyway."

Incorrect. His boss isn't breaking the Terms of Service, he is. When the website in question terminates their access, guess who's gonna get the flak? The person who *implemented* the system, not the one who designed/thought of it, especially if they are non-technical and rely on lower-order technical beings to do things for them.

Take, for example, a situation that I regularly come across:

Boss: "It's okay, we'll just copy all these Microsoft CD's and save a fortune on licensing."
Boss's Boss: "Okay. You know best."
Boss (to underling): "Copy these CD's"
Underling in theory: "Okay". Underling in practice: "We *can't* do that."

When things go wrong, the underling in theory is going to get the blame here, because it's his area of expertise and he *wrote* the system that does it. I get people suggest to me all the time that we could just install another license of Office that we don't own, or we can just copy CD's that have blatant copyright notices on them, or breach a Data Protection Act directive by doing X, or a million and one other things that I *know* we can't do. The people in charge of me barely understand the terms, let alone whether what they are doing is illegal. I have to sit and explain to my boss and my boss's boss why we can't do them. Trust me, if something got noticed, Underling in Theory would get sacked/sued every time.

"Are you really going to walk out of your job over violating the terms of service of a few web sites?"

Why not? I get asked to do all sorts of crap and I point it out and say no. If I *chose* to do it instead, then it's a different matter. But when I *refuse* to do something on legal or ethical grounds (we're not just talking ethics here - it also sounds like they have a "subscription" of some kind to the data that they are scraping, or that it's a competitors website) then if you *make* me, I will walk (been there, done that - I've turned down a good career move and more money in order to sleep at night - not that I was being asked to break the law, not that I was being asked to sell my children, but that I was being asked to do things that I didn't agree with [wasting money within a school on useless IT cruft and consultants while the kids didn't have books or paper]). I'll also report you to the BSA or whatever organisation I need to if you really press me, or the local press like I did in the above case (they didn't do anything with it, but I breathed a sigh of relief once I'd sent off the information to them - my part was done and I'd done good by myself - if the press decide to sit on something, that's on *their* conscience, not mine). You don't do illegal stuff if you're honest and your mortgage depends on a wage.

"It's not your job to worry about the ethics of the situation, that's probably not even your boss's job -- it's somewhere in your corporate legal department, the Board, or an Ethics or HR department perhaps."

Wrong. Because they won't even *know* what the problem is until it comes up in court and they have it explained to them in excruciating detail. However, someone who decides to do something that's part of their job, within their area of expertise and breaks a law (or even does something a bit stupid) that *they* should know about will get fired/sued by their own company once the shit hits the fan. So your boss *and* you might get sacked - you're still no better off and your employment reference is now a million times worse.

"just do what you're being paid to do and ask fewer questions."

It's sad that people think this is a good way to live. He's *being paid* to do his job. Which does not entail questioning his ethics or breaking Terms of Service (even if legally unenforceable) or anything else. His *job* is to stand up and say "Whoa, hold on, we can't do that". If he doesn't do that, he's not doing his job an

Re:Why have an ethics dilemma at all?

[GodKingAmit]

Because some human beings have something called a "conscience"

Re:Why have an ethics dilemma at all?

[Daimanta]

I'm sorry but this is Slashdot and am an advanced member. I do not read the articles nor the summary and I don't read the entire comment. I just post something and hope to get +5 Insightful.

Re:Why have an ethics dilemma at all?

[LionMage]

Incorrect. His boss isn't breaking the Terms of Service, he is. When the website in question terminates their access, guess who's gonna get the flak? The person who *implemented* the system, not the one who designed/thought of it, especially if they are non-technical and rely on lower-order technical beings to do things for them.

Well, hang on a second. What the GP was referring to as "liability" was pretty clearly legal liability. And in that regard, sorry, the company is liable, not the poor schmuck that the boss ordered to write the scraping application. That's one of the reasons that corporations exist -- to limit liability and shield constituent members of the corporation.

You're trying to conflate a discussion of legal liability with a discussion of "what happens when the blame rolls downhill." And the latter case is why the employee always covers his ass and is sure to inform his superiors of what consequences, if any, might ensue because of implementing a request of management.

Yes, there is a way to hold individuals liable inside a corporation -- something my lawyer friend explained to me as "piercing the corporate veil." In a case like that, it's absolutely essential to make sure that the employee covers his ass by documenting any consequences and making sure that management has been informed, then retaining that information for his own CYA file. If the employee does this right, when litigation starts, the blame will go right where it belongs: on the manager who instigated the unethical project in the first place.

If the company still, as you suggest, fires the employee as a scapegoat, there's every chance they were planning to sacrifice him all along. In that case, his days were numbered at the company anyway -- it's just a question of whether he raises ethical objections and refuses a project, and is therefore fired immediately, or whether he does the project and is eventually canned in a corporate face-saving gesture once the malfeasance is exposed. In an economy like this, I'd rather have the guarantee of income in the immediate future while starting to look for other opportunities before my current job is terminated.

If you even need to ask....

[jimicus]

If you even need to ask, you've already demonstrated a trace of ethics.

Now, sometimes having such ethics will mean you have to make difficult choices. And nobody else can make those choices for you.

While ethics won't pay the mortgage, "Reason for leaving the previous job: I was asked to do something illegal and, when I queried this, was given the ultimatum to do it or get out. I got out." is probably a heck of a lot better than "The company had to sack me after it transpired I'd done something illegal" (emails to CYA notwithstanding).

Because, make no mistake, the fact that your company has done this will get out.

The SlashDot Solution: The Day the Net stood still

[starglider29a]

We'll help you. Just send us the link to the pages yer scraping, and we'll SlashDot them out of virtual existence. Then you can:

a) Demonstrate what will eventually happen when their scheme is found out
b) "Suggest" to your boss that you need a raise, because this isn't "extortion", just "power brokering". I'm sure he'd understand.
c) Tell him "Klaatu, berada nikto, muckerfother!"
d) Remind him that you only wield this power for good

Re:can't pay hosting?!

[Anonymous Coward]

I'd have to guess they simply haven't figured out how to distribute the labor across many free workers yet. I believe it's called "crowdsourcing".

Shit Falls Downhill

[whisper_jeff]

Shit falls downhill. This mantra cannot be overstated. You must always remember that, at the end of the day, the shit will _always_ fall downhill. If a bad/illegal system is implemented, it will not be the fault of the person who requested said system - it will be the fault of who implemented the system.

How should one deal with this situation? It depends. Do you feel sufficiently confident in your job that you can say "no" without being fired? Does being fired scare you (will you be able to make ends meet and put food on the table without the job)? Do you think, especially in this collapsing economy, that you could find a new job if you lost this one? Is this even a job you want to continue with? We don't know the answers to these questions - only you do. But, regardless of the answers, you must remember that shit falls downhill. If things are going to go bad, you will get the blame. If you can accept that then go for it. If you can't, then make other plans.

Re:Shit Falls Downhill

[SatanicPuppy]

The problem is that this is a "damned if you do, damned if you don't" situation.

If you build the scraper and the content provider successfully sues you (wildly unlikely in my experience), then it's your ass.

On the other hand, if the content provider notices your scraping and cuts you off (extremely probable), then your app is worthless, and, again, it's your ass.

Building a business on other peoples data is just a bad idea. If you can't license it from them, and you can't collect it yourself, you're at their non-existent mercy, and scraping is extremely delicate and very easy to thwart.

Re:Shit Falls Downhill

[whisper_jeff]

...scraping is extremely delicate and very easy to thwart.

Then that's what he needs to tell his boss (and I agree with you). He needs to clearly inform his boss that it is probably illegal and opens his company up to (expensive) litigation and, more importantly, even if it doesn't get to litigation, the source site could make a change that renders their scraping efforts null and void. It needs to be put in a dollars-and-cents picture so that the boss realizes that the best (and only) solution is to pay the licensing fee. Doing otherwise will likely be more expensive and inconvenient. Any other depiction of the scenario won't matter to a boss that is only concerned with the bottom line.

And, if that doesn't work, polish your resume.

How does this affect your reputation again?

[Anonymous Coward]

Sure it will, if you tell people exactly what you did.

Resumes are for listing your skillsets, not whether you agree with the ethics of your employers.

pro vs. con Liability issue

[192939495969798999]

Pro: If the stuff hits the fan, the boss could say that he didn't know you did that and that it's your fault. Make sure you have email records printed at home that say that 1) you don't recommend this because it violates the terms of service, and 2) he says he doesn't care and to do it anyways and that he will be responsible, not you.

If you have that, then at least you can pass the blame to the truly responsible party. And no, I don't think it's wrong to do something illegal where the only damage is financial as long as you are absolved of the responsibility for your actions.

Con: Still, I would seriously consider looking for another job, because while the reality is that this type of illegal action is very common... things have a way of going very, very sour for the developer job-wise if and when they finally get caught.

Somewhat Similar

[xoundmind]

I was once asked to decompile a piece of commercial/non-FOSS software in my workplace. I reminded my employer that it was very likely illegal to do so. As a result, I was viewed as someone who was not a "can do" employee. Not suprisingly, I found work elsewhere.

Re:Somewhat Similar

[MightyYar]

I think that doesn't violate copyright law unless you fail to do a "clean room" implementation. The person who examines the original code can write a specification for someone else to code to. This is how Compaq cloned the IBM BIOS.

hmm.

[apodyopsis]

If I was running the website that you were scraping and I saw a repetitive bandwidth spike to one location - yes these people do check - the you can bet an investigation, change or block would be forth coming and then your Boss would have no data at all.

The same applies to "load balancing" which is obviously stupid, short sighted and plain wrong.

My first option would be education of the (PH?) Boss with emphasis on what could go wrong, how easily and why.

Management

[bogidu]

As a general rule, management is an unethical occupation. You have been asked to do something that you know is wrong and have been put in the situation where if it's brought to the light of day and makes the company look bad, you will be used as a scapegoat.

In this situation, even pointing it out to you "boss" would make you a "non-team player".

In your shoes, I would drag my feet on it and find new work asap.

Source: Prior experience with one of these "corporations".

If he's that cheap

[jerryodom]

He can't be paying you that well. Chances are you can get paid more elsewhere. Time to walk.

Hahaha hahahaha hahaha!

[EWAdams]

"Compliance officer" in an IT business... you crack me up. You should take your show on the road.

Hospitals have compliance officers because a) they're regulated, inspected, etc. and b) people can die and they can be sued to Kingdom Come.

The IT business is about as regulated as Somalia.

Re:Hahaha hahahaha hahaha!

[GodKingAmit]

When I worked at $LARGE_TECH_COMPANY we had a compliance office with an anonymous 800 number.

Re:Hahaha hahahaha hahaha!

[socsoc]

I doubt submitter works in a large tech company.

I do not have an 800 number at my place...

Re:can't pay hosting?!

[Fred_A]

If the employer can't afford hosting what are they doing to pay your salary?

Oh that's apparently somehow covered by the yearly Charity Employee Voluntary Organ Donating Session where you can volunteer marrow, a kidney or a lung. They didn't give all the details though.

Is that really worth saving 2K a month?

[Anonymous Coward]

I don't know how much you're getting paid. But, given the time it would take to create and maintain such a system, when you could be doing other things for the company, I kind of doubt it's going to save them much money anyway.

Oh, one more thing...

[east coast]

I know of someone who got caught up in something like this and here's the case for it... http://calorielab.com/news/2006/07/24/more-cowbell-diet-web-site-the-daily-plate-caught-borrowing-calorie-data/ I do not know what the final outcome was for all of this unfortunately.

Negotiate based on risk / maintenance cost

[uneek]

There are two items here of possible ethics violations.

1) Scraping the web sites.

2) Using free accounts instead of paying money for space.

Scraping the websites has two problems:

a) It crosses the ethics line. It is in fact stealing content.

b) Technically it is a very short term solution because you are relying on an interface that you don't control. Are you going to change your code every time they change the web pages?

Consider in detail the maintenance cost of this. It might be cheaper to ask the web sites to provide some kind of a feed for you and you only build the interface once.

2) Using free accounts does not sound like such an ethical problem but it is still a short term technical solution. These accounts do not exist for your corporate solution, they are not guaranteed, and their interfaces can change at any time. What is the cost for maintenance of this? What is the risk cost for this ?

Sure you need to look out for your job. However you should be able to negotiate providing evidence that the maintenance and risk costs to your firm outweigh the benefit of the current solution.

If you can't do that because you don't know how, then learn.

If you can't do that because your boss won't listen, then do the project and get the hell out of there.

Your resume does not have to be all inclusive.

Uneek

Talk to Legal.

[rjh]

Your boss is asking you to commit a felony. You probably do not want to do this.

Send your boss this email. Print off two copies: put one someplace safe, and then walk down to Legal with the other.

Bob:

After reviewing what you have asked me to do, I have come to the conclusion that it is possibly a violation of federal law. By violating the terms of service, we will be accessing computers in excess of our authority, which is a felony under the United States Computer Fraud and Abuse Act, and could expose the company to substantial financial risk.

For obvious reasons, I am not comfortable proceeding forward without getting Legal in on this. If they say it's fine, then I will certainly implement it.

... This will almost certainly torpedo your relationship with your boss, but if he's asking you to violate the CFAA then you don't have much of a relationship with him anyway. Your company's legal counsel will jump all over this and make it clear to your boss that (a) he's not allowed to violate federal computer fraud laws in order to make a buck, and (b) he's not allowed to exact retribution against you due to various whistleblower-protection statutes.

Copyright?

[cfulmer]

What makes you think you're safe on copyright issues? You didn't mention what you're scraping, but if it's somebody else's expression, your company could have a very big problem.

Depending on a number of factors, Terms of Service are legally binding, so there are going to be legal problems for you there.

How big of a company are you? Do you have a general counsel? If you do, you might want to raise your concerns with him/her. Are you a public company?

When I have been involved in hiring decisions, a big question is whether we can trust the candidate. So, candidates that talk about how bad their last employer was naturally raise a question about what they'll say if they leave us. BUT, I would actually be more inclined to hire somebody who, when faced with an ethically questionable activity, went through the right channels to resolve the problem and eventually decided to leave rather than compromise his integrity. You can still say "Overall, it was a great place to work, but I was asked to do something severely unethical and I chose to resign instead."

Look at it as an opportunity

[russotto]

To treat those TOSs as the useless virtual toilet paper they are. While the particular terms of services you'd be breaking might not be so unreasonable, the whole idea that someone can impose an obligation on you for downloading a web page is offensive.

Your application sounds pretty shady on its own, but just tell your boss that and do it anyway. When the web sites you're scraping start feeding you false data (or just 501 errors) and making your company look stupid, you can say you warned them... especially if you did so in writing or at least e-mail.

how does a weasel share?

[mbeckman]

I once worked for a company that did the same parasitic hosting thing. Eventually they sold to a bigger company for millions. But the founder cheated his employees out if their stock options. He had revealed his true character early on, but we foolishly believed that it would not be turned against us. We toiled endless uncompensated hours writing the much more complex code required to do what we called "weasel sharing". Your boss is an unethical crook. Get out now before he has a chance to cheat you too, if he hasn't already.

Free instead of $2000 a month?

[Dekortage]

The other is, for lack of better words, a "load balancing" part that requires using multiple free accounts instead of purchasing space and CPU time for less than $2,000 USD per month....

What the heck? How many free hosting accounts were you planning on managing, that you could replace $2000/month worth of hosting? Even maxing out a fully dedicated server comes out to less than a third of that, and I really doubt you would need all that firepower. Some companies cost more than this, some cost less, but either way... $2000 seems like too much.

Copyright

[Anonymous Coward]

Part requires scraping most of the site, parsing the data and presenting it as our own without human intervention. While we're safe on copyright issues . . .

I'm not sure you're safe on copyright issues. Is the information you'll gather in the public domain or somehow not subject to copyrights? IANAL but, I suspect if you parse and transform the format, it's still infringement.

You mention that you'll have to violate the sites' TOS. Just a few months ago, the federal government brought criminal charges against Lori Drew for violating Myspace's TOS. slashdot.org

Does your company have a legal department? If so, raise your concerns diplomatically in writing and recommend to your boss that he get the lawyers to sign off first. This might make the whole problem go away.

Amusing Ethics

[Anonymous Coward]

Interesting that Slashdotters who by and large rage against DRM are on the side of not scraping website content due to ethical considerations. Doesn't anyone see the irony?

Re:Amusing Ethics

[shentino]

DRM hampers rights you already have. I.e., denial of access and inconvenience with something you already own.

This is different.

When You Approach Your Boss

[Anonymous Coward]

Don't emphasize the legal/ethical aspects of this issue. S/he won't be listening, and won't care. Emphasize the practical aspects of the issue - the site cutting you off, the possible problems of parasitic computing re the owners of those computers not being under your direct control. If that doesn't convince him/her, then discuss the legal problems.

Comment removed

[account_deleted]

Comment removed based on user account deletion

One step away from way outside the line...

[bigbluemontana]

One of the questions we ask when we hire at my company is: "Tell me about a time when you face a value or ethical conflict at work. What did you do and what was the outcome". Now, you don't know the outcome yet, but frankly, I would applaud your efforts (at a minimum) for recognizing that this isn't right. Whether or not it's illegal isn't the question. Someone once said "it's not illegal until you get caught". Well, hang on for the ride. I can see your boss asking you to acquire counterfeit copies of software next. He's only a heartbeat away from that request. In summary - run, and run quickly. The economy may be in the toilet, but there are always jobs out there for people with high standards and ethics.

time for a "little bird told me"

[circletimessquare]

tell the site you are scraping, anonymously, that someone is scraping their site. it is plausibly anonymous, since someone out on the wild could notice. your boss will get an angry email and have your ip blocked from further scraping. then tell the hosting company, anonymously, that multiple free accounts are being used. this might not work out as well, since this is info a random person could not get. but you can hint the hoster in anonymous ways that allows them to discover the deceit

Don't do it

[Anonymous Coward]

My company's market cap is in the $6B range. They still balk at paying $349 for a license. Instead, to finish a project I had to take my own license to install the software with the understanding that they would purchase licenses later. Needless to say, they never did.

They won't pay the $49/year per server for a piece of encryption software.

They won't pay $25/instance donation for the 10 or so instances of an OS in use.

So I no longer bring in my licensed copies. I no longer use my license keys to download updates. I claim ignorance when the software breaks and there's no support.

Re:Don't do it

[shentino]

Or you get fired by your boss and almost literally kicked out the door by security.

I agree though, since his hesitance has almost certainly already doomed him.

Mild Flamebait

[SatanicPuppy]

I've always worked at places that were victimized by scrapers, rather than the other way around. In the early days, I'd track 'em down (where possible), and try to extract some measure of satisfaction by confronting the miscreants with their misdeeds.

In my experience, most people don't even think it's wrong; in their minds it's the same as hotlinking an image. It's not their problem if the people on the other end don't protect their data. And anyway, if we didn't want the data stolen, we shouldn't have posted it on teh interwebs in the first place.

So I'm a bit amused at the sudden vehemence of the Slashdotters who commonly decry all DRM and all attempts by copyright holders to protect their IP. I would have thought the community would have come down on the other side of this issue, but I guess music and games are different from websites, photos, and other scrapable data.

Re:Mild Flamebait

[Anonymous Coward]

So I'm a bit amused at the sudden vehemence of the Slashdotters who commonly decry all DRM and all attempts by copyright holders to protect their IP. I would have thought the community would have come down on the other side of this issue, but I guess music and games are different from websites, photos, and other scrapable data.

Actually, it's because you've clumped different opinions (that you apparently don't understand, probably because you decided before reading them that you disagreed) together into a group that doesn't actually represent any real body of people. No wonder you're amused.

Re:Mild Flamebait

[Anonymous Coward]

...I'm a bit amused at the sudden vehemence of the Slashdotters who commonly decry all DRM and all attempts by copyright holders to protect their IP. I would have thought the community would have come down on the other side of this issue, but I guess music and games are different from websites, photos, and other scrapable data.

I can't speak for the rest of /., but for me, these two issues are nowhere near the same. I'm against DRM because I feel that it's right that once I buy (for example) a DVD, it's mine. I am entitled to copy it for backups, rip it to watch on my computer, or do whatever I please, so long as it is for personal use. I do think piracy is wrong and do not advocate it in any way, shape, or form.

If I paid someone to write a bunch of content for my site and then lost the password to the server, I wouldn't think twice about scraping the content back to myself.

Re:Mild Flamebait

[Tjebbe]

yes, DRM is not copyright enforcement, DRM is copyright evasion (the producing party circumvents the copyright law in order to be more restrictive than the law entitles him to)

Re:Mild Flamebait

[plague3106]

I'm suprised too. The scrapping isn't the problem in my mind. It's on the internet, and TOS are pretty much unenforcable for simply viewing the page.

Now, creating an account, there you seem to be entering a contract. The distributed computer scheme might be an "unauthorized use of computer system," and could very well enter into legally troubling territory.

Re:Mild Flamebait

[Anonymous Coward]

So I'm a bit amused at the sudden vehemence of the Slashdotters who commonly decry all DRM and all attempts by copyright holders to protect their IP. I would have thought the community would have come down on the other side of this issue, but I guess music and games are different from websites, photos, and other scrapable data.

In my case, it's not the IP that I'm concerned about. I also don't think that type of scraping should be illegal, but it's definitely as unethical as hot-linking. Not because of the IP, but because of the large-scale bandwidth usage, which the other site has to pay for.

Re:Mild Flamebait

[fuzzyfuzzyfungus]

Beyond the usual "This just in, slashdot actually a group of individuals, not a hive mind" answer, I think there is another significant distinction that you are missing.

DRM is something that you put on my system for your benefit. Things like firewalls, password setups, and systems that feed garbage to scrapers are things that you put on your system for your benefit. As you have probably noticed, Slashdot has a userbase that is far more concerned than average with defending systems from attack(both their own, and in many cases, systems they are paid to administer). DRM is an attack on my system. Antiscraper setups are a defense on your system.

DRM and antiscraper techniques are about as similar as you setting up cameras in your store to catch shoplifters and you setting up cameras in my home to catch shoplifters.

Re:Mild Flamebait

[element-o.p.]

You make me wish I hadn't already posted, just so I could mod you up. Well said!

No dilemma

[LordActon]

There's absolutely no conflict here, moral, legal, or ethical. Keep your job, keep your head up, and learn from it.

The data are published on the WWW. Since when do ethics preclude using anything besides a browser to access them? Yes, there are technical drawbacks and technical defenses; I'm not sure your boss's idea is a great long-term strategy. But if bad system design were unethical, Microsoft would sell a lot less VB.

The TOS are one company's attempt to control how you use their information. Are they enforceable? Do they actually care, if you don't republish? And why is that your concern? That's why your firm has management and lawyers.

You're not poisoning milk here; you're acquiring data for commercial purposes. Get to work, and let the legal beagles sort it out.

Re:No dilemma

[ledow]

Copyright. The copyright holder has the right to do pretty much what they want with their own data. If that means putting up a notice that says "free to view, pay if you want to download", they can do that. Copy an image from a website and then upload it somewhere else, or put it in an advert, or print it out and stick it on your office wall. Chances are you just broke copyright law. You can't do this with anything copyrighted, no matter how easy it is to "technically" do it. Dilbert cartoons, youtube vids, Slashdot comments, it doesn't matter. If it's copyrighted, you *can't* do this.

Re:No dilemma

[LordActon]

Nope. *Facts* cannot be copyrighted. That's why data vendors require a licensing agreement limiting what you can do with them.

Besides, I didn't say anything about republication. The OP thought there was some issue with his means of retrieval, whether he fetched the data with FF and hit "save page", or wrote a program to save himself some time.

And by the way, viewing *is* downloading. Whether or not copyright holders have the right to control private use of published information is, at best, unsettled law and at variance with precedent in dead-tree publishing. "pretty much anything" doesn't remotely describe the copyright-holder's control.

Informative? Copyright is a boogey man here. Your comment should never have been modded up.

Re:No dilemma

[ledow]

*Facts* cannot be copyrighted. However, someone else's representation of those facts *can*. You can find out yourself and republish that the weather was cold yesterday, but you can't save the BBC's weather page and put it on your own. Screenscraping is highly dubious at best, as it is using an established source and format for that data. TV listings company regularly sue over people screenscraping their (copyrighted) listings of facts. Assuming the data itself isn't copyrighted (unlikely by the sound of it but you never know), you can't necessarily use any source you like to obtain those facts without opening yourself up to accusations of copyright infringement. If it's prices of certain tickets, fares, products, services, then you are into a massive grey area but it's still not clear that it's legal either, so companies, especially rival companies, or ones planning to use that information as part of their business model, should not be doing it. Their lawyers would throw a fit if they knew. Just off the top of my head, I can name five *recent* cases where this sort of thing has come up and it's gone to court with accusations of copyright infringement, etc. - several online plane ticket companies, several online concert/theatre ticket companies, the aforementioned TV listings, etc. The ones that didn't make it to court either changed their entire website to wipe out that sort of functionality of blacklisted access from the companies in question.

Republication is just an example, however, copyright applies to lots of action that you think it may not. Technically, even printing an entire downloaded webpage could be classed as infringement. It's quite possible that a court will see "viewing" and "saving" data differently, especially if it's for non-personal use. Don't forget though, even having a picture in your web cache can variously be classed as "downloaded", "a conscious action" and even "creation" of that image. It's a point that you mention yourself. And it is a horribly icky legal quagmire, but that's all the more reason to steer clear. Anyone with brains would take this to the company's legal representative, who would reliably tell you that it's an interesting case and they'll investigate for $X thousand / X amount of hours and would recommend that you don't do anything without checking with them first. Why? Because they're not stupid enough to say it's okay to do it.

Re:No dilemma

[LordActon]

The Internet is still new, legally speaking, and you're citing examples of companies pressing their claims. When photocopiers came out, many publishers claimed they were illegal. Digital Audio Tape eventually carried a levy to "compensate" the recording industry. And in 1984 Sony took the VCR to the Supreme Court.

So, yes, there's a long history of the publishing industry pressing unreasonable claims. And losing.

Please, let's not confuse cockeyed claims with settled law. Screenscraping is not legal dubious at all, much less "highly". There's no case anywhere deciding that fat-fingering the data into your file is OK while cutting-and-pasting is not. It's all a question of what is stored and, more important, what is (re)published.

If I took every idiotic court decision as a cautionary tale, I'd have to disconnect my computer from the WWW. If I let my company's lawyers make every decision, nothing would ever get done. We live in a world where we have act, every day, in the presence of uncertainty. The most prudent choice isn't to avoid all risk, but to assess it reasonably.

You agreed with my basic point: facts can't be copyrighted. So would your lawyer and so would their lawyer. My advice to the OP, then, is "go ahead and collect your facts as you see fit". If they scream, explain you have a right to the data and can't get it any other way. If they cut you off, OK. If they threaten to sue -- because surely they'll threaten before they do; they way you to stop, not take you to court to prove a point -- then you find another way. It's not a dilemma of any kind. It's just work.

How about "no logic"

[Anonymous Coward]

"direct mail has a cost, therefore spam should not be allowed because it's abusive"

That was your point, paraphrased.

Care to try something that makes sense now?

No, I predict you'll say something like "that was not my point" despite the fact that it's there to be read, so denying it is useless.

And then of course you'll attack me because that's what you do to people who point out you're wrong.

Tell the boss it won't work, give reasons why.

[mark-t]

Tell him that the very next-to-best case scenario for him (the "best case" scenario being that they never notice what you are doing) is that they notice what you are doing and blacklist you from connecting to it ever again. If at all possible, give him an estimate on the likelihood of that occurring. Point out to him very plainly that if or when this outcome occurs, then what he is asking you to do now will be all for nothing. If the chance of legal ramifications is not negligible, you should also mention that as well. Document everything. If he still wants you to proceed, then polish your resume and find another job because if he's too cheap to pay 2k a month for a service he thinks he can scam off of for free, he's probably too cheap to want to continue to pay you in a few months time, after he figures he's got what he needs from you.

Nothing illegal or immoral

[axus]

Using multiple free accounts? Automatically reading a webpage? Sounds like something I wouldn't lose sleep over if I were doing it for my personal use. There's nothing inherently immoral about breaking a ToS, in fact some of the things they put in those are immoral. It's not illegal to break a ToS either, but it can get you banned from the website. It's really just... cheezy. The main problem here is that these are temporary solutions. Fine for a one-time operation, but you can't rely on them past a month. If it's something your company wants forever, they'll have to pay you to keep doing it/updating it, and deal with the downtime everytime the website changes or the free service closes your accounts.

not to be harsh

[Bizzeh]

not to be too harsh, but this is your work place, you are getting paid to do as you are asked. get off your high horse and just do it.

Re:not to be harsh

[Nabeel_co]

If you were working as an undertaker, and your manager tells you to bury a live person, would you? I would certainly hope not.

Leading question

[bWareiWare.co.uk]

You asked the question in a leading manor and have got odd responses as a result:

'Scrapping' pages is exactly what the Internet archive or Goggle do, this is common and generally accepted practice (look at the amount spend on SEO). It is also assumed that these operate without human supervision and do not need to read or compile with the human TOS of your site. Critically spiders should compile with the 'robots.txt'. If you do this you have the moral high ground. If you don't then it can be interoperated as criminal under the laws such as the Computer Misuse Act.

Similarly no one suggests that everyone using gMail is a parasite. Most 'free' services come with a very explicit contract detailing their allowed uses. If you compile with the contract you are fine, if not, you are again breaking the law.

Probably more importantly, this is almost certainly a bad business discussion:

Given that you as an employee have judged it as ethically questionable you can be fairly sure a significant proportion of your clients are likely to feel similarly.

Even if you are complying with the contract from your free service you are almost certainly not getting a SLA in return. If the supplier decides your business is dodgy, or you are putting too much burden on their system they will shut down all of your accounts without warning or reprieve. Constantly battling this is likely to cost you more then the hosting in the long run.

Page scrapping is very unreliable. Even when the source site is cooperating they invariable break it on every edit. What will happen to your business when the source site detects your scrapping and decides to serve goatse to your spider, and hence your clients?

Re:Leading question

[StrawberryFrog]

'Scrapping' pages is exactly what the Internet archive or Goggle do, this is common

Not always - not on sites that require you to log in, and possibly to pay, before you can see the content.
Also, estate agents may put their houses for sale data freely on the internet, but they are virulently opposed to apps that scrape multiple estate agents' websites, compare and aggregate the data, and present it on another website.

Most 'free' services come with a very explicit contract detailing their allowed uses.

yes, and many of them have a paid version that allows you to do much higher volumes of transactions. Aggregating several free accounts was certainly not what they wanted you to do.

What will happen to your business when the source site detects your scrapping and decides to serve goatse to your spider, and hence your clients?

I'd classify that as "sweet revenge" :)

Re:Leading question

[grub]

'Scrapping' pages is exactly what the Internet archive or Goggle do, this is common and generally accepted practice

Most people want their pages to be found, hence the reason they tend to not mind google and other search engine scanning the site. The submitter's dilemma is different, the boss wants it done covertly and unethically.

Re:Leading question

[clone53421]

Not always - not on sites that require you to log in, and possibly to pay, before you can see the content.

Hypothetical. TFA said nothing about any such paid content, in fact TFA said there weren't copyright concerns.

Also, estate agents may put their houses for sale data freely on the internet, but they are virulently opposed to apps that scrape multiple estate agents' websites, compare and aggregate the data, and present it on another website.

That doesn't make it illegal or even unethical to do so. Their only real recourse is to detect your bot and block it or feed it garbage.

yes, and many of them have a paid version that allows you to do much higher volumes of transactions. Aggregating several free accounts was certainly not what they wanted you to do.

How do they prevent it, though? Even if they can get you on breach of contract, all they could do is terminate the hosting agreement.

Ethics or just disgust with a cheap-ass boss?

[swb]

IMHO what your boss wants is maybe not polite or nice but isn't really unethical. It is cheap, though.

I do share your disdain, though, as I have a boss who is equally cheap. Not for resale products we get on a discount for evaluation/testing purposes end up being used to support our internal production network, generally making them unavailable for what they were intended, getting us up to speed on the real thing.

I extract my revenge, though, by bugging the product reps for an additional unit on loan for testing by telling him (on the QT) that my boss is using ours for our internal production network. Sometimes I get the extra loaner in sympathy, sometimes I don't, a couple of times the product rep panicked the boss by telling him that the rep's loaner is in use and we needed to bring our own to a key sales demo (which was impossible without killing our network).

One guy mercilessly badgered the boss over the location of the NFR unit and it was hard not to laugh as the boss stammered and lied badly, since it was hard to explain why it wasn't available when the only people who should be using it were in the same room.

Maybe the story submitter needs to screw the bosses plan by emailing the site maintainers and explaining he doesn't like doing what he's doing and see if the site maintainers will submarine the bosses plan by canceling accounts.

The only thing to remember is that if you lose your job today, good luck finding another for a long, long time.

Re:Ethics or just disgust with a cheap-ass boss?

[DaveV1.0]

Violating an agreement is not unethical how?

Re:Ethics or just disgust with a cheap-ass boss?

[swb]

Most TOS are so overly broad and restrictive that you almost can't use the site without violating it, and in general it didn't sound if the submitter's violation of the TOS was anything more than technical, and did not result in real harm.

Re:Ethics or just disgust with a cheap-ass boss?

[DaveV1.0]

Prove your conjecture that "Most TOS are so overly broad and restrictive that you almost can't use the site without violating it". Show examples of same.

If the TOS says, "No screen scraping" and one screen scrapes, then one is violating one's agreement to follow the TOS.

If one agrees to do not something (or do something) and then one does (not) do said thing, one is breaking the agreement one is unethical. If one entered into the agreement intending never to live up to the agreement, one is unethical.

Re:Ethics or just disgust with a cheap-ass boss?

[Creepy Crawler]

Its cause Im text scraping. A bot doesnt need no screens!

Re:Ethics or just disgust with a cheap-ass boss?

[clone53421]

If the TOS states that the site serves banner advertisements using Google's AdSense and it prohibits blocking those advertisements, most of /.'s users would unapologetically violate the TOS with every visit.

Re:Ethics or just disgust with a cheap-ass boss?

[DaveV1.0]

And, your point is what? I know that many, if not most, /. users are not ethical. All one has to do is read the comments to see that.

Re:Ethics or just disgust with a cheap-ass boss?

[clone53421]

"Most TOS are so overly broad and restrictive that you almost can't use the site without violating it"

I personally feel that "This site uses XYZ to inject ads into the delivered pages, and by using this site, you agree not to block, hide, or otherwise interfere with the delivery of the XYZ advertisements" falls under that category. But hey; maybe you disagree. I don't think blocking advertisements is unethical.

Re:Ethics or just disgust with a cheap-ass boss?

[Fulcrum of Evil]

Regardless, your ethical duty is to not violate a reasonable TOS. Yes, that's fuzzy, but if you're going to get legalistic on ethics when you don't have an actual enforced code to begin with, then you're already acting unethical. This is about not being a sleaze.

You know better than that.

[argent]

Illegal? Running a screen-scraper isn't illegal.

Neither is waving a gun around... on a film set, or in your own home. Try it in a busy street or place of business and see what happens.

This isn't just "running a screen scraper", and you know it.

not legal, ethical

[mevets]

Law is very simple in comparison.

Re:not legal, ethical

[khallow]

Simple, eh? As I see it, ethics is the theory of deciding how much harm you're willing (or perhaps allowed) to do to someone else depending on context. Law is the application of ethics to the real world and as usual with applications, is more complex than the theory. One particular way law is more complicated is that a legal system has to have mechanisms for modifying and enforcing laws. An ethical system doesn't require either mechanism. If it is self-consistent and well-defined, then no empirical "what if" scenario will break the system of ethics, meaning no change is required. And enforcement implies that the ethical system requires you to impose (some degree of coercion) your system of ethics on others. Again this is not required for a system of ethics. It need not have scope outside of yourself.

Further, given the organic way legal systems evolve, this produces a very complex system. It need not be consistent (eg, you can be breaking the law no matter what you do) or understandable (you need not understand whether you are breaking the law, requiring consultation with experts (lawyers) to CYA). While for ethics, the fundamental problem is deciding which of two or more possible actions is the better course.

To give the current example, the actions are clearly wrong. If you grant that it is right for one can parasite a website in this way, then you need to explain why it is right for an extremely large number of parties to do so, large enough to effectively shut down the original provider of the content. And the original website shouldn't have any responsibility for the information propagated secondhand by these interlopers.

But is it legal? We don't know. Violating the terms of service might be in itself illegal or might be interpretable as a denial of service attack which is commonly illegal. And the material might well be copyrighted despite claims to the contrary. If the original website learns of the copying and does nothing, it might legalize what would otherwise be an illegal act (as I understand it, you can lose the right to certain forms of intellectual property if you chose not to enforce them, I think copyrights are one of those). It's even remotely possible that the original website might become liable if some harm comes of the secondhand information and the original website has not attempted legal action against the parasitic website (say the original website distributes medical information and the parasitic website has fatally out of date information, why hasn't the original website done something about this interloper over the past five years?).

Re:not legal, ethical

[mevets]

Law is about what is allowed; ethics is about what is right. The best that the worst can claim is that "I didn't break the law".

Re:not legal, ethical

[khallow]

Law is about what is allowed; ethics is about what is right. The best that the worst can claim is that "I didn't break the law".

So what? This trivial (how does one decide what is allowed? it requires deciding first what should be right) distinction doesn't mean ethics is more complicated than law.

You're wrong

[Anonymous Coward]

If they don't provide an API for their data, they're asking for it. It's as simple as that.

Terrible engineering

[Have Blue]

Even if you don't want to tangle with the ethical issues, ask your boss how he feels about the app constantly going down and losing data because the "parasited" service deleted all your free accounts.

Absolved? (Re:pro vs. con Liability issue)

[HikingStick]

One being absolved in the eyes of the court does not mean one is truly absolved of wrong-doing. It just means that the court determined that the action(s) did not violate the societal code of law. It does not mean that one did not violate moral law.

Re:I caught my boss having sex with a poodle.

[I cant believe its n]

And it is only monday....

This does not sound like a boss with a career path

[ouder]

There is an obvious problem when a boss asks you to do something unethical, but there is something else to consider. One good way to rise in a business is to work for someone who is "going places" and willing to take people with him or her. At very least, it is good to work for someone likely to get promoted if there is a chance to move into their position. Frankly this bozo or bozette does not sound like someone who is going to climb the corporate ladder. At best this is a short-term scam, and the boss can only look good for a short time before the wheels start coming off this project. At that point the boss and everyone associated with the project will be tarnished. If the boss owns the business and is running this type of scam, then there is not much future in this business.

The '90s want their bogus argument back.

[argent]

Should I be against spam for any other reason than I am annoyed by them?

Your argument might have carried some weight back in the mid '90s when spam was new and "a lot of spam" meant 30 or 40 messages a week.

The damage caused by spam is not mere annoyance.

The direct cost of spam was already outrageous by the end of the century. At one point in the '90s I was being charged $750 a month for excess traffic, just from the overhead of *rejected* spam. i had to drop whole *countries* at my router to get that under control. It's gotten to the point where spam is the overwhelming majority of email out there: less than half of one percent of the email connections to my server are for legitimate mail. That is, for every legitimate message, over 200 spams have to be handled, one way or another.

And even the SECONDARY cost of spam... things like incorrectly bounced and dropped mail... has long since passed 'unacceptable' and is now accelerating full tilt for the edge of the economic universe.

Ten years ago "I don't think spam should be any more illegal than billboards, flyers, or direct mailings." was a bad analogy, mister "Bad Analogy Guy". Now... you're either trolling or improbably naive.

Go higher up.

[Nabeel_co]

Maybe a solution to this problem is to go higher up and see if another person in your department at a higher lever thinks it's O.K. to proceed. If so, then you know that if you don't change jobs, you'll be in for a lot of resistance. Personally, there is no way in hell I would do that for my boss. It's just plain wrong. But it's always good to see where everyone else stands in the company, just so you are not shocked at the ramifications if you decline, and try to go higher up for support on your position.

Re:Go higher up.

[Nabeel_co]

To add: You could also transfer to another department within the same company to avoid it.

Re:Redirecting content

[Mr. Droopy Drawers]

Reminds me of a time when an Ebay'er was pointing to images on my website for an automotive auction. Didn't ask us or give us credit for the images. So, his example of "recently restored examples" became a photo of a '63 Imperial being loaded into a crusher.

How's that for Crushing the Competition?!

Safe?

[DaveV1.0]

While we're safe on copyright issues

Please explain how you are safe. If you scrape other people's website, then, generally, other people own the copyright to said information. Unless the website has nothing but public domain information, you will be violating other people's copy rights. Even on /., "Comments are owned by the Poster" so if you scrape this comment, you would be violating MY copyright.

Re:Safe?

[JoeMerchant]

Thinking to the future here - either things posted on the web are going to move into public domain, or the entire world will creak to a halt due to copyright litigation.

Apparently Google gets away with "fair use" while profiting from serving up search results on my content, both in my own pages and in my forum posts. I don't think any search spider is smart enough to read a TOS, but they certainly have no problem finding non-linked directories on my site that have explicit robots.txt telling them to go away.

Re:Safe?

[DaveV1.0]

No. That is a false dilemma. Postings on the web do not have to move into the public domain and there is no way you can justify that statement.

As an example, one can make it a requirement that one assign the copyright of a post to the website where the post will be seen and that people can make copies for the expressed purpose of view the comment.

Re:Safe?

[clone53421]

Unless the website has nothing but public domain information, you will be violating other people's copy rights.

I'd actually amend that to read "Unless you use nothing but public domain information...". Parts of the website which are copyrighted (or trademarked for that matter) can't be copied; parts which are public domain may be (and I suppose most websites with public domain information will also have some portion that is copyrighted or trademarked, if nothing more than their logo). And as you correctly pointed out, /. posts are not public domain, so we can assume his project isn't going to be scraping /. threads... he did assure us that they were safe on copyright issues.

Re:Safe?

[JoeMerchant]

If we, as a society, wish to spend (waste?) a significant portion of our lives arguing about copyright, copyleft, fair use, and lawsuit, then, yes, everything online can be protected.

For a practical thought exercise, what do you think will happen if you create a mock Sarah Palin strip-tease video (with appropriate political satire thrown in for fair use of her image and name) and post it anywhere online, regardless of copyright statement?

How is this different from any other content that you invest time and money in creating and then post online?

When the effort of defending the copyright exceeds the effort of creating the original material by a significant margin, a practical person would walk away.

Re:Safe?

[shentino]

Funny thing though...

If copyright haters know that, they could effectively DDoS the copyright holder into croaking.

It's called "Mashup"

[mrboyd]

And it's soooo web 3.0.. go for it. :)
Burn karma, burn..

Software Engineer Code of Ethics

[Anonymous Coward]

I would try to convince your boss what he is asking is wrong and that you must do it a different, legitimate way.

I'm not sure if you are a software engineer or not but we do have a code of ethics.

If you see an ethical violation it is your duty to tell the company you work for they are or are about to make an ethical violation. If you do get a bad reputation for this hopefully the new company you are going to is ethical and you can point out that the code of ethics required you to do what you did.

I don't know if this comforts you any but you could point out that the code of ethics prohibits a supervisor from punishing someone for pointing out an ethical violation.

Here is a link to the code of ethics. http://www.acm.org/about/se-code

Re:Software Engineer Code of Ethics

[justinlee37]

Psh, who appointed those guys as the philosophical guide to all those who code? Saying "we" have a code of ethics like you're some holier-than-thou order of knights is a little fantastical, don't you think? Coders are just people, like anyone else, and sometimes people lack ethics.

Sure fire way to NOT get hired anywhere else...

[freedom_india]

That is a SURE-FIRE way to NOT get hired next time.
Look, you are an employee of a corporate: protected by its immunity.
You are an agent of the corporation: The Principal (Corp) bears the full responsibility for its orders to you.
If those orders are illegal, and you declare you obey them without knowing they are illegal then the corp is responsible.
Most probably when the FBI or MPAA raids the corp offices you won't be charged: you can't be.
The max the feds can do is to offer you protection for testifying against your boss.

Re:Sure fire way to NOT get hired anywhere else...

[rjh]

You are an employee of a corporate: protected by its immunity.

Employees have no immunity from felony prosecution. Bang, period, end of sentence. There is no immunity anywhere in corporate life. There is indemnity, which is a separate issue, which protects you from civil lawsuit; it does not protect you in any way from felony charges.

If you commit a felony and the police come knocking, expect to get charged. The corporation won't be.

And if you really think that protecting your company from a lawsuit and reporting possibly felonious actions to the company's legal department will get you not hired anywhere else, you really need to spend more time in corporate America. This is the way you handle these things. You don't involve the police and you don't go to the press. You go to the corporation's own internal hierarchy and say "my manager is doing something aggressively stupid which, if discovered, will get me in a ton of trouble and expose the company to massive financial risk. Please make him stop."

What do you think Legal exists for? They're there to protect the company -- from internal threats as well as external ones.

Re:Sure fire way to NOT get hired anywhere else...

[Anonymous Coward]

Parent is 100% correct. Grandparent is magnificently retarded.

Re:Sure fire way to NOT get hired anywhere else...

[freedom_india]

You are talking about the Perfect world scenario.
Am talking about Real World.
How many whistle blowers have been hired by other corporates and been promoted to CEOs?
Legal exists to intimidate outsiders, make sure to defend the company against lawsuits: not to prevent it.
Yes there are anonymous ways to contact Legal in banks, etc., but tell me how many have done so and come out on top: in America?
Lets face the real world: Tipping off Legal or cops is considered betrayal and makes you untouchable. Laws that prevent you from being fired from THAT job exist: but somehow the spotlight shines in wrong way, so you end up at home.
Committing a felony being an agent of corporation forces the corporation to defend you in a lawsuit. Betraying it does not.

Re:Sure fire way to NOT get hired anywhere else...

[rjh]

This is not a 'perfect world' scenario. This is very much a real world scenario.

In the real world, if you commit a felony, you will get hung out to dry in a heartbeat while the company circles the wagons to try and protect the unindicted. Look at, e.g., the HP spying scandal, where people all the way up to the Board of Director level were left hanging. HP wasn't going to say they authorized the identity theft, or that they approved of it in any way.

How many whistle blowers have been hired by other corporates and been promoted to CEOs?

Right, because being a whistleblower means you're qualified to be a CEO. It doesn't. But generally, yes, whistleblowers do get protection, as long as they go within official channels and make sure to dot their is and cross their ts.

A few years ago I blew the whistle on some grossly unethical practices at my workplace. I showed up to a senior executive's office with my whistleblower ombudsman in tow and presented the issues. Didn't harm my career one bit. I left that job a few months ago and received glowing recommendations from my former bosses -- including the ones I blew the whistle on.

Y'see, their boss was the guy I reported to... and he made it very clear to his underlings that I was going to get the best recommendation letter they'd ever written.

You clearly don't know much about corporate politics or how to navigate these situations. I would respectfully suggest that while Dilbert is a great resource, it is also a rather one sided one, and you really need to get some firsthand experience.

Re:Sure fire way to NOT get hired anywhere else...

[Anonymous Coward]

You are talking about the Perfect world scenario.

No, he's not. He's talking about the law, and he's entirely correct.

Am talking about Real World.

No, you're not. You're talking about the law, and you're entirely wrong. You claimed committing a felony as an "agent of the corporation" makes you immune to prosecution. It absolutely does not. It doesn't even make for a viable defense. The company doesn't even have to pay for your lawyer. You are entirely on your own.

How many whistle blowers have been hired by other corporates and been promoted to CEOs?

None need to be "promoted to CEOs" to disprove your cartoonish claim. You're moving the goalposts.

Legal exists to intimidate outsiders, make sure to defend the company against lawsuits: not to prevent it.

True...in John Grisham novels. In almost all cases in the real world, however, the company would much rather avoid a lawsuit than defend against one, even successfully. It's cheaper.

Yes there are anonymous ways to contact Legal in banks, etc., but tell me how many have done so and come out on top: in America?

Quite a few, as this is a large part of the reason why companies have legal departments to begin with. I have personally seen it happen.

Lets face the real world: Tipping off Legal or cops is considered betrayal and makes you untouchable. Laws that prevent you from being fired from THAT job exist: but somehow the spotlight shines in wrong way, so you end up at home.

That's not the real world. That is a childish, simplistic fantasy that you're trying to pass off as world-weary cynicism. The real world is far more complicated.

Committing a felony being an agent of corporation forces the corporation to defend you in a lawsuit.

Not even remotely true. Try to cite an American law stating this. You can't and won't.

I really want to cut you some slack on your ignorance since it's obvious you're not an American, but you just keep on acting like you're an expert when it's clear that you're just taking television too seriously. I'd give it fifty-fifty odds at most that you've ever even been here.

Re:Sure fire way to NOT get hired anywhere else...

[freedom_india]

Do you have statistics that prove that whistleblowers DO get promoted over other similar qualified individuals in SAME company?
Am not saying am immune to prosecution if i commit a felony: am saying that if i do an act that turns out to be illegal when found so in court and i was acting in best interests (not ignorance, even am not that stupid), the company is bound to defend me since i acted as its agent. Am talking about protection, not immunity.
Much like Enron shielded its traders even though some were proven to have discussed hanging out california dry.
Whistleblower protections have been diluted a lot in pst 8 years: just ask Bush.
Or take medical insurance claims, where valid claims are rejected by employees representing Aetna: Do the individual employees get sued?? NO.
If the employees complain to Feds about the illegal practice, what would Aetna do? Suppose i do get Aetna convicted, get a book deal, spend the money away in LA and now come back to work, tell me which companies would rush to hire me? (am not talking about Greenpeace or EFF), even when am well qualified.
Screw corporates, take Valerie Plame: she did nothing wrong, was exposed, sued Cheney, lost, and did the CIA rush to hire her in a desk job? Apart from a $2.5m book deal, she was not hired to work with any other company, let alone mercenary Blackwater.
Again i repeat, law is different from real world implications:
You may win every court battle, yet you lose the war: court battles, press, publicity, book deals etc. Yet Erin Brockovich still has only speech appearences and not a job.

Re:Sure fire way to NOT get hired anywhere else...

[Anonymous Coward]

You are grotesquely stupid.

Do you have statistics that prove that whistleblowers DO get promoted over other similar qualified individuals in SAME company?

Again, they don't need to get promoted to disprove your claim. And by the way, YOU are the one who owes some statistics, because YOU are the one making the assertive claim. YOU have the burden of proof here.

Am not saying am immune to prosecution if i commit a felony: am saying that if i do an act that turns out to be illegal when found so in court and i was acting in best interests (not ignorance, even am not that stupid), the company is bound to defend me since i acted as its agent. Am talking about protection, not immunity.

Liar. Read your OP again. You specifically used the word "immunity". You specifically said you can't be charged. You were wrong. And you are ALSO wrong about the company being obligated to defend you. They are not. Period. They MIGHT choose to, if they think it's in their best interest, but that's NOT a given.

Much like Enron shielded its traders even though some were proven to have discussed hanging out california dry. Whistleblower protections have been diluted a lot in pst 8 years: just ask Bush.

Completely irrelevant, as neither support your claims.

Or take medical insurance claims, where valid claims are rejected by employees representing Aetna: Do the individual employees get sued?? NO.

That's a civil tort, you fucking imbecile. Denying a claim isn't a felony. No, not even a valid one.

If the employees complain to Feds about the illegal practice, what would Aetna do? Suppose i do get Aetna convicted, get a book deal, spend the money away in LA and now come back to work, tell me which companies would rush to hire me? (am not talking about Greenpeace or EFF), even when am well qualified.

That doesn't even remotely resemble the situation being discussed. You're taking a situation with an employee dealing with a potential legal problem quietly and internally (using channels set up by the company explicitly for that purpose) and comparing it to the biggest, most dramatic whistleblower scenario you could imagine. This is both stupid and dishonest.

Screw corporates, take Valerie Plame: she did nothing wrong, was exposed, sued Cheney, lost, and did the CIA rush to hire her in a desk job? Apart from a $2.5m book deal, she was not hired to work with any other company, let alone mercenary Blackwater.

Tell me what companies Plame has applied to work at, what positions she applied for, why she was rejected, and most importantly why Valerie Plame is in any way representative of the person you replied to in your OP, let alone all of corporate America. Back it all up with real evidence. If you don't, you're admitting that you're talking out of your ass.

Again i repeat, law is different from real world implications:

And you're wrong about both of them.

You may win every court battle, yet you lose the war: court battles, press, publicity, book deals etc. Yet Erin Brockovich still has only speech appearences and not a job.

Yes, she does. God damn, you're a moron.

A difference in degree AND a difference in kind

[argent]

No, I do not "get it", if "getting it" means singling out a one form of an accepted practice simply because it differs in scope to its relatives.

Why not? You have two similar, but not entirely analogous actions. One is vastly more damaging than the other. Why SHOULDN'T they be treated differently?

Differences between spam and direct mail:

Direct mail: pays the Post Office more than it costs to mail, and subsidizes first class mail.

Spam: pays a fraction of its costs, and often pays nothing, and is in such quantity that the majority of the cost of running a mail server is dealing with spam.

Direct mail: is limited by economics. The costs of a direct mailing, including materials, postage, and mailing lists, is upwards of 50c per address... and often several dollars per target.

Spam: has no economic limits, since the cost is negligible... and if it's sent by a botnet that cost is born by secondary victims.

Not analogous at all, when you compare them. No reason to treat them the same.

I was in a situation just like this

[viridari]

My employer asked me to do something that was unethical, and likely illegal. I asked to hold off on implementation until we could consult company counsel on the legality of it. Boss and director said "No. Do it. Now." I made my case, said I'd be happy to keep working there, or not, but I'm not going to do what they're asking me to do in this case.

The next day I got my walking papers. I felt more liberated than upset.

I've now worked for two scumbag marketing companies and I'm thinking it's probably best, if you have a conscience, to avoid them like the plague.

The law cares about intent.

[Chris Snook]

Just because someone doesn't put up a mechanism to prevent this, doesn't mean this use of their system is authorized. If they have a policy, either in written TOS form or something automated like robots.txt, and you violate it, this could be considered unauthorized access under the law, and you could personally be held criminally responsible.

Re:The law cares about intent.

[JoeMerchant]

Far more likely that they will:

1. Increase their PITA factor to make it harder for you to gain this kind of access (captcha, graphic presentation of results, etc.)
2. Directly block your IP range
3. Send friendly notice requesting you stop
4. Send legal notice demanding you stop

Pushing criminal prosecution would be more costly and less effective than the above. Unless you happen to be pissing off lawyers, or the government, or worst of all government lawyers.

Re:The law cares about intent.

[Chris Snook]

You're probably right, but the potential criminality is a key detail if you want to keep your job. Even in most at-will employment situations, while they can fire you for refusing to do something distasteful, they can't legally fire you for refusing to commit a crime. If they do fire you, you get to sue them, which they know is very expensive, so hopefully they'll be smart enough not to try. Of course, this varies by jurisdiction and IANAL.

Copyright safety?

[ari_j]

What is it about what you are doing that makes you think you are safe with respect to copyright issues?

no solution required.

[nimbius]

if i were the "website" you're scraping i find it hard to believe it would go unnoticed.
I'll warn you once or twice about it, then over the next weekend
create something nice in my OSS webserver that replaces your
scraped content with pro-taliban rhetoric and dancing goatseman.

I'll then forward all of your frantic phonecalls to my FOSS astycrapper.

Phone calls (Re:Spammer logic.)

[HikingStick]

I noticed you left out phone calls. Perhaps that's because you realize there are legal limits to sending unsolicited information. In the case of billboards, mass mailings, and fliers, the seller/marketer bears the cost of the communication. A person does not need to accept a flier that is being handed out. People can opt out of most mass mailings, or dump the unopened envelopes when they arrive. Billboards, while often obnoxious, don't require that I deal with them (I need not look), and most cities and states are regulating the number and size of billboards already. Spam can make a user's email inbox unusable. I know two company execs who have stopped pulling email messages on their BlackBerry devices because the volume of spam they receive makes it impractical to use the devices for email. Unlike those other forms, spam can have a documented negative impact on consumers, and on the carriers (ISPs and mail servers).

You likely don't list phone calls because legal precedent is clear that consumers don't need to accept them. Unlike the other forms of mass communication, telephone calls are considered an imposition on consumer rights and privacy, so consumers may opt out (proactively with the "do not call" list). With cell phones, it goes further, since incoming calls can cost the consumer money, and Federal law already protects cell phone numbers from unwanted cell-call-spam (this does not protect cell users from solicitation calls from vendors with which they already have a customer relationship, or in cases where the consumer has willingly waived the right not to receive those calls). Email still costs some consumers money (those on dial-up and those with limited plans, few though they may be), and the volume involved also creates a quantifiable burden on consumers. [Care to wade through the 300+ spam--not subscription messages or vendor updates--that I receive a day across my various accounts?]. Email spam also creates additional burdens for the carriers and companies that host web servers. Unlike the postal service model, where the bulk mailers pay to have their messages delivered--including all intermediary post offices, spammers send their crap across countless servers that are overburdened due to the flood. These devices require additional hardware resources, often additional support, and additional software expense to try to eliminate that crap that is coming across the wire. Then, remember that the majority of spam is sent from compromised bot machines, which are illegally accessed and used to send the crap... The "ordinary person" principle would apply here. An ordinary person would see a problem with that.

The fact that you don't makes me wonder if you are (have been) a spammer, or if you provide(d) support services for that economic machine. If not, I apologize for what is perhaps the greatest of insults (suggesting that you are a spammer). If so, you just prove yourself to be outside of the mainstream of society.

Ethics and business ethics

[golodh]

First off I agree with your ethical misgivings. What you describe would be clearly unethical if practised by ordinary people. But not, unfortunately, when practised by ordinary people in the course of a business venture.

I think you need to realize that there is a big difference between business ethics, ethics in e.g. politics, and ethics between ordinary people.

In business there is no such thing as "ethics" as a self-standing concept. Except perhaps in a company's Code of Conduct, which is only ever meant as a piece of PR material. Everything is relative to the long-term and short-term business prospects. The time period being dependent on the length of tenure of the manager who takes the decision. If it's a course of action likely to generate a net benefit within that time-frame then it's "not unethical", otherwise it's probably "unethical".

Any action that's not outright illegal *and at the same time* very much open to detection, is acceptable from a business ethics point of view, and therefore "not unethical". Things that are illegal and likely to be found out are "unethical" (provided the damage they do is not exceeded by the revenues they bring). But only because they're likely to harm the company under the above-mentioned timeframe, and for no other reason. I just thought I'd make that clear.

So I'm sorry to say that my advice is: cooperate fully with your boss and keep him happy.

You may however wish to engage in a little CYA practise, just to guard against the possibility that if this little caper is found out and embarrasses your company, your boss may or may not be replaced but *you* probably will be blamed for carrying out something that you ought to have known is unethical. You are the one with the specialist knowledge after all, right?

If anything, your boss might even claim ignorance as a defense and blame *you*, the specialist, for not alerting him.

So yes, send him a polite and low-key email explaining (briefly) why you believe that this move might be less than beneficial to the company and suggesting that he consider an alternative (if possible give him one). That's all. Having sent it, get busy with the screen-scraping.

Do NOT in any way or form mention the word "ethics", "ethical", or even worse "legal". If you do, you'll create an immediate conflict (because you imply that (a) you know better than he (b) that you're telling him how to do his job and (c) that he's acting unethically or even illegally and (d) if you produce it later, your *then* boss will see that it's less than helpful) because your boss will know instantly that you are preparing to shift the blame on him, and only engaged in a CYA exercise.

Oh, and by the way, you will need to be able to prove you sent it, so (depending on how much power your boss has on email retention) make an unobtrusive backup of your out-box on a CD a week or so after you've sent it and hide the CD.

Come to my office.

[Anonymous Coward]

Brian, come to my office. We need to talk.

Do you like your job?

[nurb432]

Then shut up and do as your are told.

If you don't, then refuse and get fired for insubordination.

Pretty simple really.

I had a similar situation

[Errtu76]

My former boss asked me to make some data public that was in a user's private folder, after the user quit the company. In the contract that the user signed it was mentioned that no such data could be accessed without the user's permission.

Write an email to your boss, expressing your disagreement with his request and that you're willing to do what he asks you, but under protest. Then if your boss replies that he agrees with that, you can do what you've been asked to do without you being responsible.

Disclaimer: i got this advice from a lawyer, but check with the law in your country before taking this advice.

Re:I had a similar situation

[MarkvW]

That advice is problematic.

If the scraping is criminal, then no amount of boss-permission will avoid exposure to criminal liability. Actually, it tends to indicate that the person committed the criminal act, while knowing it was criminal--why else would he have asked for permission.

If the scraping is only a civil wrong, then boss-permission (if provable), ought to insure that your company will indemnify you for any civil liability that you incur. On the other hand, the victim of the scraping can sue both you and your company for the act. In other words, boss-permission does not necessarily avoid a legal wrangle.

Polish that resume'

[JoeMerchant]

You obviously have different ethics from your boss. I'd start (quietly, very quietly) looking for another job NOW, while diligently working on the project you have been assigned to.

If it looks possible that your project will go live before you have another job, I would, as tactfully as possible, e-mail your boss with a clear statement of your concerns, and bcc your home account for permanent record. If he wants to proceed, you're just doing a job, I don't think it is incumbent upon programmers to start a crusade every time they see something that they think might be unethical (even though many programmers do this anyway.)

And, to reiterate, NOW is the time to find that next step in your career path. Try for better salary, benefits, etc., but if you find a straight lateral move, I'd be tempted to take it rather than continue working for a slimeball. What I see is that he values your time and effort less than the $2000/mo account - and, as other posters have said above, you're likely to be re-coding your page scraping very very often - which, when it happens, is going to make you look bad, i.e. boss says: "oh, he can't code anything that works for more than a few weeks...", which he either a) is a big enough moron that he actually believes himself (most likely), or b) he simply is giving you a rat-trap job to make you look bad.

My first boss after school hired me, then a few months later started to feel threatened by me and a couple of the other guys he brought in (yes, he really was that out of date and apparently unable to keep his skills up.) He started a campaign on me, giving me project after project and pulling me off before I could finish. After he had a pattern of 4 "unsuccessful" projects I had worked on, he pulled me in for a firing meeting. 5 years later I had his job, a few months after that, he retired. That was a unique situation where I had direct access to his boss, who recognized more or less what was going on. In a bigger place, I probably would have just moved on.

Re:Redirecting content

[Free the Cowards]

Somebody once pointed at a picture of a frosted birthday cake on my web site from a forum. So I grabbed my image editor and built a special edition of the cake just for him, where the frosting read "Don't link to my images!"

I also have a specially crafted JPEG which is under 1000 bytes but which produces a 20,000x20,000 pixel image filled with black. It will totally screw up the layout of any page linking to it if they haven't entered an explicit size for the tag.

Do it, if he's willing to "help you out"

[JavaRob]

Your employer doesn't have the right to ask you to place yourself in legal jeopardy in this way, and if the sh1t hits the fan do you really think that someone that came up with this scheme will balk at placing all the blame on you.

Absolutely. That's why you should agree to do the work, but because of the increased risk to yourself, you should ask for a "little something extra" under the table, just between you and him. A wad of hundred dollar bills passed discretely in a handshake, for example. "I help you, boss, you help me?" is a good phrase to clue him in on the situation and what's required for the project to continue. ...or perhaps he may rethink how he wants his workplace to operate?

Re:Do it, if he's willing to "help you out"

[element-o.p.]

IANAL, but that seems to open up a whole new can of worms that, IMHO, you really don't want to get involved with. YMMV, of course.

Re:Do it, if he's willing to "help you out"

[Anonymous Coward]

So...
Instead of just sliding a bit along the slippery slope, jump all the way down?

Re:Do it, if he's willing to "help you out"

[JavaRob]

The suggestion is, er, not completely serious.

Though who knows... it depends on the boss, but here's a closely-related approach that *could* make the point, depending on the boss, etc:

"So listen, boss -- if I implement this the way you suggested, I'd have to violate legal contracts with these various companies, which puts me personally at risk for legal trouble at the very least, if not losing my job and possibly ruining my reputation in the industry -- and my current salary doesn't cover that kind of risk. Which brings us into the world of under-the-table bribes and bonuses, and I'm quite sure none of us want to go *there*... so do you have a plan B?"

Re:Do it, if he's willing to "help you out"

[JavaRob]

Seemed the logical next step. There's a whole marketplace out there looking for unethical developers, of course -- might as well get started out right!

Re:Do it, if he's willing to "help you out"

[element-o.p.]

"Which brings us into the world of under-the-table bribes and bonuses, and I'm quite sure none of us want to go *there*... so do you have a plan B?"

Sweet :)

Yes. Absolutely. And then look for a job.

[Anonymous Coward]

I worked for a similar company, that continually operated in the "dark gray" area of black and white. I often noted the situation, proposed a high-road workaround, and sometimes they agreed, sometimes they didn't.

Eventually I got fired.

Its been two years out of work in a VERY shitty economy. Its wiped out my personal savings, my marriage is having a tough time, and my confidence & health is in the shitter. I'm a good worker, and good at what I do. I'm not perfect, and don't pretend to be. I'm probably just like most everyone else here.

If I had it to do over again, I'd smile and say, "Yes, sir!"

If this happens enough to bother you, then look for another job on the sly, but don't stick your neck out.

Moral/ethical issues, employment and unemployment

[Rastl]

While researching unemployment benefits (unrelated to my job, thank goodness) I found that many states have a very small set of circumstances that let you quit your job yet collect unemployment. Being told to do something that is illegal/immoral/unethical is actually one of them in the state I was researching.

Since this is kind of a grey area here is how I would look into my options.

1. Research and find out if your state is one of the ones that allows this kind of thing. If so, proceed.
2. Get something in writing as to what you're being asked/told to do. There has to be some kind of requirements documentation for this.
3. Get the TOS from the site(s) you're being asked/told to scrape, showing that scraping is not allowed.
4. Contact your local unemployment office and find out if this is enough to let you quit and collect benefits.

With these things in hand you have a decent chance of being able to keep to your morals, not take a big financial hit while looking for new work with more ethical employers, and possibly flag this place as somewhere that doesn't play well with others. I suggest this course because simply saying "Quit and work for someone better" sounds great but is not very practical in reality where you like to sleep indoors and eat cooked food.

You'll find that we've all had to suck it up and do something we know isn't the correct course of action but generally that doesn't include something ethically wrong. Mostly it's a matter of knowing that what the company wants to do isn't going to work but also knowing there's nothing we can do about it.

why all the CYA's?

[Tjebbe]

(note: IANAL, and the following comment only goes if such things are illegal. if not; just refuse because violating TOS is asking to be kicked out, which will reflect back on your customer service. anyhew:)

Why are most people focused on the CYA part, and not even remotely thinking of simply refusing to do it?

Note that I am not against covering your own behind here, you can do that even when you refuse, in case things escalate.

But the fact of the matter is that in the end, you are willfully taking part in illegal ventures, and if you can't be prosecuted for that just because your boss told you to, the law needs to be fixed. If he can fire you for refusing such an assignment, the law needs to be fixed too.

Re:why all the CYA's?

[JoeMerchant]

Calling this illegal is tantamount to calling the boss an idiot. I've never known bosses (idiots or not) to take kindly to being called out on their lack of mental prowess.

OTOH - he is the boss, and you're not, so, to navigate the larger political structure and avoid being trashed by him, CYA is a first step, albeit a lame one.

Far better is to get recognized by his boss as being more valuable than he is - some organizations make this inherently difficult.

Never more apt: garbage in, garbage out

[SmallFurryCreature]

Subject says it all

Re:Moral/ethical issues, employment and unemployme

[JoeMerchant]

If unemployment benefits come anywhere close to what you're making as salary, you need to get a new job immediately. If you've got sufficient "computer skills" to post on /., you should be making more than double what unemployment will pay you (at least, this is true in the great state of Florida, I believe it is similar across the US).

Kelly Blue Book

[Anonymous Coward]

I started working with Kelly Blue Book in California, and was asked to strip copyrighted information for use as their own. The second day on the job, I began inquiring about how legal this was, and I was told at the end of the day not to come back to work.

Sure, it was only two days worth of work, but I'd rather not be involved in something unethical or even quasi-legal when it comes to business. Some scars can be hard to hide.

Waste of money

[vw_bob]

My take on this is that, though your assignment has spawned an ethical question, the reality of the situation is economic.

Your boss believes that it will cost him less to "scrape" data from the website and use multiple free accounts than to simply pay for the data access. This may be true at first, but, ultimately, this is false.

On the off chance you've not scraped websites before, I'll tell you that this is extremely error prone. So, while this may work initially, you'll be constantly chasing down bugs in the process.

Based on your description I assume you'd be automatically logging into their systems before scraping the data. What if their login process changes? What if they restructure their website? What if they add a captcha to the registration and login process?

My point is, what your boss wants to do is, to use Steve Job's recent phrase, a bag of hurt.

I'll bet that given enough time, the cost to your company in terms of your salary to build and maintain this application will be greater than the cost to actually pay for the data and create a dependable connection.

Don't forget to factor in what it costs you when your users see bad data or error messages due to the process breaking.

It'll cost less to do it the right way. So forget the ethics of it and educate your boss on the economics of it.

the right thing to do is

[Anonymous Coward]

Email the boss, and BCC that email to the company you plan on leeching. That way--you've clearly placed the blame on him, and you get a shiny new boss in the end.

From the other side of the fence

[Dues]

As someone who works for a web company that is full of high-demand listing information, we are constantly having to deal with this kind of activity. Any site with dynamic data that is in demand will already have a procedure to deal with the kind of activity you are driving to it, as they have almost certainly dealt with it before. In our case, we have a network appliance that can detect if you are a scraper based on your traffic, and then serve you a "you've been throttled" page. In short, your script will be an exercise in futility. Here is a better solution for your company - contact the source of the data, and offer to purchase a feed from them. Chances our they will be willing to comply to keep you from loading their website with stupid traffic. You will end up having to pay money, but at least your process won't break. You should suggest this to your boss at least. A side note: your peers will definitely frown on your actions. If I knew a person behind this kind of activity they'd get branded scarlet-letter-style.

Straw man.

[argent]

There's nothing inherent in spam that says, "Hey, I'm accessing a computer illegally!"

Even before botnets, spam was unacceptable.

The damage caused by spam was already clear a decade ago.

The fact that there may be spammers who are not using botnets is a straw man. They do not have clean hands either.

Law School

[Anonymous Coward]

If you have any doubts about the ethics, then you should go to law school. Not to learn about ethics, but because you've already shown they don't really matter to you, which is the main requirement to take the LSAT.

You want to bet?

[argent]

The TOS isn't a contract.

I wouldn't want to play musical lawyers on that basis.

http://yro.slashdot.org/yro/08/07/07/1824228.shtml

Re:You want to bet?

[clone53421]

That particular person wasn't charged with a felony for violating TOS, the felony was for violating TOS with intent to cause harm. Which is an awful lot like saying "knives aren't illegal but killing people with them is"... which begs the question, in this hypothetical knife scenario, "Why do we need a law stating you can't kill someone with a knife when a law stating you can't kill someone in any manner would cover that possibility?"

In other words, why do we need to prosecute this woman for "violating TOS with intent to cause harm" when violating TOS wouldn't be an act with legal repercussions otherwise? (For that matter, as someone else said somewhere else, Megan Meier, the girl who committed suicide, was violating MySpace's TOS as well. Megan was 13; MySpace's TOS require users to be 14. Violating TOS is not and should never be a crime.)

Re:You want to bet?

[Fulcrum of Evil]

Which is an awful lot like saying "knives aren't illegal but killing people with them is"... which begs the question, in this hypothetical knife scenario, "Why do we need a law stating you can't kill someone with a knife when a law stating you can't kill someone in any manner would cover that possibility?"

How does that beg the question? It asks the question, but doesn't beg it.

Re:You want to bet?

[clone53421]

I meant it in the "asks the question" sense, not in the "begging the question" fallacy sense.

Re:You want to bet?

[argent]

That particular person wasn't charged with a felony for violating TOS, the felony was for violating TOS with intent to cause harm.

And this article is all about someone violating TOS to engage in theft of services.

In other words, why do we need to prosecute this woman for "violating TOS with intent to cause harm" when violating TOS wouldn't be an act with legal repercussions otherwise?

I'm not the prosecutor, and I'm not in agreement with the prosecutor. I am simply noting that its far from certain that violating the TOS of a site with intent to steal the services of that site is legal.

Re:You want to bet?

[clone53421]

The part that violated the TOS of the sites was explicitly different from the part that involved "distributing" use over several free accounts. Maybe I'm incorrectly making that assumption, but that's how it was presented in TFA.

Re:You want to bet?

[Fulcrum of Evil]

By which is mean you meant to say 'asks the question'? Begging the question never means asks the question.

Re:You want to bet?

[clone53421]

I suggest you google "begs the question" and "begs an answer" and tell me how many of them were "correct" uses of the "begging the question" phrase with regards to the fallacy of the same name. It often means that, albeit perhaps incorrectly used.

Re:You want to bet?

[Fulcrum of Evil]

No it doesn't bloody mean that. Just because a bunch of people are illiterate mouth breathers doesn't mean that you're right.

Re:You want to bet?

[clone53421]

Beg the question. Note definition 2.

Now fuck off. Your pedantic correction has been duly noted and I really don't care. I'm a pedant myself and you're pushing it with this one. I'll give you the correctness for correctness' sake, now just bugger off. You know what I meant.

Re:You want to bet?

[Fulcrum of Evil]

Note that Begging the Question is a formal term and popular usage counts for fuck all. Don't use sloppy language, it's just irritating.

Re:You want to bet?

[clone53421]

If you'd simply pointed out that "begs the question" is technically incorrect, in some people's opinions, when it is used in the manner I had used it (which I honestly didn't realize), I probably wouldn't have minded the correction.

Re:You want to bet?

[Fulcrum of Evil]

I figured that you read your own link, which specifically states that people are generally using the term incorrectly.

Re:You want to bet?

[clone53421]

I did, and it does, and "general use" is applicable to the English language. Check the first (original) definition of "nice"... next time you say something's "nice", keep in mind that you're using a profaned definition of the word that has come into "general use" and therefore is considered acceptable.

Re:You want to bet?

[clone53421]

By which is mean you meant to say 'asks the question'?

Say what?

Let's call us even, hmm?

Re:This does not sound like a boss with a career p

[JoeMerchant]

See: http://www.dilbert.com/strips/comic/2008-10-27/

I had a similiar situation. . .

[Anonymous Coward]

I was asked by the president of my company to conduct espionage operations on a CEO of a company that he wanted to buy us. I simply said no. A few months later I was let go, I'm not sure if it was related. Would I do it if I thought it would keep my job, no. It was against my ethics, and I would never commit a crime. I'm a good person.

They also asked my coworker, a young college intern, and he was excited by the idea of getting to conduct espionage, but he wasn't certain if he should. He asked me, and I tried to talk him out of it, in the end he decided to tell them no as well. Or, at least, that's what he told me. He could have been working on it for them anyways.

I loved working there, and I wish I still was. I really enjoyed the work, if not the people.

It's not "illegal".

[Anonymous Coward]

Not abiding by a website's TOS is not "illegal". At most there could be a copyright violation, because you're making a copy of their copyrighted data without a license to do so. Which is a civil matter and not a criminal one. (No quibbling responses will be allowed that sometimes copyright violation is a criminal matter. This is almost never the case.)

But this is an "at most". It's 95% probable that the scraping is not "illegal".

play both ends against each other

[JustNiz]

Do what you boss wants, but anonymously tip off the sites that you're scraping and having multiple accounts with. Let them deal with it legally.

Do it, But Do it Poorly

[Udigs]

That way, when it fails, you can switch to the paid accounts. Which, BTW --- $2,000? Where the fsck are you hosting? For that price I can get 6 quad dual core servers in the cloud with 8 GB of RAM each. That's A LOT of free accounts.

Maybe if you presented a cost-effective alternative your boss would jump for it?

What Should You Do?

[Timtimes]

When faced with situations of moral or legal ambiguity on a personal basis like this, your decision to 'go along' with the scheme will be influenced by multiple factors. Many companies try to instill a large bit of 'team spirit' into the operation as a binding agent. Seems to be specially prevalent among the shysters. Cue the Enron employee conversation regarding screwing over "Grandma Millie" on her electric bill and remind yourself that there was an entire corporate mindset that involved thousands of people, all pretty much resolved that it was a noble mission to screw people over as badly as possible (with special glee reserved for those most damaged). What more would they have done, and how far would they have gone in pursuit of money and career position if they were already laughing at the suffering they were inflicting on OLD PEOPLE??? Business ethics like that result in individuals robbing the entire nation's (world?) banking and finance industry, and then for kicks getting the government to hand over the treasury as a 'reward'. You pull off something that crass and evil in the business world and they'll fly you and your minion off to a half million dollar holiday weekend in the Bahamas. It often boils down to how much internal dissonance you can tolerate versus your ability to otherwise provide an alternative for providing Maslow's basic hierarchy of needs and what level of suffering you are willing to go to in regards to your personal integrity. Enjoy.

case law

[Benjamin_Wright]

There is some case law support for enforcement of web terms of service. It's a complex topic. For discussion, see http://hack-igations.blogspot.com/2008/05/google-privacy-policy-terms-of-service.html and http://hack-igations.blogspot.com/2008/02/contracts-for-patient-privacy.html -Ben

OR in simpler words

[Anonymous Coward]

If you have to ask "is this ethical" it isn't.

Email != documented proof

[MadCow42]

Why do people, especially tech-savvy people, believe that having an email record is "proof" of anything? It's just like accepting faxed copies as good as originals. In this day and age, this stuff is SOOOOOOO easy to fake!

If email was considered proof, I could prove that my boss offered me a $2M bonus for keeping quiet about his gay love affair with the CEO. Unfortunately, it wouldn't hold up past the first glance.

If you want proof of submitting your claims, do it the old fashioned way... use the legal process. "You've been served". Short of that, it's all hearsay.

MadCow.

Re:Email != documented proof

[Uberbah]

Then print off a copy and have it notarized.

Re:Email != documented proof

[1u3hr]

If email was considered proof,

My printouts of emails from my former boss were accepted as proof by a magistrate when I sued him for late payment of wages. Anyway, courts can also accept oral testimony based on your memory of events, your own notes, etc. It's up to the opposition to disprove them then.

Re:Email != documented proof

[LunaticTippy]

You should look into e-discovery. There are billions spent every year on this. I've been at 2 companies that had expensive e-discovery actions. If emails weren't valid in court, courts wouldn't be demanding them and companies wouldn't have to spend millions gathering electronic documents.

With any technology there is always a risk of counterfeit evidence, from spoken word to paper to holograms. Just because something can be forged doesn't mean it is never used as evidence in court.

I would polish up my resume...

[Wornstrom]

Does the furniture in your workplace consist of folding chairs and tables? These duct tape and popsicle stick type projects tend to become maintenance nightmares...

And while you're at it..(Re:You're Right...)

[bwcbwc]

Make sure you use your boss's name and email for all contact information on the user accounts you setup for the scraping.

Business Philosophy Lesson for Blackfolks

[Timtimes]

In the mid nineties at Jackson State, I took a course on marketing ethics that highlighted some prominent examples of large corporate malfeasance in relation to marketing. A noted example was marketing of high priced baby formula to third world countries by Bayer. I also had a post-grad class on management ethics that highlighted a case where a large defense contractor deliberately downsized the brakes on a military jet to save a few dollars, thereby knowingly risking the lives of the flight crews. The troubling thing about these courses was the emphasis on how the whistle blowers always ended up suffering. It was STRESSED that you should NOT be a whistle blower. I was specially critical of this message, given by a white life insurance agent (doubling as college instructor) to a predominantly black audience (graduate class at Jackson State University). As the only other white guy in the class (of 20), I questioned this teacher's ethics in carrying such a message. His answer to my concerns? "These young black people have a hard enough time in business without going on a crusade of justice and righteousness within the business community". He was right I suppose. The young black students needed to be indoctrinated with the idea that white folks have set-up a business ethic that believes it's only illegal if you get caught, and then, we'll have our cronies cover the mess you make. American business schools seem to be in the business of making sure that you can suppress whatever humanist values you hold that would be a hindrance to the organization. They teach that the values of top management are loyalty over all. I wonder how many of my black classmates got suckered off to Arthur Andersen when they were heavily recruiting minority employees way back then? LOL. Enjoy.

Ask yourself the real question

[popsicle67]

This Monkey you call a boss obviously knows this isn't kosher so your real question is "Why is it me" not "Should I do this". If you are the only one who can do it then you must ask "Will he throw me under the bus if this blows up in his face?".

Unauthorized access to a computer system.

[argent]

Not abiding by a website's TOS is not "illegal".

We'll see. There's at least one case where violation of a TOS is being argued as "unauthorized access to a computer system". This one is still in the pipeline, but I sure wouldn't risk it.

See http://yro.slashdot.org/yro/08/07/07/1824228.shtml

Be loyal to the Company, not the Boss

[natoochtoniket]

1. This design is clearly unethical, because it violates the TOS of the subject web sites.
2. It may be unlawful, but the company should consult its lawyers on that.
3. The design won't work for long. Web sites change page formats very often, so screen-scrapers require a lot of unscheduled maintenance programming.

So the thing to do is to express your concerns, in written form. The memo should say:

Re the first application:
1. The proposed design appears to violate the TOS of the subject web site. Because TOS agreements are a form of legal contract, we should consult the company lawyers before proceeding.
2. The company that runs the web site may discontinue the site, change the screen formats, or feed invalid/wrong data to the application. This will make our application unreliable and prone to unscheduled outages.

Re the second application:
1. The proposed design appears to violate the TOS of the web hosting site. Because TOS agreements are a form of legal contract, we should consult the company lawyers before proceeding.
2. The cost of developing the distributed application, and maintaining it, will probably exceed the $2000/mo that would be avoided.
3. If we go ahead with this development, the web hosting site may discontinue the free service at any time.

Finally, the memo should ask for a go/nogo decision on each of the two projects.

If the decision is "go", in writing, then don't worry about it. If it is verbal, reply in writing to confirm that verbal communication. In either case, update you resume.

If the decision is "nogo", take credit for saving the company from potential liability. And, of course, propose an alternate design.

Just another day...

[tripdizzle]

I used to work in customer service for a fairly large e-commerce outsourcing company. We broke the TOS's given on our websites daily. The terms of service were basically thought of as best practices. If you wouldnt break them, you wouldnt be there for more than a day, but most employees never even read them.

I think you're confused about what spam is...

[argent]

Spam is unsolicited broadcast messages to people you don't know.

It doesn't matter if you offer an "opt out" option or not... giving people "one bite at the apple" with an opt-out mechanism doesn't change the nature of the process, because there are too many potential spammers for "one bite" to scale. In the average city there are at the very least tens of thousands of businesses, organizations, and activist individuals who can make a good case that they should have the right to spam you with a valid "opt-out" address. On the Internet, every city on the earth is next door to you, that's millions of "legitimate" spammers... if "one bite" was acceptable. That's just not manageable.

If they already know you, if they already have some kind of relationship to you, that cuts the number of entities that can make an argument that they should be able to send you a single opt-out message from millions down to dozens. THAT's manageable.

That's the difference between a legitimate (if possibly annoying) email and spam.

When best-buy sends you an e-mail about their latest deals, that's spam.

It may be spam, or it may not be. If you gave them your email address in the course of your business relationship with them, it's not spam. Because you're not someone they don't know.

Open up the data in the appropriate format.

[Anonymous Coward]

I've been in a similar position, scraping part numbers and sales data from a manufacturer's website.

Basically if they had offered a proper database account we could do all of the data pulling The Right Way(tm). Instead we are forced to strong arm the information out of their website instead (and that was NOT violating the TOS for that particular instance).

Fix the problem at the root: have the company open up its data in the appropriate format (possibly with a small fee) and their customers will be able to get the information they need without maligning the web server.

-keless

Hixardt

[DankNinja]

    I was recently in a similar situation where I had to choose. I worked for Hixardt Communications (in Pensacola, FL) - they are a pretty small "IT Firm".

    I started seeing complaints come in to the abuse box from the upstream provider (Southern Light) about spamming from the Hixardt network. They were the scam-type "click here, you've won something". Apparently, the morons at Hixardt allocated 4 /24 networks to only 6 systems to spam from hundreds of fake host names.

  Southern Light gave these guys these ranges with the provision that they would not violate their ToS against spamming. Well, the owner of Hixardt and his buddies decided to do it anyway and received a 3 month letter from Southern Light that told them to stop it or pay them for the addresses. I was expected to help Hixardt out of this situation by moving them to Cox and lying about the # of addresses needed. I refused and was fired by Mike Hicks. But, at least I can sleep at night.

I smell a sock puppet.

[argent]

Either a sock puppet or someone who is improbably naive.

But your fuel tax pays for the road that goes in front of the billboard. So yes, you ARE paying to see that billboard.

You would pay the same amount whether the billboard was there or not.

The electric company is using the taxes and profit from YOUR electric bill to run poles and wires to that billboard

No, the billboard owner is.

They're also paying property taxes, usually at an extremely high rate, as well as up-front fees.

If they don't pay tax and power bills in your city, talk to your local representatives.

Re:I smell a sock puppet.

[Anonymous Coward]

Sock puppet? GP posted anonymously... -.-

I mean, yes, he's a raving loon, but hardly a sock puppet.

Re:I smell a sock puppet.

[argent]

When you're dealing with spammers, an AC counts as a sock puppet. It's SOP for them, leaving anonymous defenses of spamming on forums. What's surprising is that some spammers seem to have mod points now.

You're asking the wrong question

[pushf popf]

Sure, it's immoral, and you're violating the other website's TOS. That, however is completely irrelevant.

What is relevant is that any feed you use that isn't backed up a valid contract, can and will disappear at random times, sometimes permanently, as well as contain data you weren't expecting and be missing data you were expecting.

Ask your boss how happy he'll be when the domain owner sells to a spammer and his scraped data is now "Male Enhancement" ads instead of weather data.

Only following orders?

[Anonymous Coward]

To paraphrase

"My boss tells me that my job requires me to kill people - is that wrong? Should I do it?"

This is a nobrainer.

Re:Moral/ethical issues, employment and unemployme

[mschuyler]

I once worked for the Department of Employment Security. They sent me to two weeks of school learning the system. I worked one week before they laid me off. Of course, by then I knew all the rules for getting unemployment and managed to double my claim.

Basically the issue is this: If you quit, then the onus is upon you to prove you quit for cause. If you are fired, the onus is upon the employer to prove you were fired for cause. Within those two parameters anything can happen, and a lot depends on whether the former employer tries to fight it. Whether or not your weekly check from unemployment is enough to pay your bar bill, having gone through the process of claiming unemployment may help you with a legal paper trail that may be useful in the future. Your former employer may wind up trying to bite you, and your furure employer may have questions. The paper trail can help you prove your case.

Re:Moral/ethical issues, employment and unemployme

[uneek]

Hi:

Being someone who has collected such benefits in the recent past, I can say that this is not a good alternative.

Given a choice between getting paid 1/3 of my salary or being unethical, I would choose the latter.

Although I would do my best to negotiate my way out of it.

I would not recommend this...

[Tord]

The first steps are fine, but I would not recommend you to take the option step of blowinging the whistle unless you really feel strongly about the site or people you "victimize" and see it as you moral responsibility.

If you accept the job and then turn around and blows the whistle you have acted maliciously against your employer. They may have questionable morality but the fact is that you have agreed to work for and being loyal to them, don't sink to their level. They might even have legal grounds to sue you if they find out since you clearly have willingly sabotaged their business.

The only way to take the moral high ground here is to first try to make them change their mind and if that doesn't work refuse to take part in the scheme or at least demand in writing that management take full responsibility. Yes, that could have very bad consequences too. I don't envy your situation, I've been there myself a few times and have not always made decisions that were smart or made me feel good in the long run...

Of course, if things went far enough I would blow the whistle, but I don't get the impression this is one of those cases. It would be a totally different matter though if you weren't working for them or in any other way had promised your loyalty. In that case I would recommend you to blow the whistle as a concerned citizen.

Re:I would not recommend this...

[quanticle]

They may have questionable morality but the fact is that you have agreed to work for and being loyal to them, don't sink to their level. They might even have legal grounds to sue you if they find out since you clearly have willingly sabotaged their business.

Since when did going in to work require you to hang up your morals and ethics at the door? If your employer is doing something unethical, many would argue that you're obligated (morally) to blow the whistle on it, since to do otherwise allows people to profit from unethical examples - setting a bad precedent. If your employer is violating a contract, you'd call them out on it. And that's exactly what a Terms of Service agreement is - a contract specifying the terms by which you may use the other site.

Don't worry about it

[OrangeTide]

The TOS aren't (usually) legally binding. IANAL but they can only cut off your access. As for the ethics, it's for your boss to worry about because he/she takes the responsibility. You can point out your concerns but ultimately it isn't your decision. If your boss listens to your concerns, but disagrees. That's fine. If your boss chews you out for not being a team player or won't even discuss the issue, then that is a better reason to quit than some ethical dilemma.

You can quit over it pretty safely, because in many states employers aren't allowed to speculate why you quit when being called as a reference. But I wouldn't quit unless there were some other factors that make you unsatisfied with your job. I would be willing to suggest that if you want to quit over something like this that you're just looking for a reason to quit, and that you either figure out why you don't like your job and keep it or quit before you get fired over something stupid/careless.

It's not like they are grinding stray dogs/cats into dog food, or hiring overseas child sweatshops. Your ethical issues seem to rank pretty low compared to the real shit that happens in business.

Scrap the Scrapper

[Anonymous Coward]

I say build the scrapper. Then after a week or so make it break yourself. Inform your boss that they discovered your scrapper and fed it false information. Also include that finding it once so quickly implies they will find it if you attempt to perform a work around. Then tell him it would be dangerous to continue less he wants legal actions taken against him. Finally recommend against using this method again for the previously discussed reasons.

Ombudsman is not your lawyer

[mathmathrevolution]

The ombudsman is there to represent the interests of the employer. He protects the company interests by adopting a neutral, sympathetic tone. He is there to collect and address grievances from the staff and recommend solutions. Whenever possible a good ombudsman will arrange a win-win situation. But if the situation dictates that either the company or the employee lose, the ombudsman will do his best to act like the employee's trustworthy friend while ensuring that the employee loses.

An ombudsman could be clever and underhanded. A slimy ombudsman could, for instance, recommend an employee talk to a recommended "stress management expert" -- then in an eventual trial the company could smear him in court as an unhinged loony undergoing therapy. Now obviously this would violate the ombudsman Standards of Practice, but guess what? Company's don't hire ombudsmen to adhere to some Norwegian Social Democrat's "Standards of Practice", they hire ombudsmen to make problems disappear and to prevent them from happening in the first place.

Example of Ombudsman BS: My girlfriend recently went to her company's ombudsman to complain that the aerobics instructor the company had hired was batshit. The ombudsman immediately broke confidentiality explicitly violating his "code of ethics" and the personal guarantee of strict confidentiality he gave her. The ombudsman casually revealed her complaint to his secretary, even though he was told that his secretary and the batshit aerobics instructor were friends. The angry and still batshit aerobics instructor confronted my girlfriend about this after the next aerobics class. And the ombudsman never got back to her.

If you are being victimized by a line manager in a way you know upper management would disapprove of, going to the ombudsman might result in a positive change. But even in that case you still might be better of talking to your own lawyer.

If you have been victimized by your company and are considering seeking legal remedies, get a lawyer. The ombudsman is not your lawyer. If you belong in a union, then your union may be willing to assist you, but they might sell you out. Ultimately, only your lawyer is your lawyer. 99% of the time the ombudsman is just a person who acts really sensitive to your issues and tries to assuage you with sentimental cliches.

Re:Ombudsman is not your lawyer

[nabsltd]

If you have been victimized by your company and are considering seeking legal remedies, get a lawyer.

Agreed, but in this case, there is no "victimization" going on.

It's merely a matter of opinion of what a company should do about a purely internal matter. Although the issue is ethical in nature, it is no different from a business perspective than if your manager told you "use blue text on the website" and you disagreed.

Bringing in a lawyer to that sort of thing can do nothing but harm.

I guess I'm just some goody goody

[otopico]

But if you think it's unethical, it probably is. You can still try to explain to the boss why it's wrong and if he makes you choose, well, you get to decide.

For me, I'd have to leave. You boss sounds like a sleaze and people like that aren't worth your loyalty, as most of the time, they have zero loyalty to you. If he will screw over people he doesn't know, because he doesn't want to pay them, why do you think he wouldn't screw you over if it directly benefited him? Get out while you can.

People need to learn to have principles, and just because it's not, as one reply put it, 'grinding stray dogs/cats into dog food', doesn't make it any less wrong.

At the end of the day you have to live with yourself. Can you do it in good conscience? If you can't then you probably shouldn't.

The only thing necessary for the triumph of evil, is for good men to do nothing.

Re:Redirecting content

[cynical kane]

Can you post the URL?

get the corporate legal counsel involved

[jay2003]

Get the corporate legal counsel involved. If your firm is large enough to have an in-house attorney, this is really easy to do. Just tell your boss that there are certain legal risks to the company from scraping and you are going to talk to the corporate counsel to make sure your doing it the right way.

If you work for a large company, it's highly unlikely anyone objects to asking legal. Large companies are very CYA and no manager wants a paper / email trail where an employee said maybe we should ask legal for guidance and the manager said not to do it. Smaller companies there can be more resistance because they have less of legal/regulatory compliance mindset. If you frame the issue as you need to protect the company from liability, it's harder for your boss to object.

If the attorneys say 'ok' and you still have ethical issue with it, then you have to decide whether you want to work there.

I'll say it as a hardcore internet anarchist

[Yvanhoe]

If you post information on Internet, you allow users to use it the way they like.

I would also add that getting raw information from a website is a feature that, if missing, is completely ethical to add. There are countries where reverse-engineering for the goal of enhancing interoperability is legally protected (Scandinavian countries) I see this as just an occurrence of this principle. It may not be legal in every country but I see it as clearly ethical.

Oh, by the way, by reading this you agree to send me your life savings and your first born. Cownboyneal has your IP, he will send you a notice shortly.

Re:I'll say it as a hardcore internet anarchist

[Anonymous Coward]

You wouldn't want my first born. He tore the beating heart out of the last person who said that. And ate it. Live. Then he laughed.

I hope your boss doesn't read slashdot...

[Wiseazz]

The problem may just resolve itself.

Re:Redirecting content

[Free the Cowards]

Assuming you mean the 20000x20000 jpeg, it appears to be circulating the net as a file called "dontloadthis.jpg". Here is an example.

Re:Moral/ethical issues, employment and unemployme

[Rastl]

OP was asking for options. This is one if he can't work there but doesn't have another job lined up immediately. 1/3 of an income (it's more in my state, btw) is better than 0 income.

This was more in response to the "Just quit dood" kind of answers that continually crop up when there's some kind of job dilemma. Having another job lined up immediately is nice but not necessarily possible. And staying there means working on the project.

So before the unemployment insurance compensation gets dismissed out of hand realize it's one of many options. And it was one he may not have considered since most people think you can only collect those benefits if you're fired.

FUBAR

[Anonymous Coward]

If they pull the "answer by phone"-trick then I'd say you are deep into FUBAR country and you need an ally if you are to stay at the company.

The very best solution is to interview like crazy and jump ship ASAP, in fact, this is step 0.

Even if it's not possible to get a single smoking gun email from the boss, start needing clarification on many simple details, be sure to include incriminating info in every email, so when the loathsome bastard gets into the habit of answering the simple questions you will be able to build proof that he knew full well what was going on.

You need to realize that the evidence gathering will only be useful in getting the bastard fired and keeping yourself out of legal trouble, it's not going to save your future with the company.

Talk to HR, your bosses boss and/or colleagues and see if it's possible to someone on your side, once you know who you can trust start BCCing them on every single incriminating email to your boss and forward them copies of the answers. .. but get out.

Re:Redirecting content

[Anonymous Coward]

So how's that JPEG compression workin' out for you when your entire palette is ONE COLOR?

Don't do it

[PatTheGreat]

No, dummy, you shouldn't do it. Ethics don't change even if your boss told you to. If you wanna get all Godwin with me than I can tell you that troops are still responsible for having followed an order if it's unlawful, that is, something like this. My point here is if you do this, you're still in trouble. Don't do it. Tell your boss it's illegal (or at least against the TOS) and that you shouldn't do it. If he still wants you to do it, tell your boss's boss what your boss wants you to do. Wash, rinse, repeat until you aren't being told to do it or you manage to find work that doesn't require you to break rules.

#3 is a bad idea

[Yaur]

because you are actively interfering with your employers business. You are better off just refusing to do it on legal grounds (be sure to document this) and then suing them if they fire you for not breaking the law.

Same problem...

[sbeckstead]

I had nearly the same problem. My boss asked me to do something that was questionably legal but quite unethical. I found a work around that was both legal and ethical within the parameters of what he wanted me to do but it was very inefficient. When I presented to him the completed work he told me "Next time do it my way" so I found another job, gave less than a week notice and left as quickly as I could. I felt no regrets at leaving and giving less notice than my "contract" required because they had me as a contractor and I was actually a statutory employee if they had tried to hold me to the "contract" I would have gone to the labor board and the IRS and they would have been screwed and they knew it. But you probably don't have that much leverage and my advice would be, if you can't find an ethical way to do it object to your boss and his boss simultaneously and if still ordered to do it publicly and noticeably retain a lawyer and let them fire you. I think in the end you are going to wind up leaving this company any way unless your boss is removed over this.

Even major ISP's are involved in this stuff

[Anonymous Coward]

Hi there. I congratulate you on your ethics. My ISP (BT Retail in UK) are involved in doing this sort of thing with a product called BT Webwise.
http://www.donottrustwebwise.org
It is installed in the ISP network, between the user and the ISP and intercepts, redirects, and profiles all the customer's browsing without their consent. If they opt in to having that profiling passed on to a company called Phorm then the results of their browsing including the copies of website pages and a unique profile of their activity is used to create a customer profile to deliver behaviourally targetted adverts. Website owners are basically ripped off with their copyright ignored. BT Retail are getting away with this, with neither police, nor the ICO interested. I see that Ryanair sued a screen scraping site in the Irish courts and came out on top. When I wrote to BT's legal team asking if they minded me ripping of THEIR websites this way, and saying that if they didn't reply I would assume their consent, they never replied. It's wrong to steal website content, as it is covered by copyright just like any other intellectual property, and the UK government have said so - but they don't do anything about enforcing that it seems.

hmm... sounds familiar...

[Narcocide]

My advice, don't compromise your principles for your boss. Quit and don't even look back.

Re:Redirecting content

[Free the Cowards]

At over 400,000 pixels per byte, I'd say pretty damned well.

To quote a song...

[Stormwatch]

"When the truth walks away / everybody stays / 'cause the truth about the world is that crime does pay / so if you walk away / who is gonna stay / 'cause I'd like to make the world be a better place." -- The Offspring, "Have You Ever"

Re:Redirecting content

[fifedrum]

I usually redirect hotlinkers to pictures that either

a) advertise my site/interests exclusively as in "join my drum corps!"
b) show flesh in as naughty a fasion as possible, animation helps here

not because it costs me anything, but simply because I can, and it's funny to me... but the response is definately tuned to the source of teh intrusion and a forum for kids who like a certain cartoon doesn't get the tubgirl treatment while a forum for bag pipers gets worse

Anecdote

[elistan]

Here's my anecdote for your enjoyment and education: A friend worked at a cheese plant. Her job was to do QA tests - bacterial and antibiotic content of the incoming milk; protein, fat and moisture content of the outgoing cheese, etc. She was once told by the boss to falsify the reports of the outgoing product, to make it look like higher quality cheese than it really was. (No health issues, just a question of how much the customer would pay for the product.) She refused, as she was required to sign her name on the report. Instead, she performed her tests like normal, gave her boss an unsigned report with the real numbers, along with a blank report with no signature. He filled in what he wanted and signed it himself. She did experience retribution - poor performance reviews and reports saying she wasn't doing her job. I guess she could have gone to the FDA or USDA or something, but I'd bee surprised if anything positive would have happened, and she would have been working in a hostile environment in the meantime. Instead, she ended up quitting soon after, without even having a new job lined up. Fortunately, she quickly found something better and was much happier. Karma, if you believe in such a thing - she did the right thing and it definitely was the right move for her.

So my advice (IANAL) - refuse to do the work you know is wrong, prepare for a crappy work environment, and get that resume and those contacts warmed up. Because you should be in a better job regardless.

make your boss break the TOS

[tnordloh]

I hate to say it, but you should probably go ahead and involve Human Resources now, in order to raise your comfort level. Bring a printout of the Terms of Service agreements in question to them. Let them veto the whole thing, and you can be done with it.

Remember, TOS can be read by your boss and your HR department differently than us literal-minded Computer Guys would read it, and if they can read it in a manner that allows them to justify writing the screen-scraper, then I say follow your boss's orders.

In any case, by bringing this to the attention of HR, you have done your due diligence, and unless you are a legal expert, you can't be expected to do more than that.

In reality, you really have nothing to worry about by coding this up (as some earlier poster said). Also, screen scrapers on the web are more common than flies in a barnyard. It's far too expensive to sue your company, when they can just block your ip.

The other way

[Stu101]

Instead of just scraping, why not see if you can get a deal to get a copy of the data, updated as needed. This will cause much less traffic, cpu resources etc and the other party may just get a little cash !

In response to your sig...

[MythoBeast]

So only women think that you are overrated?

Sorry, couldn't resist.

Re:In response to your sig...

[Free the Cowards]

Remember what site you're posting to, now.

WANLs -- why not ask one (and remember Chip)

[Krishnoid]

if fired, hire lawyer and sue the company for unjustified dismissal

Why wait until you're fired? Many lawyers provide a half hour consultation pro-bono (you pay like a $40 administrative fee). Also be sure to read what happened to Chip Salzenberg in a very similar situation, when he tried to tell his management what was going on.

talk to hr

[roc97007]

You might want to have a private word with an HR rep. Make it clear that it's private, and tell them that you've been asked to do something unethical and you're not sure how to proceed. In cases like this, a company will tend to err on the conservative side and put the skids on the project rather than take the chance on liability. That should solve the problem. How to do it without revealing you as the finger man might be difficult.

If you do have to walk, make sure you get some face time with an HR rep on your exit interview.

Good luck.

A mole, huh

[Krishnoid]

I wonder if he could anonymously tip off someone who would aggressively want to prosecute these kinds of problems; like the legal department of one of the largest sites being scraped. We've got the BSA and gpl-violations.org -- why not something for this?

Best Answer by far

[XLR8DST8]

Let your boss know the risk he is facing and then ask him if he really wants to risk being caught and shut down unexpectedly, or worse, finding someone has poisened his data.

It's just not good for business.

now that is an intelligent and pragmatic answer.

now one with any instinct toward self-preservation will see this is much better than the 'get it in writing', flat-out refusal, or any other threats or immature behavior rank with self-righteous indignation any other of the slashdot amateur lawyers have suggested.

here you are not only voicing your own personal concerns but you are also giving your employer more insight into the situation and making him aware of the potential risks. you are preserving your own sense of morals and at the same time helping him refine his own.

this is not only taking the high road in a sense but also showing your boss that you're on his side and looking out for him as well, rather than just being difficult. he will appreciate this.

acting as a 'whistleblower' will only get you on his shit list and cause him not to trust you. so while you're interviewing at other places you will not have his support or recommendations later on.

also, to truly be on both sides of the fence, one should try being an employer as well. not all businessmen are evil. not all programmers are ethical.

in fact, i wouldn't be surprised if half of those who are saying this man is unethical for 'stealing' the data might in another breath have no problems with downloading music illegally.

when you think about it, he may actually be more ethical, as, at least he's using some kind of loophole within the existing system rather than completely working to break something outright(like circumventing DRM.)

i'm just sayin'.

I was just following orders...

[jotaeleemeese]

That is not an excuse, and most importantly, it is not a legal defence,actually in many industries (Finance and Banking for example) that is an admission of guilt...

No, no, no.

[jotaeleemeese]

You guys come with some ideas that frankly are astounding.

In many industries and legal localities that will not get you out of hot water at all, and as a matter of fact doing work that you think or know is illegal could be enough to make you an accomplice.

This chap should check if there are any company procedures to deal with such a situation and follow them scrupulously. If there are none, he must seek legal advice because just by knowing about this he may be already in trouble.

Re: Resume!

[TaoPhoenix]

"Coded a tecnically innovative bot that leverages the proportionate cost differential of linear purchasing of services vs. the hedged risk of total price of litigation."

Little Bobby

[Dogtanian]

I told you to scrape Slashdot, not read it. Now get back to work!

I've only one thing to say to Anonymous Slashdot Scraper... '); DROP TABLE rippedoffcomments; -- Goodbye!

it's what you're paid to do ????

[jotaeleemeese]

Unless you work for the Camorra, breaking the law or being unethical is not something that can be classed as proper employment.

You are paid to do a legal, I would say also ethical work.

Why anybody would consider legitimate to do something unethical just because one is receiving a salary is beyond me.

It really depends on the company.

[jotaeleemeese]

In serious companies, there will be a whistle blowing procedure, and frankly what one should do is to followi it with fastidiousness.

Such procedures will normally ensure that the whistle blower is protected.

If a company does not have such things in place then you are in very muddy waters. Depending on the industry (Finance and Banking being obvious ones) being in the knowledge that something fishy is going on puts you in hot water, and failing to report a problem may mean you become legally responsible for any wrongdoing as well. In such a company in such industries you should really talk to a lawyer (there must be very few companies like those left after ENRON, but who knows, life is full of surprises).

If the industry in question is not so unforgiving the best thing to do is to organize a tactical retreat, either to a different area in the same company or to another company altogether.

Doing something illegal or unethical is bound to end in tears and you don't want to be one of the parties doing the crying ...

No it doesn't

[jotaeleemeese]

Dire need does not give you moral or ethical grounding to commit unethical or illegal acts.

It is so simple but one needs to be a big man in order to live up to the most basic precepts of ethics and codes of conduct.

It depends.

[jotaeleemeese]

If your products or services end in properly regulated industries they may come after you.

This is frankly no laughing matter and the ramifications are ugly and unexpected, the easiest thing to do is not to do such stuff.

Pretty simple frankly.

Nonsense.

[jotaeleemeese]

Any company not hiring you because you are paying attention to their reputation is a company you don't want to work for anyway.

Doing your job is no defense.

[jotaeleemeese]

Not even an excuse, I am surprised how many folks around here don't even think that "doing their job" under such circumstances may be illegal.

So what do you do if they ask you to start a CD duplication business without paying the Copyright holders? You go ahead with it?

Honestly ....

Re:Doing your job is no defense.

[juuri]

No where did I say someone should do anything that would be illegal.

In my case it definitely *was* legal, but was something that personally bothered me on a moral level. Consider it a grey area where the law was definitely on the side of my employer but something you know in your heart just doesn't feel right. Like I stated earlier a lot of people might not even have given a second thought to it, but I had a different viewpoint because of my experiences up to that point.

This is not true.

[jotaeleemeese]

People working for a company are immune in regards to their personal assets when it comes to pay back any debts.

In any other circumstances where a company asks you to do something illegal regulations are pretty clear: as soon as you know you are supposed to report it (either to somebody in your company designated for that purpose or to regulatory authorities, if your industry is not regulated to such extent you may still be liable and should consult a lawyer).

The "I was following orders" excuse is the lamest one you can chose to protect yourself against corporate wrongdoing.

Re:This is not true.

[freedom_india]

The "I was following orders" excuse is the lamest one you can chose to protect yourself against corporate wrongdoin

The only way to force a corporate's Legal defend you is this excuse, unless you want to spend millions.
Tell me, how many employees have been convicted of wrongdoing versus CEOs?
Enron, WorldCom, etc., all had their CFO and CEO convicted: not the Traders or employees who clearly some things were illegal.

Re:Redirecting content

[W2k]

So sayeth Firefox 3.0.3 (Windows):

The image "http://lug.wsu.edu/~ben/dontloadthis.jpg" cannot be displayed, because it contains errors.

IE also refuses to render it. But when downloaded to desktop, I got a thumbnail, and I can open it in Paint.NET just fine (it's actually 0x808080 rather than 0x000000). Paint.NET swallows 3.2GB of memory when I do this. Good thing I have plenty of RAM and a 64-bit OS :)

What a load of tosh.

[jotaeleemeese]

If he likes his job or not is immaterial.

Depending on the industry and his legal locality he may actually be breaking the law, liking his job will not save him from legal trouble.

Re:What a load of tosh.

[nurb432]

its just a TOS.. its not like you are killing someone. If there is a civil suit, the company is liable since they ordered you to.

So ya, its back to 'if you like and want to keep the job'. Its your own moral decision to make.

In the UK that would be unfair disimissal

[jotaeleemeese]

You would have them by their b@@@s begging for mercy in an employment tribunal....

Since when companies authorize illegal activity?

[jotaeleemeese]

Your boss saying go ahead kid, it is ok *is not* ethical, and you will not be legally in the clear in all likelihood.

I am sure you can come up with many situations that are much clearer than this (let me try one: your boss tells you to hit with a car the star developer of your competition. He even signed a paper for you. If you think you would be OK you are really silly).

Here's my advice:

[V!NCENT]

Find another job you feel comfortable with. When you are asked why you want to quit your current job than say that your current company requires you to engage in illegal activity. Simple as that...

Re:Redirecting content

[jesser]

It will totally screw up the layout of any page linking to it if they haven't entered an explicit size for the tag.

That only really works against pages that use the HTML TABLE element for layout. Paragraph wrapping is normally not affected by the width of the page (e.g. due to a single long word or image elsewhere on the page), but tables cause it to be affected. To be fair, most forums use table layouts.

Re:Redirecting content

[Free the Cowards]

Yeah, I was thinking of forums. You're right that on other sites it will do less damage. (It will still push everything below it down a great distance, which hurts.) In my experience forums are by far the worst hotlink abusers but your experience, of course, may vary.

Re:Redirecting content

[Anonymous Coward]

Link please!

In other news

[Anonymous Coward]

Scrap is to scrape as lose is to loose.

Please update your spelling books kids!

Anonymous Email

[puggsincyberspace]

I would do the work, get it working, then a anonymous email to the sites concerned...

Just words

[Anonymous Coward]

TOSes are just words, meaningless words made by someone who mistakenly thinks they have any legal meaning.

Get Legal Involved

[RzUpAnmsCwrds]

The solution to these kind of issues is pretty simple. Tell your boss that you are concerned about the scraping causing liability issues for your company. CC your legal department contact on the issue.

If you believe CCing your legal contact when you have a legal concern is going to get you in trouble, there are real problems with your manager. You can go speak to your legal contact in private, and voice your concern. If you don't have that option, it's time to find another company.

Installing "Home" software for corporate use.

[Anonymous Coward]

My employer asked not only that I install "Free for Home Use" licensed software on corporate machines; but also on machines that are being sold. (We're a small OEM.)

I put my foot down. Told him that in spite of the fact that previous 'lead technicians' were willing to do this; I was not. If he was to provide me a signed letter from his manager authorizing this, and absolving all employees, I would do it. (I knew that his manager wouldn't.) Thankfully, he backed down, and agreed instead to put a link to said "Free for Home Use" software on the desktop.

Re:Installing "Home" software for corporate use.

[Fulcrum of Evil]

And if that turns out to be a criminal offense, how far would the signed paper protect you?

Convince him on the technical merits

[selan]

If your boss doesn't care about ethics, try convincing him that his plan is flawed from a technical perspective.

• Free accounts are unreliable and unstable. They have no obligation to provide any level of service and can be canceled on you at any time.
• Screenscraping only works as long as the sites you are scraping keep their format--and that can change at any time without notice. Again, not very reliable.

It's been said before

[Anonymous Coward]

If the company is sued and the blame has to go somewhere, will the boss defend you or leave you as the scapegoat? "I just told him to get me some data like this."

If he's unethical enough to not pony up $2,000 (a drop in the bucket for a decent sized company) on a lark, he's probably going to happily leave all the blame on you.

If $2,000 dollars is really too much for him to afford, and this project is of vital importance to your company, consider that your company may not be sufficiently solvent to survive and may have an untenable business model. If it's really important, a higher up will watch to make sure it gets done right.

If your boss was concerned with you, he wouldn't be asking you to do this. He probably doesn't much care about you. Many are desperate for jobs, you can be replaced.

Consider your boss'es temperment. If thwarted, will he fire you from spite? Are you in a right to work state where he can drop you "without reason"? Do you feel comfortable that if you said no you would be gone? Can you apply for a different position in the company? Do you want one? (Are other bosses going to be similar?) Is your business retaliatory, laid back or governed-by-external-regulations sort? If the latter, does your company believe you can't spell compliance without liance? (Dilbert rip-off)

Even if this is minor, what will this embolden your boss to ask for later? If your scared saying no will get you fired, consider that the sort of person who would fire you for being ethical is the sort who will use your fear of being fired to manipulate you endlessly (long unpaid hours, canceled vacations for sudden "urgent" priorities, ever greater offenses you commit under him for fear of him turning up previous things. Hopefully at the moment, your boss has no dirt on you. Keep it that way if possible.

One more important question, ask if he'd be willing to consider negotiating to find an acceptable price with the company, explaining that they likely will catch on and block you, if not sue you. That they HAVE to or someone else would siphon ALL their content and force them out of business. If he's unwilling to even CONSIDER negotiating with the other company for a somewhat lesser feed, but willing to steal it all, dump him. If he tries negotiating, they've got a heads up if he tries scraping later. If the data isn't worth his $2,000, is it worth anything to your company's customers? Will they find the original source superior? Do you add enough that if you did this legit you could compete?

Quit

[fortapocalypse]

If you don't agree with your boss, and your boss's boss (if you have one) is unlikely to care, then find another job, whilst working on what your boss asked you to do. You'll likely be happier somewhere else if your boss is that much of a tool. Note though that a lot of (usually small) business can be in the grey area, so you are likely to run into similar (although maybe a little more "grey") situations in the future.

Escalate

[Anonymous Coward]

If you like the company, and the world (apart from the morally/ethically questionable project, would it be an option to speak with your manager's manager?

Simply have a quiet chat and tell him your concerns, and the fact that your manager has disregarded them, although they would put the company in a position where they may have legal troubles.

Worse come to worse, you have let them know about the issue, and tried escalating it, if the manager's manager tells you to do it anyway, then I would suggest that the culture in your current employer is not conducive for an information worker - if they value another's work so little as to refuse to buy it, chances are they will not value what you do for them.

I'll second that...

[SaDan]

The company I work for redirects scrapers, or offers to sell the data in an easier to obtain format.

I have a different opinion

[jawahar]

Engineering is about thinking and acting out-of-the-box

Yodlee

[jawahar]

I believe http://www.yodlee.com/ also does the same.

Learn to read.

[argent]

Try "direct mail has a cost that limits its abusiveness more than spam, therefore it doesn't need to be restricted as much".

Re:Redirecting content

[Ant P.]

The other thing that'll do is get their users EXTREMELY pissed off at them when the site crashes their browser as it tries to create a 400 megapixel uncompressed image in RAM.

I had a similar thing happen to me once, so I gave them a 2MB HTML page of tags. 2MB after gzipping, that is.

Re:Redirecting content

[harry666t]

Man, can you share that jpg? :D

my 2 cents

[globeadue]

Might want to consider pointing your boss to this thread? And if he still wants to go forward, cover your butt, either have the email thread that you have advised him that you believe that proceeding with this plan will violate the terms of service from the companies involved. or have him sign a note indicating such.

Re:You're Right, Of Course What YOU can do

[davidsyes]

I would almost suggest... no, i WILL suggest... would it be OK to - upon detecting scraping applications - to have a server-side script monkey with the page numbers? I am sure they could first download the page and then scrape it locally, but if they are scraping live, and bogging down the site, would it be legally justifiable to alter the numbers to damage their reputation?

Wouldn't this be a case of "vengeance is mine" if YOU own the data, and have screen-scrapers not paying for some fair access amount to it?

Re:You're Right, Of Course What YOU can do

[davidsyes]

Sorry, my previous comment WAS redundant...

Re:Redirecting content

[mackyrae]

One of the women on LinuxChix has a png (I think) with corrupted headers so that it makes the browser load an insane amount of memory and crash the browser. Modern browsers are bit smarter about going "oh that's corrupted, nevermind" though.

Does your company have a legal department?

[billstewart]

I don't know the details of what you're trying to accomplish here, so I can't comment on the legality, but it sounds like you think there are some issues, and you _and_ your boss need to discuss it with your company's lawyer (if you have one.) And you need to explain to your boss that terms of service can be legally binding, and can have copyright issues, and also that web-based data providers not only change data formats enough that your lost programming time might cost more than buying the data you need, but also that they can look for automated scraping and hand you bogus results.

The robots.txt files were originally designed to tell robots to stop doing heavy-duty queries on sites that couldn't handle it, but also to keep outsiders (especially spammers) from abusing content. And your code needs to follow robots.txt, because many many sites have robot traps that do unfriendly things to unfriendly visitors ("Nice lookin' robot you got there - be a shame if anything happened to it!") - or do userfriendly things to them.

how are you seeking your answer?

[vldmr_krn]

I hope you walk. You won't be damaging your "reputation and future employment opportunities" by leaving.

You can do the project; you can decline to do the project but stay with the firm; you can walk. How much future self-esteem would each choice cost? How much would each choice affect your present circumstances? What's the best trade-off?

TOS are backed by cracking laws

[Paul Johnson]

I seem to recall that in the US terms of service have been found to define "authorised access" to a computer, and access outside of the TOS is therefore unauthorised. That puts you in direct violation of US anti-cracker laws about unauthorised access to a computer. If more than $5,000 worth of "damage" is caused (including investigation and cleanup costs) then it carries a maximum of 5 years in the pen. If its done for gain (as in this case) then thats 10 years.

Re:Redirecting content

[Anonymous Coward]

Reminds me of that guy who asked /b/ what to replace the most frequently leeched image on his site with. (in case you were wondering, the winning picture was spincock.gif)

And furthermore...

[hyades1]

You've already received a lot of good advice on how to handle this situation. No matter the outcome, though, I'd be looking around for a transfer or another job.

Your boss has given you a pretty good idea of where his moral compass points...which means that if this particular attempt at douchebaggery falls through, you can rest assured he'll come up with another one, probably sooner rather than later. You can also be confident that if there are consequences, he'll have a sacrificial lamb ready for butchering. Can you guess who that might be?

You're already screwed

[shentino]

There are four situations

1. It succeeds. Your boss, being devious enough to foist this on you in the first place, is going to grab all the glory anyway and your efforts will be of no benefit to yourself. You'll be so sick of yourself your performance will decline, and/or your boss, knowing how easy you are to manipulate, will make your job such a living hell that you come to case 3 anyway.

2. It blows up in his face, his company gets hosed/sued/banned, you don't get the results...yada yada yada. Shit indeed rolls downhill, and as your boss has more benefit of doubt than you, anything goes wrong and you'll eat it. Again, because he was sleazy enough to foist this on you. Result?

3. You stand up for yourself, and your boss promptly fires you for insubordination, or starts cooking up bad reviews of your performance, or otherwise exacts revenge in a way that is toxic and probably fatal to your career.

4. You get in touch with someone honest enough to bring this nonsensical scheme to a crashing halt, and nobody is the wiser. At best, you get accolades for saving the company, and at worst, your boss gets vindictive at you for blowing the whistle and you get shown the door.

So either you'll get screwed on credit if it works, turn into a scapegoat if it blows up, get booted for having morals, or possibly get rewarded for saving the company.

Refusing to compromise your integrity will not cause any harm that wasn't already going to happen.

Worst case is the same no matter what you do, and best case absolutely depends on you doing the right thing.

Re:Redirecting content

[pacinpm]

Somebody once pointed at a picture of a frosted birthday cake on my web site from a forum. So I grabbed my image editor...

I think the whole idea of HTML is pointing to resources and not duplicating them. Do you work in RIAA or what?

Re:Redirecting content

[Free the Cowards]

Using other people's resources without their permission is not considered acceptable behavior where I'm from. The correct course of action is to find a picture of a birthday cake which explicitly has permission to copy it, then put it on your own hosting and linked to that. Using a person's birthday cake image without their permission is wrong. Linking to it on their server, well, you can do it but you're trusting them to serve up the right stuff. If they object to you taking advantage of their service and decide to change the data they return for that URL, you have only yourself to blame.