Should You Break TOS Because Work Asks You?

An anonymous reader writes "My boss recently assigned me a project that was all his idea, with two basic flaws that would require me to break multiple web sites' Terms of Service (TOS). Part requires scraping most of the site, parsing the data and presenting it as our own without human intervention. While we're safe on copyright issues, clearly scraping like this is normally not allowed. At times it might also put a load on those sites. The other is, for lack of better words, a 'load balancing' part that requires using multiple free accounts instead of purchasing space and CPU time for less than $2,000 USD per month. The boss sees it as 'distributed' computing when in reality it's 'parasitic.' My question is: am I wrong about the ethics? If I do need to walk, how best can I handle it without damaging my reputation and future employment opportunities?"

Comments

You're Right, Of Course

[eldavojohn]

My question is am I wrong about the ethics?

You don't even have to ask that question, this isn't even one of those interesting cases or gray areas. What you're planning to do is wrong--even though you could probably escape any legal ramifications. It sounds pretty clear that this site creates profit from these overly priced accounts for information that you obviously value at some amount. Getting it for free (regardless of the TOS) could put you at some risk for litigation. Using the term "load balancing" or even "distributed computing" is hilariously misplaced here.

If I do need to walk how best can I handle it without damaging my reputation and future employment opportunities?

Look, I understand what's it like to be looking for a job when the economy is bad. If there are forces keeping you pinned to this employer, I don't know of them. What I would retort with is "How can you keep working this job without damaging your reputation and future employment?" I mean are you going to put in your resume that you coded a technically innovative but bandwidth stealing parasitic botnet to duplicate content from a website that asks for a monthly payment to normally access it at that volume?

I would suggest you propose the $2k/month route and if your boss balks at it, start interviewing with other companies. If you have to leave and you're worried about being blacklisted as a 'whistleblower' (and your boss just might be that kind of guy) then tell him it's for monetary reasons that you're leaving and wish him the best of luck in his future scams.

Re:You're Right, Of Course

[Lumpy]

I say do it. but EMAIL your boss with your concerns and then continue.

when the shit hits the fan you have documentation to throw him under the bus hard and watch the wheels crush him.

Honestly It's all about CYA in the business world. If your boss tells you to do something unethical or wrong, document it every way you can and hold onto that so you can hand him over.

Why do you have any loyalty to them? they have none for you.

Re:You're Right, Of Course

[JosKarith]

What they're asking you to do is at the least immoral, possibly even illegal. Your employer doesn't have the right to ask you to place yourself in legal jeopardy in this way, and if the sh1t hits the fan do you really think that someone that came up with this scheme will balk at placing all the blame on you. Someone really needs to have a little chat with your boss about ethics...

Re:You're Right, Of Course

[SatanicPuppy]

All that will happen is that the site in question will blacklist your scraping application. I work for a media organization, and we deal with this stuff all the time. It's far more cost efficient for us to simply whack the application than to try and track down the jokers. It's actually pretty trivial to nail an automated scraper: they're obvious on the logs.

So the few times I've had someone ask me to do this sort of scraping, my response is usually that sure, fine, it works, but it's very easy to spot on the logs, and the information is very likely to become unavailable at unpredictable intervals.

In the long run, it's usually pretty futile to scrape in the first place. When you're stealing content just to drive traffic, you tend to have a crappy site. The only time I ever did a professional scraping app that was "justified" and "legal", the victim was another business unit within the same corporation, and we had every right to the data that they "couldn't" compile for us.

Re:You're Right, Of Course

[jimicus]

How is splitting and allowing the work to be done by someone else to do any more ethically sound than doing it yourself?

At the risk of invoking Godwin's Law so early on, how is taking this approach any different from saying to yourself "I'm just following orders"?

Re:You're Right, Of Course

[DrLang21]

Agreed. This is a case of CYA. I would also consider discussing it with HR, depending on the reputation that your HR group has for protecting internal whistle blowing activity.

Re:You're Right, Of Course

[Hognoxious]

Would it be possible to detect the scraper in real-time and redirect it to some fake/spoof data? It needn't be goatse. But it could be!

Re:You're Right, Of Course

[yincrash]

yes

Re:You're Right, Of Course

[Andy Dodd]

Maybe not in real time, but once someone detected a scraper at a given IP, they could easily change their site to feed that IP fake data instead of blocking it.

If I were in the scrapee's position, I'd probably do that because it's the best way to attack the scraper. From order of least effort on the scrapee's part to most:
1) Blocking it makes it obvious to the scraper that they've been found out, and they'll work around it, then you'll need to block them again, on and on the cat-and-mouse game goes.
2) Feeding them mostly good data but with lots of inaccurate information scattered about is nearly impossible for them to detect until it has irreparably damaged their reputation and/or caused them to make bad decisions based on the data.
3) Suing them is a pain in the butt, even more effort than 2)

Re:You're Right, Of Course

[SatanicPuppy]

Should be. It depends on what kind of data they're downloading, and whether they're just crawling link by link and hoovering up everything, or whether they're looking for something specific.

Either way, spiders and scrapers usually have programmed scan intervals which have no relation to an actual human's browsing...or they just hit the page as hard as they can, but that is so easy to block that almost no one does it that way. Even if they add a little randomness, it's only efficient to run a scraper if it's hitting every few seconds at max, and even the most ADD user won't keep that up.

Ironically, the easiest way to nail 'em is to put up a subset of "no robots" pages; if the robots crawl those pages, blacklist 'em. Every legitimate spider will respect those files.

Otherwise, if you're running a site with a ton of data, and something is crawling it sequentially, you can absolutely redirect their queries to whatever you want. I'd be wary of doing something cute (if you can call goatse "cute") for fear that you'll have an occasional false positive and redirect a user from a high bandwidth location to that site.

Re:You're Right, Of Course

[GrpA]

I've also had similar requests in the past, and in both cases I did the work. I considered the request, decided they were ethical (even if somewhat unusual) and so did it. That's something you're going to have to figure out for yourself - whether you're going to do it or not.

I've been on the other side of the fence also...

If you're relying on data for commercial use, putting yourself in a position where you need that data is a risky thing...

I had a scraper once come after me. I caught them - as the previous poster pointed out, it's easy... I didn't block them. I captured and redirected their requests so I could control what they got and, well, sent them some information that made them look really, really stupid. They were angry, but there wasn't much they could do.

They were just enthusiasts - they had no business risk in their application suddenly failing.

Let your boss know the risk he is facing and then ask him if he really wants to risk being caught and shut down unexpectedly, or worse, finding someone has poisened his data.

It's just not good for business.

GrpA

Re:You're Right, Of Course

[garcia]

So the few times I've had someone ask me to do this sort of scraping, my response is usually that sure, fine, it works, but it's very easy to spot on the logs, and the information is very likely to become unavailable at unpredictable intervals.

Depends on how you do it. I tend to use tor and a random wait time between gets to bring down the data over a few hours (up to a few days) and in one instance, because the URLs were easily guessed, I randomized the list to make it seem as if the hits were going to pages all over the place. I was never banned for any scraping activity that I have done.

In the long run, it's usually pretty futile to scrape in the first place. When you're stealing content just to drive traffic, you tend to have a crappy site. The only time I ever did a professional scraping app that was "justified" and "legal", the victim was another business unit within the same corporation, and we had every right to the data that they "couldn't" compile for us.

It's not futile. Scraping provides a plethora of information in a useful format from places that aren't willing (or unable) to provide data in the necessary format. I used scraped data of course schedule information from MnSCU to develop a weekly report that showed data about how many courses were filled at other area institutions. It was to our competitive advantage to have this information and while it was publicly available, the system wouldn't provide it to us in the DW. I used that data for a variety of different reports than I originally intended and it would not have been possible otherwise.

While I wish that the data had been provided in a better format for my use, it wasn't and that's what made scraping necessary. Plus *I* was the one who got to determine what information I was allowed to glean from the data rather than whatever the system decided was appropriate for our needs.

Re:You're Right, Of Course

[dsoltesz]

If you've gotten to the point of asking Slashdot, you know the answer: it's unethical and you need to be looking for a new job if you can't get this resolved.

These first three responses are probably all you need. Start with talking face-to-face with the boss, outline the ethical and technical problems (focus on the technical "ya know Mr. Boss, this is gonna eventually break") and propose a better solution. Follow up with e-mail summarizing the meeting (definitely document).

If you can't get the boss to buy in on a reasonable, ethical solution, then go ahead and do it his way (it's what you're paid to do) or quit (if you can afford it) documenting why you're leaving (make sure HR gets a copy of your resignation notice). Either way, look for a new job and get the hell out of there. Think positive, don't worry about the economic doomsday crap the news makes up to keep us on the edge of our seats, and don't be too proud to take a pay cut or something resembling a "demotion" (you can always work your way back up). Hell, start your own consulting business.

Re:You're Right, Of Course

[Anonymous Coward]

Something else to consider, while there is clearly no ethical question (it's unethical) if your company wants to some how provide "media" or "content" but has to steal it from other sources, how do you expect to make money?

I've seen this before, I've worked at security companies that did the same with with Nessus and Snort. Take their IP, paint it like our own, put some marketing on it and act like we created it. The whole story was that we'd just do it to get started and then start our own effort. Thing is, if that's your core business then shouldn't you actually invest in doing it? Ultimately the company in question was told to stop using nessus.org and had to negotiate some deals and they'll most likely never make a cent.

If the data is important enough to steal and important enough that you need it then you had better understand why it's not important enough for your company to invest in creating it. What's the business plan here? You'll never get better at creating it while you're dependent upon stealing it.

Re:You're Right, Of Course

[Phreakiture]

Tread wicked carefully!

Chip Salzenberg got his ass burned back in 2005 by grumbling about his employer's ethics regarding screen scraping. I heard him speak at YAPC::NA in Toronto that year, and from what he was saying, they were able to take his every legitimate action (e.g. logging in remotely to work from home) and twist it in court into something less than legit (e.g. unauthorized access). It's their word against his, and they hold the access logs. Your best bet, if you want to make a stand about the morals, is get the hell away from there first.

Re:You're Right, Of Course

[alta]

I'm for giving them entirely bogus data that would cause them to loose customers. Not sure exactly what kind of site's we're talking about, but if a customer goes looking for chicken soup recipes and ends up getting porn... I think your boss will realize that they're on to you and won't suggest stealing from them any longer.

Re:You're Right, Of Course

[cerberusss]

I'd advise against discussing it with HR. I've encountered the following situation: I talked to a HR manager about something that obviously should've remained confidential. However that same HR manager was part of the management team and thus had two hats on. She proceeded to inform the management team, to my astonishment.

I've come to the conclusion that HR is just a staff department and owes allegiance to, you guessed it, the management team. Not you.

Re:You're Right, Of Course

[mea37]

And if you ever wondered, when <insert crisis here> broke, how things could go so horribly wrong... it's because of people who think like this guy.

"Don't worry if it's the wrong thing to do; just document that it wasn't your idea!" And apparently never mind the idea of personal responsibility.

When there are no negative consequences for doing the right thing, ethics is mostly a curiosity. Ethics exist to guide you when the right path isn't easy. And yes, you are personally responsible for your own ethical behavior, regardless of whether someone with a bigger paycheck -- or even someone who signs your paycheck -- says otherwise.

Does it mean you have to walk? That depends on your boss. If you do, the best way to preserve your reputation is to avoid mud-slinging. Your current employer might want to try to harm your reputation, but it's extremely unlikely he'll get far (certainly not without exposing himself to legal liability). So just don't shoot yourself in the foot by ranting about the situation in interviews, etc.

Re:You're Right, Of Course

[Alpha830RulZ]

The OP isn't in legal jeopardy. The TOS of the site being scraped at at best a contract (if the employer has a paying agreement with them) and are just words otherwise. If the contract is being violated, the employer is completely liable for the acts of the employee.

I'd just do it. I'd point out to the boss that most sites have logging and other measures in place that may render the work product unreliable, or possibly unobtainable.

I remember someone at our company writing a scraper for for Yahoo some years back. Yahoo blocked our domain, and they had to go back to Yahoo, hat in hand, to get us allowed again. Now Yahoo has (when I last checked) a 4000/request a day limit.

You could combat this with use of proxies, but at that point, you have a case to tell the boss, "you know, if the news media found out about this, what are you going to say to them?" Normal fear should solve the problem then.

An anonymous email to the scrapee's web admin, noting that they might watch traffic from IP thus an such, might also elicit a fun little "I told you so" opportunity to the boss.

I don't think this is worth quitting over. This is just an uninformed boss, who, if the OP is adroit, may become a little more informed.

Re:You're Right, Of Course

[MindKata]

"It's just not good for business."

I find this discussion yet annother interesting insight into the (lack of) ethics of some company bosses. I've often found to my surprise, the ethics of sales people, marketing people and bosses are at times very different from that of programmers and other workers in a company. Some time ago Slashdot discussed "Ethics in IT" and its interesting how it fits with this discussion. Here's the link, it gets interesting how much it fits this discussion, once you get to the part that discusses how some bosses lack of empathy towards others...
http://slashdot.org/comments.pl?sid=448546&cid=22377570

Some bosses have contempt for other people, so considering doing this kind unethical business behaviour, is well within their usual thinking.

Re:You're Right, Of Course

[level4]

Definitely possible!

Any company with a website that contains "regularly updated data that might be interesting for competitors" has probably already got some kind of anti-scraping system in place. This guy's boss thinks he's being clever and original - of course he's not, any company with a site of any value and popularity has already seen this a million times.

What they return basically depends on the mentality of those who work there. The "by the book" professional types will just blackhole the IP or return a "too many visits from this IP" page.

Companies with a more BOFH type guy in charge might very well start "playing" with the data. Instead of the "too many visits" page you might find yourself getting a page with some of the data changed around randomly. Believe me, there are *many* people around who think it is just the height of comedy to fuck with people who are basically stealing their stuff anyway.

They will turn it into a game - and, when the erroneous data turns up on the thieving web site (if that's what this guy's company is running), a few screenshots of that site with the modified data suddenly becomes pretty good evidence in a court, if they're of the "legal remedy" persuasion.

Scraping data is a last resort, not the first thing you try. Forget the ethics - the fact he's working for a company willing to be that insanely cheap and stupid in the first place should be a signal to run far, far away in itself.

Re:You're Right, Of Course

[theaveng]

>>>The other is, for lack of better words, a "load balancing" part that requires using multiple free accounts instead of purchasing space and CPU time for less than $2,000 USD per month. The boss sees it as "distributed" computing when in reality it's "parasitic".
>>>

Can someone explain what this means? Multiple free accounts of what? Gmail? I'm confused.

Since scraping is detectable, I would follow this course of action:
- tell the boss you think "we'll get caught"
- if boss appears to want to fire you, then go ahead and do the action, but ask for him to put it in writing
- note on the order you think it's a bad idea; keep original for yourself and hand copy to boss
- write the program
-
- (optional)
- from your home computer (using an anonymous account), tell the website what your program does, and explain you would have been fired if you had not complied with your bosses' wishes, but feel it's unethical to scrap data.
- watch as Boss looks like fool when website with stolen bandwidth decides to bar his company's access
- if fired, hire lawyer and sue the company for unjustified dismissal
$ profit

Re:You're Right, Of Course

[SatanicPuppy]

Yea, but unless you're running that list across a botnet, the IP addresses are a give away.

Even if you are running it across a botnet it's pretty easy to pick out the patterns using some pretty trivial statistical hacks...If you graph bot traffic it looks like a heartbeat; even if you randomize the access times they don't match "human" numbers (unless you add so much random that it ceases to be an efficient scraper...If you could hire a guy to browse the site and write down the data faster than you can scrape it, they beat you.)

I've never actually been banned for it either, but it's all a crapshoot. I used to work for a company that did GIS data and we smote scrapers on a near-hourly basis, and that one turned freak-nasty because when we found a really good scraper, we'd feed them 60% crap data, and with GIS it's not easy to tell good data from bad.

Things like posted schedules, imho, are the real legitimate use for scrapers. Those people want their data to get out, but they may lack the tools to put it out there.

Re:You're Right, Of Course

[Chapter80]

The proper way to document this in email is something like this:

Boss-
I'm able to do the data scraping and should have it up and running by the end of the day.
- Your faithful employee

In case you are wondering about the technical details, here they are:

The scraping is implemented with a perl script which is activated using cron.

We scrape the site twenty times per minute, which is a violation of their terms of service. By doing this, of course, we risk that they may shut us off at any time, or even provide us with fake data.

The typical PHB will read the first two lines on his blackberry, and you're golden. Worst case he or she will scroll down - but the managerial brain is set to shut down at the word "perl". The word "cron" is a failsafe - in case the PHB also has ADD.

Later when s/he comes back and says "why didn't you warn me", you can point to the text "beneath the fold" of your email.

Re:You're Right, Of Course

[SatanicPuppy]

The example that would leap to my mind is a number of services that allow you to "map" an ip address to a geographic location...I use one of those for my job search homepage, and it only allows ~200 queries a day for the "free" account...It would be plenty useful to have as a free service (targeted advertising), and if you set up enough "free" accounts, you could use it that way.

Since I'm doing all my job searching away from where I'm currently living, I use mine to make sure that my job searching page always looks "under construction" for people who live where I live. My boss actually checks it occasionally, I guess to make sure I'm not trying to leave.

Re:You're Right, Of Course

[nabsltd]

Unless it's something about you, personally, then an HR employee has no requirement to keep it "confidential".

In other words, if you are talking about your health insurance, your personal information, etc., that's not general "company business" and shouldn't be spread around. But, if you bring them some information about someone doing something that could be detrimental to the business, they really do have an ethical requirement to pass that along. A good HR team would know not to bring your name into it if you are "snitching" until it was absolutely necessary, but sometimes that happens sooner than you would like.

What you were thinking of is a company ombudsman. These people are somewhat like your "lawyer" within the company, and are there for the employees. What they would do is explain to you your options (go to management yourself, let them do it and respect your anonymity as far as possible, never divulge your name even if that means the complaint can't proceed, etc.), and then help you implement them.

Re:You're Right, Of Course

[AndersOSU]

You're absolutely right. The problem is being right, like being ethical, doesn't put food on the table.

The reason crises happen is three fold, first people with power see a competitive advantage in acting unethically, second people in charge or monitoring unethical/illegal behavior aren't up to the task, and third people tasked to do the work don't raise bloody hell when asked to do anything unethical.

In order to solve the problem you only need to fix one of those. The problem is, the first two options involve convincing people to act against their personal interests. People contain a remarkable survival mechanism, the ability to justify and rationalize difficult actions. Going after people who stand to gain by acting unethically is the business equivalent of abstinence only education.

Re:You're Right, Of Course

[interiot]

The thing is, you can get away with a lot more low-level scraping than you think. If it's something where you don't need to load significantly more pages than an average surfer (you just need to repeat it several times a day), it isn't necessarily going to stick out in the logs that much. And a lot of admins just don't have the time to analyze their logs (Wikipedia allows hotlinking of their images, for instance... combined with the fact that anyone can upload any picture, this is rife for abuse. But there are better things for them to spend their time on). Also, some admins don't have the tools/skills to drill down and hilight the entries that would make it clear someone is scraping.

If you're relying on data for commercial use, putting yourself in a position where you need that data is a risky thing...

Now that I agree with. Scraping is a gamble. It's possible that an admin could spot you on your very first run, because on close inspection, your requests do look different (you don't immediately load images or CSS/JS subpages like a browser does, for one).

Since you have no idea if the site will block you on the very first fetch, or the billionth, it's not something you should rely on for business.

For personal use though, it can be very educational. There's a lot of data out there, and if you can find a novel way of analyzing it, it can be very rewarding intellectually.

Re:You're Right, Of Course

[postbigbang]

"When there are no negative consequences for doing the right thing, ethics is mostly a curiosity. Ethics exist to guide you when the right path isn't easy. And yes, you are personally responsible for your own ethical behavior, regardless of whether someone with a bigger paycheck -- or even someone who signs your paycheck -- says otherwise."

No, just because you don't get spanked doesn't mean that an ethical obligation can be ignored. Were that the case, civility would evaporate. The OP is in a tenuous position, and clearly feels the ethical breach at hand. Sleeping at night, and staying with one's own moral and ethical code takes courage. I'm hoping he/she finds a work around. Thieves are everywhere on the Internet, and scraping is just over the 'line' of cross-linking, which is nominally fair-use.

It's my opinion that if you don't want it linked, then say so or don't post/write the page. Scraping involves more issues related to copyright, which the OP says aren't involved. If they are, then it's a different legal story. Asking an employee to commit an illegal act is conspiracy. If the act has dubious or unclear ethical implications, then it needs to be documented (see above posts about throwing the boss under the wheels) and executed presuming the recourse is the corporation's, not the employee. If the employee is a contractor, then I'm guessing the contractor probably needs liability indemnification to proceed. IANAL.

Re:You're Right, Of Course

[interiot]

I really agree with this. If someone is already going to the effort of writing a lot of scraping code, it's already worth it to them to buy one of those $10-15/month shell accounts online that have SSH access. SSH gives them the ability to forward local TCP requests to that remote IP, their scraping app just has to have the ability to use a SOCKS proxy. This means scrapers have a proxy IP that 1) doesn't show up on any of the open-proxy DNSBLs, and 2) is fast and reliable enough for them to get real work done. And if you block them, they just pay another $10-15 to get another reliable IP.

Re:You're Right, Of Course

[FilterMapReduce]

from your home computer (using an anonymous account)

And an anonymous IP address through Tor or the like, just to be safe.

Re:You're Right, Of Course

[TheLink]

And could prove to be very amusing for a future slashdot submission if they encounter a BOFH.

There are just so many things that could be done.

They're planning on taking data from some site and pumping it to others and they have _ZERO_ assurance that it's going to be good data and continue to be good data.

When you do stupid stuff like this, if you're not careful very bad things could happen (SQL injection, maybe even malware slipped in) and they could just go "nope not us", and while you could try to sue them it's pretty darn hard to prove since you requested the "bomb", and it only appears once and never appears again.

If you're lucky it's just going to be goatse/tubgirl.

If you're not, it could be a lot worse. Just imagine the BOFH thinking "What should I do today to them and their users" and rubbing his hands with glee.

Just slightly tampered data will be bad enough.

Re:You're Right, Of Course

[plague3106]

The problem is that you can end up on the hook legally, if there are legal ramifications. You clearly knew it was wrong, and did it anyway.

Re:You're Right, Of Course

[level4]

Did I say I didn't approve?

I am definitely in the "bastards who find actual physical pleasure in fucking with my enemies" camp ; )

Re:You're Right, Of Course

[orclevegam]

Depends a lot on how they're doing the scraping. It's not terribly hard to write a scrapper that fairly realistically duplicates human behavior, although as you pointed out if they're using it to feed their own processes it does put some demands on how often etc. it's forced to run which could make it stand out from normal activity. Of course, given 20 or so of these bots all scraping from different IPs, so long as you balanced their duty cycles so they were all offset from each other you could have scraping going on for 24 hours without ever deviating from normal browsing patterns. The downside to something like that though is that it requires a certain amount of insight into the layout of the site, you can't just randomly follow every link on a page as that's waaaay too obvious, so if the site layout changes it can break your scrapper(s) until a dev can sit down and update them.

In other words, it's totally doable, and even in a "undetectable" way, but it's fragile, a total pain in the butt, and overall just not worth the headache. Just pay the damn company for proper access to the data and be done with it, it'll be a whole hell of a lot simpler. If your boss doesn't understand that then he's a moron, get out now before you have to do something that will end up on thedailywtf.com, much to your eternal shame I'm sure. Better yet, get out now, and then submit the details of the project you were asked to make to thedailywtf.com, I'd love to read exactly what "genius" ideas this guy has come up with to save a bit of money in one area by paying a bunch of money in another.

Re:You're Right, Of Course

[Clover_Kicker]

Yeah, subtly wrong data is a million times worse then goatse, the scraper might not notice for weeks or months...

Re:You're Right, Of Course

[orclevegam]

Believe me, there are *many* people around who think it is just the height of comedy to fuck with people who are basically stealing their stuff anyway.

You say that like it's a bad thing. Now where did I put my cattlepro... I mean cable tester.

Scraping data is a last resort, not the first thing you try. Forget the ethics - the fact he's working for a company willing to be that insanely cheap and stupid in the first place should be a signal to run far, far away in itself.

Seconded. I used to think my managers were daft, then I started reading thedailywtf.com and I gained a much greater appreciation of exactly how bad things can actually be. From the description this guy gives, he's definitely dealing with someone well on his way to ending up on that site.

Re:You're Right, Of Course

[onepoint]

I have a site which I get paid very well to manage under contract. I have traps all over the place.
one of my traps is for email scrapers ( php script that loads about 5000 names randomly over 50 pages )
another page I love is my "crap" page, when an ip hit's 20 times in 2 minutes, it loads up and ask if you are a human or a computer, humans end up on the recaptcha page and on there merry surfing way, computers end up in crap section, which is 100% non real data at bargain prices and very specific key word phrases that I can track down. causes more harm, since it also loads porn pictures and goat se and a few other ugly images. it works rather evilly well.

got a few others, but that's the basics.

if you want to read more about bot blocking go to incredibill page, that will help you alot

Re:You're Right, Of Course

[Lumpy]

(4) you mention the next morning "what the hell happened to your car? did you run off the road? the side is all scraped up."

(5) days later, loosen the cores of all the tires scrader valves, replace the caps loosely as well.

(6) subscribe the PHB's email address to every porn site you can find.

(7) put ad in local alternative magazine for boss looking for homosexual urban tantric partners, give the main switchboard and home numbers.

PHB's are very easy to deal with.

Re:You're Right, Of Course

[Misch]

It's happened. ESPN connived a way to get to another sites private database and reported the data as its own. The website injected some fake data which ESPN picked up and reported and were caught.

Re:You're Right, Of Course

[level4]

Troll?! What part of that comment constitutes any kind of troll?

Please save your mod points for my real trolls! You won't have to wait long, I've just opened my 4th Asahi Super Dry ...

Re:You're Right, Of Course

[interiot]

So a dedicated scraper would change IPs, write some code to detect and avoid the potholes, and then resume scraping.

There are lots of decent ways to detect scrapers or hotlinkers. But I haven't seen any idea yet from either side (web admins, or scrapers/hotlinkers) that can't be bypassed with enough work. It really seems like it's something of an arms race.

But the arms race hasn't progressed very far, even for attractive targets with some amount of money (porn sites), because it's just a lot of work for both sides.

If you want legal advice...

[Anonymous Coward]

...ask a lawyer.

Re:If you want legal advice...

[dintech]

No, no, no. If you want legal advice, ask Slashdot! Given enough time, you'll get an answer that is exactly what you hoped for and you can ignore or mod down the ones you don't like. It couldn't be simpler and it's a whole lot cheaper!

Just ask yourself this:

[Anonymous Coward]

Did the contractors on the Death Star deserve to die?

Re:Just ask yourself this:

[Intron]

"Did the contractors on the Death Star deserve to die?"

Depends on whether it was the ones that did the weapons array or the ones that did the low-flush toilets. Oh wait, Halliburton did both.

Re:Just ask yourself this:

[Quince alPillan]

Because there have been a couple people who don't understand the reference from Clerks...

Randal: So they build another Death Star, right?
Dante: Yeah.
Randal: Now the first one they built was completed and fully operational before the Rebels destroyed it.
Dante: Luke blew it up. Give credit where it's due.
Randal: And the second one was still being built when they blew it up.
Dante: Compliments of Lando Calrissian.
Randal: Something just never sat right with me the second time they destroyed it. I could never put my finger on it-something just wasn't right.
Dante: And you figured it out?
Randal: Well, the thing is, the first Death Star was manned by the Imperial army-storm troopers, dignitaries- the only people onboard were Imperials.
Dante: Basically.
Randal: So when they blew it up, no prob. Evil is punished.
Dante: And the second time around...?
Randal: The second time around, it wasn't even finished yet. They were still under construction.
Dante: So?
Randal: A construction job of that magnitude would require a helluva lot more manpower than the Imperial army had to offer. I'll bet there were independent contractors working on that thing: plumbers, aluminum siders, roofers.
Dante: Not just Imperials, is what you're getting at.
Randal: Exactly. In order to get it built quickly and quietly they'd hire anybody who could do the job. Do you think the average storm trooper knows how to install a toilet main? All they know is killing and white uniforms.
Dante: All right, so even if independent contractors are working on the Death Star, why are you uneasy with its destruction?
Randal: All those innocent contractors hired to do a job were killed- casualties of a war they had nothing to do with. (notices Dante's confusion) All right, look-you're a roofer, and some juicy government contract comes your way; you got the wife and kids and the two-story in suburbia-this is a government contract, which means all sorts of benefits. All of a sudden these left-wing militants blast you with lasers and wipe out everyone within a three-mile radius. You didn't ask for that. You have no personal politics. You're just trying to scrape out a living.

(The Blue-Collar Man (Thomas Burke) joins them.)

Blue-Collar Man: Excuse me. I don't mean to interrupt, but what were you talking about?
Randal: The ending of Return of the Jedi.
Dante: My friend is trying to convince me that any contractors working on the uncompleted Death Star were innocent victims when the space station was destroyed by the rebels.
Blue-Collar Man: Well, I'm a contractor myself. I'm a roofer... (digs into pocket and produces business card) Dunn and Reddy Home Improvements. And speaking as a roofer, I can say that a roofer's personal politics come heavily into play when choosing jobs.
Randal: Like when?
Blue-Collar Man: Three months ago I was offered a job up in the hills. A beautiful house with tons of property. It was a simple reshingling job, but I was told that if it was finished within a day, my price would be doubled. Then I realized whose house it was.
Dante: Whose house was it?
Blue-Collar Man: Dominick Bambino's.
Randal: "Babyface" Bambino? The gangster?
Blue-Collar Man: The same. The money was right, but the risk was too big. I knew who he was, and based on that, I passed the job on to a friend of mine.
Dante: Based on personal politics.
Blue-Collar Man: Right. And that week, the Foresci family put a hit on Babyface's house. My friend was shot and killed. He wasn't even finished shingling.
Randal: No way!
Blue-Collar Man: (paying for coffee) I'm alive because I knew there were risks involved taking on that particular client. My friend wasn't so lucky. (pauses to reflect) You know, any contractor willing to work on that Death Star knew the risks. If they were killed, it was their own fault. A roofer listens to this... (taps his heart) not his wallet.

Re:Just ask yourself this:

[Rennt]

Haven't you seen "Clerks"

Dante: My friend is trying to convince me that any contractors working on the uncompleted Death Star were innocent victims when the space station was destroyed by the rebels.
Blue-Collar Man: Well, I'm a contractor myself. I'm a roofer... (digs into pocket and produces business card) Dunn and Reddy Home Improvements. And speaking as a roofer, I can say that a roofer's personal politics come heavily into play when choosing jobs.
Randal: Like when?
Blue-Collar Man: Three months ago I was offered a job up in the hills. A beautiful house with tons of property. It was a simple reshingling job, but I was told that if it was finished within a day, my price would be doubled. Then I realized whose house it was.
Dante: Whose house was it?
Blue-Collar Man: Dominick Bambino's.
Randal: "Babyface" Bambino? The gangster?
Blue-Collar Man: The same. The money was right, but the risk was too big. I knew who he was, and based on that, I passed the job on to a friend of mine.
Dante: Based on personal politics.
Blue-Collar Man: Right. And that week, the Foresci family put a hit on Babyface's house. My friend was shot and killed. He wasn't even finished shingling.
Randal: No way!
Blue-Collar Man: (paying for coffee) I'm alive because I knew there were risks involved taking on that particular client. My friend wasn't so lucky. (pauses to reflect) You know, any contractor willing to work on that Death Star knew the risks. If they were killed, it was their own fault. A roofer listens to this... (taps his heart) not his wallet.

Kevin Smith knows his stuff.

Hilarity ensues when...

[Anonymous Coward]

...you build a system that closely relies on this nonstandard (and unsupported) method of getting information, they change it and it breaks.

Either by accident, or because they spot a load of particular access patterns from your address, figure out what's going on and intentionally break it.

Re:Hilarity ensues when...

[Paeva]

I would think this would be a good way to address the issue with your boss. He wants to save some money to get, as he thinks, the same thing for free. But in fact, there are potential downsides to playing that game. He may be disregarding potential legal issues, but he should be less willing to disregard practical issues. If this other company discovers what you're doing, they could make it a little harder to access, or they could ban your company's entire subnet and send a letter indicating that if you'd like to get access again, then you'll have to start paying them for the service you've been stealing.

The key is that, in the meantime, your boss' plan will seem like a dramatic failure that should have been foreseen.

Re:Hilarity ensues when...

[Sockatume]

Workaround: write a program which generates random data using a small amount of harvested data as a guideline. If the boss is too lazy to generate the data he's meant to be generating, then he's probably going to be too lazy to check that the data you're "harvesting" is actually accurate.

Re:Hilarity ensues when...

[SatanicPuppy]

That's pretty common. John McCain had an issue with that earlier in his campaign when his MySpace page got hit. The guy who did the original template wasn't keen on having his images hotlinked from such a high volume site and made a hilarious substitution (which was widely misreported as a "hacking" incident in the media).

The AC is dead on. If you depend on someone elses data, they are going to notice, and they are going to remove your access, or, worse, start feeding you crap.

Short answer... "no".

[argent]

If your boss asks you to do something illegal, don't. If he doesn't agree, you should probably be looking for a new job, already. If he's willing to play these kinds of games with another company, what makes you think he won't do the same to you?

Uh...

[Anonymous Coward]

No. By your own admission you think its wrong. Next?

Sigh

[MyLongNickName]

Okay, this one is simple. You know what is right and what is wrong. The reality is that 99% of the folks will do what the boss asks without even raising a fuss. The reality is that you will be damaging your career if you don't go ahead.

Now, the other reality is that shit flows downhill. That is, if this project gets questioned, the boss will claim ignorance, and put the blame on you. Your job is to cover your ass.

Email is a good documentation tool. "Clarify" the request, asking if this is what he intends for you to do. Remove the emotion. Put in only facts. Put in a piece about your not being sure, but this may be a violation of terms of service. Ask if he wants you to proceed. Forward your sent email to a personal account.

By the book. This one is so simple that it should be in the FAQ.

Re:Sigh

[MyLongNickName]

I've worked in a large company, who's name I can't reveal for fear of litigation but essentially, using email in a CYA fashion would get you fired. (Terms of contract, they can end the contract at any time for any reason, but the money was good)

Bullshit. What do you put in the subject line "This is a cover my ass email"? You are only clarifying what the boss is asking for. It is basically to be used if ever there any question about what you were told to do. There is not such thing as a "you can't cover your ass in an email policy". The only thing that could be prohibited is forwarding to a personal email address. If that is the case, print the sucker out.

Re:Sigh

[sydb]

It's only any good if the other party co-operates. The boss can easily phone you or walk up to you and say "Yes I want you to do it." and you have no record, and for many people this is their default mode of operation because that way no-one can pin anything on them. Unless they're singing their own praises, when everyone gets cc'd in.

I used to find it infuriating but fury gets you nowhere in the workplace.

Re:Sigh

[realisticradical]

Really? Were you working for the mafia?

That kind of policy makes me think of companies like Enron, Arthur Anderson, or the cigarette companies. Any company that would fire people for trying to protect themselves from company sponsored illegal or unethical activity must be engaging in lots and lots of it. The best part it it sounds like the policy is simply a CYA policy to protect those at the top.

The part I don't understand though is how did they differentiate between CYA emails and actual questions about projects? If, "Sorry boss, I was a bit confused at the meeting we concluded that I should go about the project by [illegal activity]" gets you fired how did they ever get anything done?

Re:Sigh

[mollymoo]

You must have some pretty weak employment laws where you live.

Re:Sigh

[MyLongNickName]

You bring up a good point which leads to lesson #2: Written trumps verbal. If shit hits the fan, you halve your email. if your boss then says that he verbally told you not to proceed, you only have to say that you have no recollection of any such conversation. He is on the defensive as he has nothing to back it up. If he was "appalled" at the thought of breaking the TOS, then he would have written back and clarified.

Now, if you want to double cover your ass, give him status reports via email. Ask questions. You are covered.

Now to answer some other questions about whether to quit or not. You have to make that decision on your own. For screen scraping, I wouldn't quit over something so mundane. Sorry. Especially if you are a grunt. You voice your concerns, and go on. The reality is that 4 times out of 5 if you voice your concerns like this in a written manner, that the boss will back down. I have faced it twice in a grunt position with two different managers, and both times I got thanked for bringing it to their attention. It is all in how you deliver it. If it comes across as "I am ethical and you are a piece of shit", then your career is hurt. If it comes across sa "I am trying to look out for your well being and that of the company", it can be a positive. Wording is everything.

Re:Sigh

[MyLongNickName]

On a similar note, email is great for lifting others up. A few years ago, I got promoted to a mid-level grunt, and worked a lot more with other business units. In big organizations, units don't play nice with other units. On the few occasions, someone went out of their way to provide good service and actually be helpful, their boss got an email letting them know. Maybe two or three emails a year, but those folks are the ones who are going to get the bigger raise and have chances at promotions.

Unfortunately, 95% of all emails about people are complaints. Do your job well, and the best you can hope for is to be ignored. I have personally tried to reverse that now that I am a low-level manager. I can't change the world, but I can influence my dusty corner of it.

Re:Sigh

[VShael]

No, not bullshit.
This is explained to all new hires. Clarifications are to be done face to face. Not over the phone, not over email.
Email is considered a completely permanent record (even if it's not actually permanent in practice) and would be the equivalent of having a politician speak to a journalist on the record.

ie. you don't ever do it unless you have to, and then, you clear it with the press office first.

Well, the company I'm talking about was exactly like that. (But not politicians and journalists.)

Spammer logic.

[argent]

If you can access it, it was designed to be accessed.

So you're totally behind email spam, you don't think spam should be considered unethical, let alone made illegal?

Re:Spammer logic.

[BadAnalogyGuy]

Should I be against spam for any other reason than I am annoyed by them?

I don't think spam should be any more illegal than billboards, flyers, or direct mailings.

Re:Spammer logic.

[d3ac0n]

I don't think spam should be any more illegal than billboards, flyers, or direct mailings.

The flaw in this argument is that your three counter-examples (Billboards, Flyers and Direct mailings) are paid for entirely by the SENDER. IE: Billboards are paid for up-front before they are mounted, Flyers and direct mailings have printing costs paid up-front and delivery costs (either the local govt. mail service or paid people to manually give it to you) paid up-front as well.

Spam, on the other hand, is largely delivered on the backs of OTHER payers. Both through the incredibly high bandwidth costs (HOW much of the total Internet traffic is Spam now?) and through ancillary costs such as costs for software and hardware to filter Spam out, and human costs in terms of work-hours wasted manually going through spam. Not to mention the costs to people and networks infected with Spam botnets.

This is what makes Spamming SO profitable, and why it won't go away. Because the costs for Spam are decentralized to millions of people otherwise not directly involved, even a return as little as .01% will turn a HUGE profit. This just doesn't work in the regular advertising world. It's also why it's Illegal is several countries now. It is essentially stealing service from millions of other people and generating millions of dollars of expense for hundreds of companies around the globe, for what are largely scamming and phishing operations.

Get it?

Good.

Check with Compliance Officer/Department

[Ohmaar]

I work in health care, so maybe it's different in your industry, but every hospital I've worked for has had a compliance officer with an anonymous 800-number for compliance questions. This is DEFINITELY the kind of stuff they want to know about.

Do what geeks do best

[GreyyGuy]

Fix it. He wants to do something on the cheap and look good. But the way he wants to do it is going to fail spectacularly. And when it fails, so will you. If this puts any amount of load on the services it is using, it will get picked up by the service provider. Maybe not today, but it will. And then the accounts will get turned off and possibly your IP addresses blacklisted, and then it all goes away. So give him a better solution. If he is balking at the $2k/month find a cheaper service. There is almost always one. Compare the cheaper solution to the time spent fixing it when the free service cuts you off. Provide examples of free service cutting people off.

And unless you are looking for some very specific information, I would expect someone to provide an RSS feed with something similar that is supposed to be used for this sort of thing.

Re:Anything on the web is available for access

[jthill]

By those rules, taking candy from a baby isn't unethical.

Why are you asking *US*?

[elrous0]

Only YOU can decide how far you're willing to go for your job. You're essentially asking us what your own ethical limits are.

Business sense

[ThePyro]

Even if your boss doesn't care about the ethics of this scheme, he probably does care about ramifications to the business. What happens when you get caught? All your development work will have been wasted because they'll shut you down at the very least. There's potential for a lawsuit, which is an expensive proposition even if you win. Damage to your company's reputation may make it harder to do business. And as another poster already mentioned, this isn't exactly a gem of a project to put on your resume.

Re:Anything on the web is available for access

[BadAnalogyGuy]

Babies really shouldn't be given candy in the first place.

A character check?

[juuri]

Having been put in a position once before that an employer asked me to do something I found to be frankly quite lacking in a moral nature here's what I ultimately decided to do.

After considering the work for a while, both why I didn't feel like performing the work personally and why the company desired this functionality I finally decided to do the work, but inform my boss and his boss that I was uncomfortable creating this before hand and giving them clear notice of the whys.

Firstly I did the work because it was simply my job and I had signed onto the job. It's something a *lot* of people might not have given a second thought to creating, obviously as they both had no problems with the work since they asked me to continue even after raising my concerns. Secondly because it wasn't really "that bad" and having steady income of cash dolladolla bills allows me to have nice things like somewhere to live and food I wanted to see if it was something I was over-reacting to.

After completion? Yep, I still felt like shit. So I gave them my notice and told them in the my resignation letter why I was leaving and referred them to the early notification of my objections. So, for me, it was a good learning experience about myself and having done it in this manner I have no problem explaining it to future employers as my reason for leaving this particular job.

Who cares?

[1u3hr]

would require me to break multiple web sites' Terms of Service (TOS).

A website's "terms of service" are not the Ten Commandments. They're not laws, or even moral rules. They're just what one company wants you to do. You don't work for them, why do you care? If they notice and complain, it's your boss's problem, legally; and morally, I wouldn't lose any sleep.

Only thing to do is cover your ass and get your boss to put his instructions in a memo so he can't blame you should problems arise.

Really "scraping a website" is not a moral question on the scale of collaborating with Nazis. It's a business. Other businesses are your rivals, not your friends. They'd fuck you over in a minute.

Re:Who cares?

[MightyYar]

I wrote a little script to search multiple cities in Craigslist, simply because they don't offer the function at any price. People can say I'm a jerk, but I really don't care because it saves me a lot of time.

Re:Who cares?

[Courageous]

Planning ahead of time to breach a contract, with malice aforethought, may not be as free of moral constraint as your letting on.

C//

Re:Who cares?

[1u3hr]

Planning ahead of time to breach a contract,

What "contract"? No contract was mentioned.

Re:Who cares?

[1u3hr]

representative of what the populace thinks about the issue

If you read the instructions to moderators, that is not at all what moderation is supposed to do. It's supposed to highlight posts worth reading, and push ones not so out of sight. It's not meant to be a "poll" on "what the populace thinks". Otherwise every "me too" on a subject you agreed with, or "fuck you" on one you didn't would be modded up.

And if a poll was wanted, why limit it to the few moderators?

Hey, anonymous! This is your boss.

[jollyreaper]

I told you to scrape Slashdot, not read it. Now get back to work!

one approach

[buddyglass]

1. Tell your boss it's a bad idea to break these websites' terms of service. He'll probably override you and tell you to do the project anyway.
2. Code up the project just like he asks. Demonstrate that it works.
3. Shortly afterwards, email the sites in question from a non-work friend's account and let them know (with specific information) the accounts and IP addresses that are violating their terms of service. Hopefully the accounts will be disabled, and/or your employer's IP range will be blocked.
4. Throw up your hands and tell your boss, "Well, I guess they figured out what we were doing!"

Re:one approach

[uneek]

1. Tell your boss it's a bad idea to break these websites' terms of service. He'll probably override you and tell you to do the project anyway.
2. Code up the project just like he asks. Demonstrate that it works.
3. Shortly afterwards, email the sites in question from a non-work friend's account and let them know (with specific information) the accounts and IP addresses that are violating their terms of service. Hopefully the accounts will be disabled, and/or your employer's IP range will be blocked.
4. Throw up your hands and tell your boss, "Well, I guess they figured out what we were doing!"

Thats pretty stupid.

It doesn't solve the ethical or technical problem.

It worsens his / her relationship with the boss.

Re:one approach

[gEvil (beta)]

The boss will just say "You're a smart guy. Find a way to get around their protections."

Re:one approach

[DingerX]

No need to rat them out. Just give the boss the 411 on the "Hidden Costs" of doing things that way. Ethical arguments are well and good, but when you're asked to do something like this, it's clear that the ethical arguments mean nothing compared to economic ones. Guaranteed system-wide outages and worse catastrophic failures (=poisoned data) are going to cost a lot. Since you cannot predict when they will happen (only that they will happen), or what they will look like, you can't give an estimate for the downtime while you develop counter-countermeasures; nor can you guarantee that those counter-countermeasures will succeed for more than a brief period (especially once they're "on to you").

The fact is, what he's proposing will be more expensive and disruptive than doing it the legit way. And it is your job to point that out. You can also then point out that while you personally consider it unethical, the industry-standard reaction does not usually involve a lawsuit, but rather to deploy simple countermeasures that disrupt and embarass the amateurs stupid enough to perpetrate it.

CYA by Asking!

[cliffiecee]

The whole idea sounds pretty scummy, based on your description. Multiple free accounts? yeesh.

So why don't you just ask the webmasters of the sites you're about to scrape? I'd bet the site owners would settle for a few hundred per month to provide you with data in whatever form you require. And it's cheaper than the $2000/mo. for a server, etc. (If these sites are "bigger" than what a few hundred a month would buy, then you damn well better ask (see below).

Ask your Legal department about this as well. They can be extremely helpful in stopping hare-brained ideas like this. If the websites in question are big enough to take action against this, YOU'RE the one left holding the bag, not Mr. Bright Idea Guy.

WARNING: All of this assumes your boss is partially sane and reasonable!! If he's a jerk, you are hosed. I'm sorry.

If you even need to ask....

[jimicus]

If you even need to ask, you've already demonstrated a trace of ethics.

Now, sometimes having such ethics will mean you have to make difficult choices. And nobody else can make those choices for you.

While ethics won't pay the mortgage, "Reason for leaving the previous job: I was asked to do something illegal and, when I queried this, was given the ultimatum to do it or get out. I got out." is probably a heck of a lot better than "The company had to sack me after it transpired I'd done something illegal" (emails to CYA notwithstanding).

Because, make no mistake, the fact that your company has done this will get out.

Re:It's your job...

[I cant believe its n]

I don't have my copy of the ACM code of ethics

Well, look no further: The ACM Code of Ethics

Some sections relating to this issue would be:
1.1 Contribute to society (and human well-being.)
1.2 Avoid harm to others.
1.3 Be honest and trustworthy.
1.5 Honor property rights (including copyrights and patent.)
1.6 Give proper credit for intellectual property.
1.8 Honor confidentiality.

Re:Really?

[j-pimp]

You can quit, whiner! If my boss asked me to rob a liqueur store, I wouldn't conduct a poll on the police fraternity league website first. I would quit and then report him.

I would report him and then ask the police if I should quit. They might want a mole.

Hahaha hahahaha hahaha!

[EWAdams]

"Compliance officer" in an IT business... you crack me up. You should take your show on the road.

Hospitals have compliance officers because a) they're regulated, inspected, etc. and b) people can die and they can be sued to Kingdom Come.

The IT business is about as regulated as Somalia.

Re:Anything on the web is available for access

[megamerican]

Babies really shouldn't be given candy in the first place.

Then taking candy from a baby IS the ethical thing to do!

Re:Anything on the web is available for access

[MightyYar]

Mmmmmmm... ass burgers.

Tell the boss it won't work, give reasons why.

[mark-t]

Tell him that the very next-to-best case scenario for him (the "best case" scenario being that they never notice what you are doing) is that they notice what you are doing and blacklist you from connecting to it ever again. If at all possible, give him an estimate on the likelihood of that occurring. Point out to him very plainly that if or when this outcome occurs, then what he is asking you to do now will be all for nothing. If the chance of legal ramifications is not negligible, you should also mention that as well. Document everything. If he still wants you to proceed, then polish your resume and find another job because if he's too cheap to pay 2k a month for a service he thinks he can scam off of for free, he's probably too cheap to want to continue to pay you in a few months time, after he figures he's got what he needs from you.

Re:Why have an ethics dilemma at all?

[ledow]

This is a shortsighted view of the problem here.

"You're getting paid to do a job, and you're not going to be personally liable should anything go wrong anyway."

Incorrect. His boss isn't breaking the Terms of Service, he is. When the website in question terminates their access, guess who's gonna get the flak? The person who *implemented* the system, not the one who designed/thought of it, especially if they are non-technical and rely on lower-order technical beings to do things for them.

Take, for example, a situation that I regularly come across:

Boss: "It's okay, we'll just copy all these Microsoft CD's and save a fortune on licensing."
Boss's Boss: "Okay. You know best."
Boss (to underling): "Copy these CD's"
Underling in theory: "Okay". Underling in practice: "We *can't* do that."

When things go wrong, the underling in theory is going to get the blame here, because it's his area of expertise and he *wrote* the system that does it. I get people suggest to me all the time that we could just install another license of Office that we don't own, or we can just copy CD's that have blatant copyright notices on them, or breach a Data Protection Act directive by doing X, or a million and one other things that I *know* we can't do. The people in charge of me barely understand the terms, let alone whether what they are doing is illegal. I have to sit and explain to my boss and my boss's boss why we can't do them. Trust me, if something got noticed, Underling in Theory would get sacked/sued every time.

"Are you really going to walk out of your job over violating the terms of service of a few web sites?"

Why not? I get asked to do all sorts of crap and I point it out and say no. If I *chose* to do it instead, then it's a different matter. But when I *refuse* to do something on legal or ethical grounds (we're not just talking ethics here - it also sounds like they have a "subscription" of some kind to the data that they are scraping, or that it's a competitors website) then if you *make* me, I will walk (been there, done that - I've turned down a good career move and more money in order to sleep at night - not that I was being asked to break the law, not that I was being asked to sell my children, but that I was being asked to do things that I didn't agree with [wasting money within a school on useless IT cruft and consultants while the kids didn't have books or paper]). I'll also report you to the BSA or whatever organisation I need to if you really press me, or the local press like I did in the above case (they didn't do anything with it, but I breathed a sigh of relief once I'd sent off the information to them - my part was done and I'd done good by myself - if the press decide to sit on something, that's on *their* conscience, not mine). You don't do illegal stuff if you're honest and your mortgage depends on a wage.

"It's not your job to worry about the ethics of the situation, that's probably not even your boss's job -- it's somewhere in your corporate legal department, the Board, or an Ethics or HR department perhaps."

Wrong. Because they won't even *know* what the problem is until it comes up in court and they have it explained to them in excruciating detail. However, someone who decides to do something that's part of their job, within their area of expertise and breaks a law (or even does something a bit stupid) that *they* should know about will get fired/sued by their own company once the shit hits the fan. So your boss *and* you might get sacked - you're still no better off and your employment reference is now a million times worse.

"just do what you're being paid to do and ask fewer questions."

It's sad that people think this is a good way to live. He's *being paid* to do his job. Which does not entail questioning his ethics or breaking Terms of Service (even if legally unenforceable) or anything else. His *job* is to stand up and say "Whoa, hold on, we can't do that". If he doesn't do that, he's not doing his job an

Re:No dilemma

[ledow]

Copyright. The copyright holder has the right to do pretty much what they want with their own data. If that means putting up a notice that says "free to view, pay if you want to download", they can do that. Copy an image from a website and then upload it somewhere else, or put it in an advert, or print it out and stick it on your office wall. Chances are you just broke copyright law. You can't do this with anything copyrighted, no matter how easy it is to "technically" do it. Dilbert cartoons, youtube vids, Slashdot comments, it doesn't matter. If it's copyrighted, you *can't* do this.

Leading question

[bWareiWare.co.uk]

You asked the question in a leading manor and have got odd responses as a result:

'Scrapping' pages is exactly what the Internet archive or Goggle do, this is common and generally accepted practice (look at the amount spend on SEO). It is also assumed that these operate without human supervision and do not need to read or compile with the human TOS of your site. Critically spiders should compile with the 'robots.txt'. If you do this you have the moral high ground. If you don't then it can be interoperated as criminal under the laws such as the Computer Misuse Act.

Similarly no one suggests that everyone using gMail is a parasite. Most 'free' services come with a very explicit contract detailing their allowed uses. If you compile with the contract you are fine, if not, you are again breaking the law.

Probably more importantly, this is almost certainly a bad business discussion:

Given that you as an employee have judged it as ethically questionable you can be fairly sure a significant proportion of your clients are likely to feel similarly.

Even if you are complying with the contract from your free service you are almost certainly not getting a SLA in return. If the supplier decides your business is dodgy, or you are putting too much burden on their system they will shut down all of your accounts without warning or reprieve. Constantly battling this is likely to cost you more then the hosting in the long run.

Page scrapping is very unreliable. Even when the source site is cooperating they invariable break it on every edit. What will happen to your business when the source site detects your scrapping and decides to serve goatse to your spider, and hence your clients?

Terrible engineering

[Have Blue]

Even if you don't want to tangle with the ethical issues, ask your boss how he feels about the app constantly going down and losing data because the "parasited" service deleted all your free accounts.

Re:Redirecting content

[Mr. Droopy Drawers]

Reminds me of a time when an Ebay'er was pointing to images on my website for an automotive auction. Didn't ask us or give us credit for the images. So, his example of "recently restored examples" became a photo of a '63 Imperial being loaded into a crusher.

How's that for Crushing the Competition?!

Re:Shit Falls Downhill

[whisper_jeff]

...scraping is extremely delicate and very easy to thwart.

Then that's what he needs to tell his boss (and I agree with you). He needs to clearly inform his boss that it is probably illegal and opens his company up to (expensive) litigation and, more importantly, even if it doesn't get to litigation, the source site could make a change that renders their scraping efforts null and void. It needs to be put in a dollars-and-cents picture so that the boss realizes that the best (and only) solution is to pay the licensing fee. Doing otherwise will likely be more expensive and inconvenient. Any other depiction of the scenario won't matter to a boss that is only concerned with the bottom line.

And, if that doesn't work, polish your resume.

Re:Mild Flamebait

[Anonymous Coward]

...I'm a bit amused at the sudden vehemence of the Slashdotters who commonly decry all DRM and all attempts by copyright holders to protect their IP. I would have thought the community would have come down on the other side of this issue, but I guess music and games are different from websites, photos, and other scrapable data.

I can't speak for the rest of /., but for me, these two issues are nowhere near the same. I'm against DRM because I feel that it's right that once I buy (for example) a DVD, it's mine. I am entitled to copy it for backups, rip it to watch on my computer, or do whatever I please, so long as it is for personal use. I do think piracy is wrong and do not advocate it in any way, shape, or form.

If I paid someone to write a bunch of content for my site and then lost the password to the server, I wouldn't think twice about scraping the content back to myself.

I was in a situation just like this

[viridari]

My employer asked me to do something that was unethical, and likely illegal. I asked to hold off on implementation until we could consult company counsel on the legality of it. Boss and director said "No. Do it. Now." I made my case, said I'd be happy to keep working there, or not, but I'm not going to do what they're asking me to do in this case.

The next day I got my walking papers. I felt more liberated than upset.

I've now worked for two scumbag marketing companies and I'm thinking it's probably best, if you have a conscience, to avoid them like the plague.

no solution required.

[nimbius]

if i were the "website" you're scraping i find it hard to believe it would go unnoticed.
I'll warn you once or twice about it, then over the next weekend
create something nice in my OSS webserver that replaces your
scraped content with pro-taliban rhetoric and dancing goatseman.

I'll then forward all of your frantic phonecalls to my FOSS astycrapper.

Re:Mild Flamebait

[Tjebbe]

yes, DRM is not copyright enforcement, DRM is copyright evasion (the producing party circumvents the copyright law in order to be more restrictive than the law entitles him to)

Re:Redirecting content

[Free the Cowards]

Somebody once pointed at a picture of a frosted birthday cake on my web site from a forum. So I grabbed my image editor and built a special edition of the cake just for him, where the frosting read "Don't link to my images!"

I also have a specially crafted JPEG which is under 1000 bytes but which produces a 20,000x20,000 pixel image filled with black. It will totally screw up the layout of any page linking to it if they haven't entered an explicit size for the tag.

Do it, if he's willing to "help you out"

[JavaRob]

Your employer doesn't have the right to ask you to place yourself in legal jeopardy in this way, and if the sh1t hits the fan do you really think that someone that came up with this scheme will balk at placing all the blame on you.

Absolutely. That's why you should agree to do the work, but because of the increased risk to yourself, you should ask for a "little something extra" under the table, just between you and him. A wad of hundred dollar bills passed discretely in a handshake, for example. "I help you, boss, you help me?" is a good phrase to clue him in on the situation and what's required for the project to continue. ...or perhaps he may rethink how he wants his workplace to operate?

Waste of money

[vw_bob]

My take on this is that, though your assignment has spawned an ethical question, the reality of the situation is economic.

Your boss believes that it will cost him less to "scrape" data from the website and use multiple free accounts than to simply pay for the data access. This may be true at first, but, ultimately, this is false.

On the off chance you've not scraped websites before, I'll tell you that this is extremely error prone. So, while this may work initially, you'll be constantly chasing down bugs in the process.

Based on your description I assume you'd be automatically logging into their systems before scraping the data. What if their login process changes? What if they restructure their website? What if they add a captcha to the registration and login process?

My point is, what your boss wants to do is, to use Steve Job's recent phrase, a bag of hurt.

I'll bet that given enough time, the cost to your company in terms of your salary to build and maintain this application will be greater than the cost to actually pay for the data and create a dependable connection.

Don't forget to factor in what it costs you when your users see bad data or error messages due to the process breaking.

It'll cost less to do it the right way. So forget the ethics of it and educate your boss on the economics of it.

From the other side of the fence

[Dues]

As someone who works for a web company that is full of high-demand listing information, we are constantly having to deal with this kind of activity. Any site with dynamic data that is in demand will already have a procedure to deal with the kind of activity you are driving to it, as they have almost certainly dealt with it before. In our case, we have a network appliance that can detect if you are a scraper based on your traffic, and then serve you a "you've been throttled" page. In short, your script will be an exercise in futility. Here is a better solution for your company - contact the source of the data, and offer to purchase a feed from them. Chances our they will be willing to comply to keep you from loading their website with stupid traffic. You will end up having to pay money, but at least your process won't break. You should suggest this to your boss at least. A side note: your peers will definitely frown on your actions. If I knew a person behind this kind of activity they'd get branded scarlet-letter-style.

Re:Sure fire way to NOT get hired anywhere else...

[rjh]

You are an employee of a corporate: protected by its immunity.

Employees have no immunity from felony prosecution. Bang, period, end of sentence. There is no immunity anywhere in corporate life. There is indemnity, which is a separate issue, which protects you from civil lawsuit; it does not protect you in any way from felony charges.

If you commit a felony and the police come knocking, expect to get charged. The corporation won't be.

And if you really think that protecting your company from a lawsuit and reporting possibly felonious actions to the company's legal department will get you not hired anywhere else, you really need to spend more time in corporate America. This is the way you handle these things. You don't involve the police and you don't go to the press. You go to the corporation's own internal hierarchy and say "my manager is doing something aggressively stupid which, if discovered, will get me in a ton of trouble and expose the company to massive financial risk. Please make him stop."

What do you think Legal exists for? They're there to protect the company -- from internal threats as well as external ones.

And while you're at it..(Re:You're Right...)

[bwcbwc]

Make sure you use your boss's name and email for all contact information on the user accounts you setup for the scraping.

You're asking the wrong question

[pushf popf]

Sure, it's immoral, and you're violating the other website's TOS. That, however is completely irrelevant.

What is relevant is that any feed you use that isn't backed up a valid contract, can and will disappear at random times, sometimes permanently, as well as contain data you weren't expecting and be missing data you were expecting.

Ask your boss how happy he'll be when the domain owner sells to a spammer and his scraped data is now "Male Enhancement" ads instead of weather data.

I would not recommend this...

[Tord]

The first steps are fine, but I would not recommend you to take the option step of blowinging the whistle unless you really feel strongly about the site or people you "victimize" and see it as you moral responsibility.

If you accept the job and then turn around and blows the whistle you have acted maliciously against your employer. They may have questionable morality but the fact is that you have agreed to work for and being loyal to them, don't sink to their level. They might even have legal grounds to sue you if they find out since you clearly have willingly sabotaged their business.

The only way to take the moral high ground here is to first try to make them change their mind and if that doesn't work refuse to take part in the scheme or at least demand in writing that management take full responsibility. Yes, that could have very bad consequences too. I don't envy your situation, I've been there myself a few times and have not always made decisions that were smart or made me feel good in the long run...

Of course, if things went far enough I would blow the whistle, but I don't get the impression this is one of those cases. It would be a totally different matter though if you weren't working for them or in any other way had promised your loyalty. In that case I would recommend you to blow the whistle as a concerned citizen.

Re:I would not recommend this...

[quanticle]

They may have questionable morality but the fact is that you have agreed to work for and being loyal to them, don't sink to their level. They might even have legal grounds to sue you if they find out since you clearly have willingly sabotaged their business.

Since when did going in to work require you to hang up your morals and ethics at the door? If your employer is doing something unethical, many would argue that you're obligated (morally) to blow the whistle on it, since to do otherwise allows people to profit from unethical examples - setting a bad precedent. If your employer is violating a contract, you'd call them out on it. And that's exactly what a Terms of Service agreement is - a contract specifying the terms by which you may use the other site.

Re:Redirecting content

[cynical kane]

Can you post the URL?

Little Bobby

[Dogtanian]

I told you to scrape Slashdot, not read it. Now get back to work!

I've only one thing to say to Anonymous Slashdot Scraper... '); DROP TABLE rippedoffcomments; -- Goodbye!

Re:Redirecting content

[W2k]

So sayeth Firefox 3.0.3 (Windows):

The image "http://lug.wsu.edu/~ben/dontloadthis.jpg" cannot be displayed, because it contains errors.

IE also refuses to render it. But when downloaded to desktop, I got a thumbnail, and I can open it in Paint.NET just fine (it's actually 0x808080 rather than 0x000000). Paint.NET swallows 3.2GB of memory when I do this. Good thing I have plenty of RAM and a 64-bit OS :)