Slashdot Mirror
Should You Break TOS Because Work Asks You?
An anonymous reader writes "My boss recently assigned me a project that was all his idea, with two basic flaws that would require me to break multiple web sites' Terms of Service (TOS). Part requires scraping most of the site, parsing the data and presenting it as our own without human intervention. While we're safe on copyright issues, clearly scraping like this is normally not allowed. At times it might also put a load on those sites. The other is, for lack of better words, a 'load balancing' part that requires using multiple free accounts instead of purchasing space and CPU time for less than $2,000 USD per month. The boss sees it as 'distributed' computing when in reality it's 'parasitic.'
My question is: am I wrong about the ethics? If I do need to walk, how best can I handle it without damaging my reputation and future employment opportunities?"
I say do it. but EMAIL your boss with your concerns and then continue.
when the shit hits the fan you have documentation to throw him under the bus hard and watch the wheels crush him.
Honestly It's all about CYA in the business world. If your boss tells you to do something unethical or wrong, document it every way you can and hold onto that so you can hand him over.
Why do you have any loyalty to them? they have none for you.
Do not look at laser with remaining good eye.
All that will happen is that the site in question will blacklist your scraping application. I work for a media organization, and we deal with this stuff all the time. It's far more cost efficient for us to simply whack the application than to try and track down the jokers. It's actually pretty trivial to nail an automated scraper: they're obvious on the logs.
So the few times I've had someone ask me to do this sort of scraping, my response is usually that sure, fine, it works, but it's very easy to spot on the logs, and the information is very likely to become unavailable at unpredictable intervals.
In the long run, it's usually pretty futile to scrape in the first place. When you're stealing content just to drive traffic, you tend to have a crappy site. The only time I ever did a professional scraping app that was "justified" and "legal", the victim was another business unit within the same corporation, and we had every right to the data that they "couldn't" compile for us.
ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
Would it be possible to detect the scraper in real-time and redirect it to some fake/spoof data? It needn't be goatse. But it could be!
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
Maybe not in real time, but once someone detected a scraper at a given IP, they could easily change their site to feed that IP fake data instead of blocking it.
If I were in the scrapee's position, I'd probably do that because it's the best way to attack the scraper. From order of least effort on the scrapee's part to most:
1) Blocking it makes it obvious to the scraper that they've been found out, and they'll work around it, then you'll need to block them again, on and on the cat-and-mouse game goes.
2) Feeding them mostly good data but with lots of inaccurate information scattered about is nearly impossible for them to detect until it has irreparably damaged their reputation and/or caused them to make bad decisions based on the data.
3) Suing them is a pain in the butt, even more effort than 2)
retrorocket.o not found, launch anyway?
It's only any good if the other party co-operates. The boss can easily phone you or walk up to you and say "Yes I want you to do it." and you have no record, and for many people this is their default mode of operation because that way no-one can pin anything on them. Unless they're singing their own praises, when everyone gets cc'd in.
I used to find it infuriating but fury gets you nowhere in the workplace.
Yours Sincerely, Michael.
I've also had similar requests in the past, and in both cases I did the work. I considered the request, decided they were ethical (even if somewhat unusual) and so did it. That's something you're going to have to figure out for yourself - whether you're going to do it or not.
I've been on the other side of the fence also...
If you're relying on data for commercial use, putting yourself in a position where you need that data is a risky thing...
I had a scraper once come after me. I caught them - as the previous poster pointed out, it's easy... I didn't block them. I captured and redirected their requests so I could control what they got and, well, sent them some information that made them look really, really stupid. They were angry, but there wasn't much they could do.
They were just enthusiasts - they had no business risk in their application suddenly failing.
Let your boss know the risk he is facing and then ask him if he really wants to risk being caught and shut down unexpectedly, or worse, finding someone has poisened his data.
It's just not good for business.
GrpA
Enjoy science fiction? "Turing Evolved" - AI, Mecha, Androids and rail-gun battles. What more could you want?
Tread wicked carefully!
Chip Salzenberg got his ass burned back in 2005 by grumbling about his employer's ethics regarding screen scraping. I heard him speak at YAPC::NA in Toronto that year, and from what he was saying, they were able to take his every legitimate action (e.g. logging in remotely to work from home) and twist it in court into something less than legit (e.g. unauthorized access). It's their word against his, and they hold the access logs. Your best bet, if you want to make a stand about the morals, is get the hell away from there first.
www.wavefront-av.com
"It's just not good for business."
I find this discussion yet annother interesting insight into the (lack of) ethics of some company bosses. I've often found to my surprise, the ethics of sales people, marketing people and bosses are at times very different from that of programmers and other workers in a company. Some time ago Slashdot discussed "Ethics in IT" and its interesting how it fits with this discussion. Here's the link, it gets interesting how much it fits this discussion, once you get to the part that discusses how some bosses lack of empathy towards others...
http://slashdot.org/comments.pl?sid=448546&cid=22377570
Some bosses have contempt for other people, so considering doing this kind unethical business behaviour, is well within their usual thinking.
There are 10 kinds of people in the world... those who understand binary and those who don't.
>>>The other is, for lack of better words, a "load balancing" part that requires using multiple free accounts instead of purchasing space and CPU time for less than $2,000 USD per month. The boss sees it as "distributed" computing when in reality it's "parasitic".
>>>
Can someone explain what this means? Multiple free accounts of what? Gmail? I'm confused.
Since scraping is detectable, I would follow this course of action:
- tell the boss you think "we'll get caught"
- if boss appears to want to fire you, then go ahead and do the action, but ask for him to put it in writing
- note on the order you think it's a bad idea; keep original for yourself and hand copy to boss
- write the program
-
- (optional)
- from your home computer (using an anonymous account), tell the website what your program does, and explain you would have been fired if you had not complied with your bosses' wishes, but feel it's unethical to scrap data.
- watch as Boss looks like fool when website with stolen bandwidth decides to bar his company's access
- if fired, hire lawyer and sue the company for unjustified dismissal
$ profit
FOX NEWS.com should be BANNED from television and internet. Have the Congress take it over and give us Truespeak.
On a similar note, email is great for lifting others up. A few years ago, I got promoted to a mid-level grunt, and worked a lot more with other business units. In big organizations, units don't play nice with other units. On the few occasions, someone went out of their way to provide good service and actually be helpful, their boss got an email letting them know. Maybe two or three emails a year, but those folks are the ones who are going to get the bigger raise and have chances at promotions.
Unfortunately, 95% of all emails about people are complaints. Do your job well, and the best you can hope for is to be ignored. I have personally tried to reverse that now that I am a low-level manager. I can't change the world, but I can influence my dusty corner of it.
See my journal for slashdot ID's by year. Mine created in 2005. http://slashdot.org/journal/289875/slashdot-ids-by-year
The example that would leap to my mind is a number of services that allow you to "map" an ip address to a geographic location...I use one of those for my job search homepage, and it only allows ~200 queries a day for the "free" account...It would be plenty useful to have as a free service (targeted advertising), and if you set up enough "free" accounts, you could use it that way.
Since I'm doing all my job searching away from where I'm currently living, I use mine to make sure that my job searching page always looks "under construction" for people who live where I live. My boss actually checks it occasionally, I guess to make sure I'm not trying to leave.
ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
And could prove to be very amusing for a future slashdot submission if they encounter a BOFH.
There are just so many things that could be done.
They're planning on taking data from some site and pumping it to others and they have _ZERO_ assurance that it's going to be good data and continue to be good data.
When you do stupid stuff like this, if you're not careful very bad things could happen (SQL injection, maybe even malware slipped in) and they could just go "nope not us", and while you could try to sue them it's pretty darn hard to prove since you requested the "bomb", and it only appears once and never appears again.
If you're lucky it's just going to be goatse/tubgirl.
If you're not, it could be a lot worse. Just imagine the BOFH thinking "What should I do today to them and their users" and rubbing his hands with glee.
Just slightly tampered data will be bad enough.