Slashdot Mirror
Robo-chattel? New Legal Challenge to 'Bots
milomilo writes "Extending on the eBay vs. Bidder's Edge case, the NY Times reports (free registration required) that a Manhattan judge has granted a preliminary injunction against Verio from using 'bots to harvest up-for-renewal prospects from Register.com's WHOIS. The theory's that bots use up a piece of the target system's resources, denying its use to the owner. (Question: would search engines be different, presumably because they also confer a benefit on the target by making it findable?)"
I.E. you nonmaliciously (meaning, it isn't a DOS, you're actually getting information) ask for large gobs of information off of some site, the way these bots did.. or the way a spambot might.. they call this "denying services", but still, it's a simple the questioner requests, the answerer replies. If it's "unauthorized use".. well, how can you talk about unauthorized use on a public server? How can these things, authorization and to who, be implied on a public internet? Should it be the job of the requester to not go where they clearly shouldn't be, or the job of the requestee to keep them out?
Or look at it in terms of a port scan. I request things from each of these ports, thus figuring out which are open (and thus vulnerable to attack). I've seen people try to procecute this based on "unauthorized usage of machine".. well hold up, who said you had to authorize something? This person is just sending pings to ports, on a machine that by its presence on the internet you have implied responds to traffic. Why on earth would you need "permission" prior to using a system? If so, how would that permission be obtained? .. but of course none of this changes the fact that the port scan is almost always part of a malicious cracking attack.
Or, let's say-- hypothetically-- there was a single-line javascript that, if accessed from a windows NT machine, would cause the kernel to be overwritten by 0s. If you put that up on a web page, would that be "hacking"? You didn't break the machine yourself; you politely ask the machine to break itself, and it complies. Is that your fault?
But then, when you get down to it, all forms of "cracking" could be seen as requests. I request you process this block of information that just happens to cause a buffer overflow... you didn't have to process it, now did you? That last bit doesnt' really sound reasonable.. you have to draw a line somewhere, you have to note somewhere that it's no longer a request but an attack. Somewhere, for the sake of sanity, you have to draw the line, and how do you do that? Intent? How do you prove intent in court? What's the difference between the slashdot effect and a DDOS, at an abstract level?
But still how the hell can you say it's illegal to ask for something because the questioned might give you an answer even though they don't want to...? That's where the law is heading, where it's been heading for awhile, and that's completely absurd.
There is no right answer here, is there?
Irritable, left-wing and possibly humorous bumper stickers and t-shirts
Question: would search engines be different, presumably because they also confer a benefit on the target by making it findable?)"
The standard search engines, such as google, altavista, etc, know and obey robots.txt, which is the same as Register.com's policy of not allowing spambots search through their site. If, after a robots.txt file is in place and the search engine continues to index it, I would say there's a good legal case there.Now, more interestingly is tools that 'mirror' web sites; they still are using a resource that you've made publicably available, except doing it over a timeframe that is much shorter than a human can do it, which usually means more resources used up at the server end. These bots tend not to follow robots.txt rules, and are only defeatable by User-Agent blocking. If the above ruling stands, does it apply here?
Take it a step further: Ebay has taken action to stop meta-Ebay sites that index their site and make it easier than ebay's search engines to find things or to search multiple auction websites. Even though the information that is up is publically available from ebay, and IIRC, they still won, mostly because the information is still ebay's property and they didn't like it on other sites.
Which all leads to an interesting question: when you click on a link, does that start a clock in which you have temporary copyright ability to download the information to your local computer, and after some time, that ability 'expires'? If so, sites that index or mirror without further authorization could find themselves in trouble...
"Pinky, you've left the lens cap of your mind on again." - P&TB
"I can see my house from here!" - ST:
AMSTERDAM - TEASERS
TEASERS
(EX-CHooters)
Damrak 36
1012LK Amsterdam
T. 0031-20-4287508
FIRE WARM UP PARTY in the Teaser's to 22.04.00 of 13.00 - 16,00 o'clock
The Teasers of sport bar, typical American bar, was always the meeting place of the NFL Europe fan (particularly with the Scots) for their Party's.
Special flag are the waitresses, who are inferior to our Pyro's hardly. And if one has then times birthday, then the waitresses let themselves also which "nice" be broken in (not truely, Living putting)! Who would not loving have exchanged with you gladly!
Freely after the slogan: Man, those are thick, man!
By the way: The Teasers was called in former times Hooters , like the American branches. The name did not change, but in the concept to anything. It continues as in the Hooters. It was probably probable more a license problem. The Disco is inferior also during the day no Discothek. A D.J. with a violent sound system and Lightshow brought still each Partymuffel in tendency. The meal is very good in addition, not cheap. The Damrak connects the Dam Square with the main station. The bar can be attained by both workstations within 5 minutes. Beside the Teasers an excellent typically Dutch Frittenbude with megaportions of Pommes and thousands saucen is direct.
What really ticks me off is that "The Old Media", through which many people still get their news, has latched on to Slashdot as "The New Media", meaning that Slashdot will be reflecting on my own efforts, and the efforts of anybody else trying to run a 'new media' style website. This is why I post this; Slashdot's flub-ups are personal and affect us all. The flub-ups affect people running new media sites (by tarnishing the reputation in the eyes of the Old Media press who doesn't care to dig past their original generalizations), they tarnish the reputation of Open Source (as they have been labelled the spokesperson of the Open Source movement by the same collection of media entities), and they tarnish the reputation of VA Linux. (Hey, anybody at VA listening? This is not good return on your investment!)
Slashdot editors, wake up! You are not invincible. You can be replaced, and in Internet time, too. Please get some ethics, before you convince thousands or millions that the New Media doesn't have any!
There are a million ways around the injunction. (see example above) I think from a moral stand point the judge is correct. Unfortunately being morally correct doesn't mean a damn thing.
SPAM uses extra CPU cycles, in some cases spam causes users to go over quota (is that a DOS attack against my users?) Is SPAM outlawed, is anyone REALLY doing anything about it?
NO
Do i think that this ruling will change the unsrupulous? No. Hopefuly Verio will show us that it is a "good guy" and go about it's business the right way.
"Science is about ego as much as it is about discovery and truth " - I said it, so sue me.
Just because the machine is connected to the public internet does not mean that the machine is open for anyone to use however they please. This is enshrined in UK law these days (Note the gradual disappearance of "Welcome to hostname" for login prompts, it can be argued it's an explicit invitation for hackers to enter your machine
I mean, your telephone is connected to the public network but would it be OK for me to set up a bot to constantly dial your home to see if you'd dropped the price on the car you were selling?
Rich
'from the what'll-they-think-of-next dept' ? I already wonder what he was thinking when posting the article...
Btw, think about the 'hooters' admin that have its site slashdotted right now...
1 reply beneath your current threshold.
I miss the days when CT and Hemos regularly patrolled their site, and would fix problems rather quickly. One can only assume that now they spend their days surfing for Hooters sites :-)
/robots.txt file. If a web site wants to be indexed, they put permissive rules in robots.txt. Verio is spidering for their own commercial gain, and ignoring a number of posted policies against it. That is apparently what the judge has ruled on, violating an explicit request not to harvest. What is funny is that register.com doesn't have a robots.txt file, so does that give people permission to spider the site?
/., but I'd also believe that Hemos just pasted the wrong link into the story from one of dozens of open browser windows, and didn't really double check before posting. Haven't we all done that at some point :-)
ObOnTopic post:
Question: would search engines be different, presumably because they also confer a benefit on the target by making it findable?
The difference between what Verio is doing and search engines is one of implied permission.
Web sites either grant or deny permission to search engines based on the
the AC
Maybe this was just a simple hack of
Hemos is like...sci-fi fans;he thinks technology is cool, but he hasn't bothered to understand the science it's based on
"Secondly, robot.txt is often a server level setup file. If you get some
free space with the likes of AOL/Freeserve/Geocities you have no control
over the indexing of your site. Additionally, some (albeit poor) ISPs
don't offer configuration of this file."
If you can't control your ISP's robots.txt, in the header of each page
put:
<META NAME="ROBOTS" CONTENT="NONE">
I have 70MB of pages on my ISP and robots were costing me a bundle
till I put this in. Now all major robots ignore it and only a few
oddball wiseguys occasionally download the entire site (even though I
offer them a compressed version of the entire site at my ftp site)
This is good in the sense that another company should not peruse a site to gather contact information for marketing purposes. I've always thought these kinds of practices were dishonest to say the least. I think this kind of behavior should be curtailed.
On the other side of the coin. This is bad, because of this:
"If I don't like your linking to my site, or searching my site, even though it is open to the public, and I say, 'Stop,' you have to stop . . . whether you are actually hurting me or not."
Crawlers and Links shouldn't be penalized. It's a way of finding useful information (as opposed to finding new business... ...A way of getting contact information of people who probably don't want to be contacted, anyway.)
I also understand the reasoning behind the robots.txt file. If the information being gathered by a crawler will be outdated (in the case of eBay auctions) then it's a good way to selectively remove portions of your site for searching, because it's not appropriate.
---
The (Hopefully) Great Slashdot Blackout
... if they can't check the link, will they even read the article?
------ Curiosity killed the cat. {satisfaction brought it back | it didn't die ignorant | lack of it is killing mankind
i was in a meeting then lunch, so i missed the hooters link, but the fact that slashdot changed it like nothing happened without an update is wrong and scary! i mean, this reminds me so much of "1984" where the history is rewritten as time goes by.
------ Curiosity killed the cat. {satisfaction brought it back | it didn't die ignorant | lack of it is killing mankind
It's worth noting that search engines honor the robots.txt protocol, so any web site can easily opt out of being indexed. There isn't anything like that in WHOIS. If I remember right, ebay lists its auction items as off-limits for bots in robots.txt. I see that as the strongest distinction between search engines and the cases mentioned here.
- Russ
Who would go to Hooters in Amsterdam? "Well, I can go smoke the best weed on Earth, go see a live lesbian sex show, boink two prostitutes at once....or I can go see chicks in small shorts and eat chicken wings." And who is the dorky guy in the corner of the bottom picture? Hemos?
-B
For those of you not afraid of goatse.cx, http://www.nytimes.com/2001/01/12/technology/12CYB ERLAW.html
-=Best Viewed Using [INLINE]=-
Load-sharing boxes for server farms ought to have this feature. And it should go into Apache.
http://www.nytimes.com/2001/01/12/technology/12CYB ERLAW.html
I did notice, however, that the required registration at the "New York Times" was not free...
information wants to be expensive...nothing is so valuable as the right information at the right time.
Obliteracy: Words with explosions
If this was not Register.com's WHOIS service that was being used, then I would consider it a little more like a company that makes photocopiers looking in a public phonebook for big businesses, calling them up and saying "Hey, we'd like to do business with you and we'll beat whatever your current photocopier service is charging you".
---
seumas.com
Oh my god! The Hooters girls were bots all along? I feel so dirty!
(And yes, the Hooters girls do use up resources on the target system, if you get my drift)
--
"Everybody must be allowed to access web resources" is a statement from the POV of the accessors. Consider that statement from the point of view of the server managers: "We must allow everybody to access our resources in any way they choose."
Do you really want to make that statement? If you put up a public resource, must you allow people to abuse it if they wish? Or can you take actions to stop such abuse, esp. as it nearly always does real, if not always a lot of, damage. In the case of Bidder's Edge vs. eBay, eBay was suffering real slow-down of service, which affects its bottom line. Must eBay allow it?
Perhaps the real danger is not so much the rulings per se, but the legal doctrines being used to make them: "Under the reasoning in the Register.com case, "you don't have to prove harm or show any evidence of harm," he said. "Harm will be presumed." He said that he fears the Register.com case will "spread like Kudzu" through the court system."
At any rate, just recognize that things are somewhat more complicated then they may seem at first. It's tempting to oversimplify in either direction, but the truth is probably complicated.
Oh no, baby... are you trying to tell me that all along you've been fembots? But that's just not groovy, baby!
(Translation: Click the link, Hemos. Or even just hold your mouse over it to see where it goes.)
--
This is 1.11% of the "total load", meaning 1.11% of the CPU usage, not 1.11% of the CPU. Nothing here argues that the machines were fully loaded. Had they been, then yes, eBay was prevented from using that load. But it's doubtful that eBay had a 100% load all of the time, so it's doubtful they were using those resources.
-no broken link
The simple fact is that every single interaction between any two computer systems requires resources in the way of memory, processor time, network bandwidth and sometimes disk access. Now the judge seems to indicate that if you go over my service as it was designed, and retrieve information, and then use that information in a way I don't like, I can forbid you from using my service.
That's completely unrealistic. Unless the searching routine is basically stomping the server by requesting as fast as possible, there is no real damage being done that isn't done by a regular cyber-squatter wannabe trolling the database in this case. If there is an issue with the DoS sort of effect, ask the other party to back off, or alter the server software to restrict the rates at which requests are accepted from certain IP addresses or blocks. Better yet, negotiate a new service where the database query is run locally by the whois provider, bundled up, and distributed for fee to anyone interested.
The thing is, it is public information. This sort of legal adjustment of the reality is foolhardy in the extreme. If I take a quote from an article on a news site, citing the reference properly, and use it as a portion of a work that results in something the originating news site doesn't like, can they forbid me from using the site now?
The ramifications are a lot further reaching than just bots. It's all a matter of degree.
The ruling itself is good, but it seems like there needs to be a better logic behind future rulings. What happens when a company is affected as Register was but cannot show an appreciable system burden, or at least one that a judge will accept? Using trespassing as part of the arguement makes some sense, as a website or database could be construed as a property, though I wonder whether trespass laws are written to sufficently to cover virtual property? There comes a point when trying to use analogies for the Interent becomes futile. The Internet isn't necessarily like anything else. The Internet is its own thing. Its time we had policymakers that understand that and deal with it appropriately. http://unholyrouter.com
There is no guarantee that the content has been read or understood.
The WHOIS database states that running bots through it violates their user agreement. Doing so with webpages, however, is encouraged by most sites, and can be blocked with a robots.txt file at any time.
Interested in open source engine management for your Subaru?
I know a lot of people here are very anti-regulation, but I think it would be great if case law established that web robots must obey the Robots Exclusion Standard. Since it's a widely-known standard, I think it can be fairly argued that robots that choose to disregard /robots.txt are in danger of tresspassing to chattels. Using the standard also would allow bots to fulfill their helpful role, while providing a clear distinction between what is and what is not acceptable.
/robots.txt
Sure, one might argue that people might be unaware of the standard, but that is seldom an excuse. I may be unaware of fire/electrical codes, but I'll still get in trouble if I don't adhere to them, because I'm putting others at risk and thereby imposing a cost upon society (fire trucks and insurance don't come free). Web crawlers that index data in violation of the Robots Exclusion Standard impose a cost on companies and society just as well, in the end requiring people to by bigger pipes, faster servers, and so on (thereby using more power, dumping more old computer components into landfills and more chipmaking chemicals into the environment).
My point is that web crawler operators live in a society, just like everyone else, and they too must be held accountable for the consequences of their actions, particularly when they willfully disregard the requests of web site operators as expressed in
This is amusing. Now we get to see who actually clicked the link, and who posted blindly without bothering to read the article.
-Todd
---
"The details of my life are quite inconsequential..."
Outside of the issue of bandwidth, there is the issue of profitability. Part of many websites income is derived from banner ads. If a bot scours a website to harvest the content, it prevents the end users from seeing some of the advertisements they would have normally have seen by exploring it on their own. One example of where this hurts is a Meta Search engine that trolls several search engines and produces a compiled list of search results. Search engines have millions of dollars invested in hardware, software, bandwidth and staff to make it all work. Every single query has a real monetary cost associated with it. Every free service has its cost. A Meta Search engine bot like that only does about %5 of the total work involved producing the results or content that it displays. Now, a site like that makes its OWN profit from users with minimal money out of its own pocket (of which none goes to the companies doing most of the real hard work).
I completely understand why companies would get cranky about someone repeatedly grabbing computationally intense data from their site and profiting from it as they suck money and resources away from the provider of said data.
One way websites are about to track when people visit a site is "tracking gifs". Usually very small 1x1 pixel images that give them a general idea of how many visitors a site is receiving. Reports are generated and they get PAID by advertisers based upon this info. Bots RARELY grab anything but content. If you go to a webpage with images embedded in it, your web browser individually requests (most of the time barring cached data) each image. Since bots don't tend to request this graphical "fluff" intended for hyoomans, owners of the site notice an increase in site traffic and resource drains and a decrease in "ad impressions" when a bot "isn't polite". Yes, I do realize that some make revenue in other ways, but bots can use up resources faster than humans can regardless. With people your traffic usually scales slowly up or down. You can add or remove hardware to deal with the demand. Sometimes when a bot hits your webserver it is such a huge spike in requests for data it kills the server. Then all the legitimate users get cranky. Or, a more minor form is that it makes the s e r v e r sllluuuugggggiiish.
Anywhoo... Thats my $1.25 on the matter. Cheers. - Sartian
My bad - not Hemos'. Flame him if you want for not checking every link in every story - with the volume of submissions what they are I can't say's I blame any of the good folks at /.
(Why the bad link? A friend had just ICQd me that he was headed to Amsterdam for a P2P conf. and wanted the name of the place all the 'football' fans go to. Cut and pasted crosswise. I suppose he'll be wondering why he should go to a 'bot lawsuit in the City of Sin... ;-)
And for those of you who got their panties in a bunch about the 'unacceptable' or declining quality of /. (ACs, anyone?) - so quit reading it already and run your own. MHO - pretty damn fine job of turning a homebrew blog into a major news source - whynchYOU try it!?
Windows in a public building are obviously meant to be looked through. However, if you stood long enough, gazing through the windows of a local store, they could have you removed if there are "no loitering" laws. Even though you are using the sidewalk for standing and the windows for looking, as they were meant to be.
Similarly, if you worked at one store and went to your competitor's, pen and paper in hand, and strolled the aisles noting their prices (so your store can meet/beat them), you might be asked to leave. Despite the fact that you are just writing down prices that are clearly there to be read.
Finally, various retailers, esp. car dealers, place "No wholesaler or retailer" restrictions on their best sales, even though their products are meant to be bought and other retailers may want to do just that.
It seems to me that analogous laws already exist. Just because something is available in the public realm it doesn't follow that anyone can avail themselves of it to any extent; at least not under current U.S. laws.
-----
D. Fischer
ShoutingMan.com
http://channel.nytimes.com/2001/01/12/technology/1 2CYBERLAW.html
Although here is this interesting bit from the middle of the article:
"It is a greater offense to steal men's labor, than their clothes"
"Question: would search engines be different, presumably because they also confer a benefit on the target by making it findable?)"
This ruling is obviously rediculous; who hosts a website that can't handle 1 extra connection?
But just to play devil's advocate, search engines are slightly different, since you can always specify in Robots.txt which robots, (none even) can access your site, and what they can access (effectively controlling their time on your site).
I would like to see a law against robots which do not adhere to robots.txt though...
Oh wait, this is the internet and it shouldn't be online if it's not meant to be accessed... sorry, I forget sometimes...
Ace
It is somewhat similar to why a single person dubbing a cd to tape but I can't copy a ton of cds and resell them. I can browse for prices as a shopper but I can't suck all their prices off their web site to always make my prices 5% lower. I am really worried about this direction of cyber law. As technology gives the average joe the ability to do everything on a large the scale, the "mass" part of the legality test is broken. Hence, the Metallica Naptster battle. Although the people are not profiting from it, the number of the copies ditributed pushes the courts to Metallica's side. I worry about the quite nontechnical courts ruling on matters of technology that will effect the US for years to come. I don't mind corporations squabbling in court, as is the case with register and verio. I do believe that access to a site by a specific entity should be throttled so as to not starve off the regular consumer. I do believe that breaking those limits should be considered illegal. What I don't want to see is the corporate world coming after the regular consumer who is not making money off the situation. The fact is, if music was sold for a modest profit and music companies were pickier about who they try to turn into super stars they could make a lot of money while charging much less money. Technology is forcing the music industry to improve its product and provide real value for the consumer. I see the likes of Napster and the free music sharing phenom. to just be another example of how competition provides better value to the consumers, as it is suppossed to. This form of competition should not be stiffled just because it may cut into the profits of current companies. The law should not be meant to protect the status quo but protect the regular citizens.
-- soldack