There's a letter in favour of software patents that Nokia's patent department has been circulating, trying to get European CTOs to put their companies' names to.
"All of Europe's innovators, including individual inventors, small and
medium size enterprises (SMEs), as well as large multinational
companies, require patents to protect their inventions, provide
incentives to undertake research and development in Europe, and to
promote licensing and technology transfer", claims the letter.
"Nokia doesn't seem to be counting Opera among the European innovators",
comments Håkon Wium Lie, CTO of Opera Software Inc, an innovation leader
in the web browser market and producer of much of the software used in
Nokia's mobile phones.
And, as Hartmut Pilch president of FFII and speaker of the Eurolinux
Alliance explains, Opera is just one of "more than 5000 European CTOs
and 2000 CEOs who have publicly endorsed our petitions against software
patents".
Pilch continues:
"The Nokia patent department's claim that patents are needed to fund
research in the software sector looks like a desperate attempt to
mobilise the misconceptions of people who are not familiar with the ICT
sector. All the economic studies we know of, including those ordered by
the European Commission and by member state governments, have shown that
software patents are only of very secondary importance as a means of
securing investment in research and development. The main drivers of
competitive advantage are copyright, in-house capability, unavoidable
complexity, and the ability to react quickly to customer needs. In
fact, according to the most detailed economic studies, patent
investments in the United States have actually tended to reduce spending
and divert it *away* from R&D investment in this sector. These points
came out particularly clearly in the testimony given by directors of
large companies to the US Federal Trade Commission at governmental
hearings in the USA last year".
"The letter from Nokia is written from a perspective of a corporate
patent lawyer concerned about a possible erosion of his department's
importance within his company. Ministers should see it for what it is."
and, later on:
FFII's most hollow laughter is directed at the claim that the Irish
proposed text would not "unduly hinder interoperability".
Jonas Maebe, Belgian speaker of FFII, explains:
"The Industry committee, and the Legal Affairs committee, and the full
session of the European Parliament, all demanded a special provision to
allow data to be inter-converted between different packages and software
platforms. Otherwise companies can use software patents to lock in
users' data to a particular program or operating system, and competition
would be impossible".
"It's a systematic problem. Each and every market niche is individually
potentially at risk. That's why, in the final vote in September, the
European Parliament voted in favour of the provision by 393 votes to 35".
"But according to Nokia, the Council Working Party has 'responded' to
the European Parliament's call, so everything's all right. And how
(despite a valiant last-ditch opposition by the Luxemburgers) does the
Working Party propose to respond ? By deleting the European
Parliament's clause entirely, and instead replacing it with a recital
that says any problems can be left to existing antitrust law".
"Remember, this is the antitrust law which has just taken four years, at
vast expense, to go after a/single/ accused company, Microsoft; which
Microsoft says it can tie up in appeals for another four years; and
which at the end of the day looks like the case will be settled with a
cosy cross-licensing deal between Microsoft and Sun, and Samba
definitely not invited to the party".
"One starts to wonder about what kind of idealised dreamworld these
people live in."
"The European Commission draft requires Microsoft to share proprietary information with rival server makers"
Note however that the information will be available only to those who pay for a licence; and who accept MS's terms and conditions.
"The draft decision says the US firm must share more of its protocols with rivals, charging a reasonable royalty. It will be left to Microsoft to work out the precise solution, with close oversight by the commission". -- Guardian
The number of IP licences taken out actually fell after they were mandated by the US judgment.
A better solution would be for to require the protocols be openly published, with their value for non-monopolistic use netted off against the fines the EU wants to impose.
The penalty for misusing protocol specifications to reinforce a monopoly should be the loss of control of those specifications.
So from paragraph 1, it seems as though the applicant (the one wishing to do a raid, for example) will need to demonstrate to "competent judicial authorities" that there is a clear and present danger of evidence being destroyed.
Umm, no.
That was more or less what the original EU Commission text proposed.
But if you re-read paragraph 1 carefully, it now says that the orders shall be granted "in particular" under those circumstances.
What it actually requires is that judges must be allowed to grant such orders at their own discretion, "on application by a party who has presented reasonably available evidence to support his claims that his intellectual property right has been infringed or is about to be infringed".
The directive in fact overrules existing safeguards which say that the orders can only be granted when there is irreparable harm or destruction of evidence likely.
The only European countries which currently provide for such orders are the UK, Ireland and France.
In these countries the orders are indeed granted directly to the plaintiffs, in secret, without the defendants' case being put, authorising the plaintiffs themselves to go ahead.
Here's what a standard thousand-page textbook on UK Intellectual Property law has to say about such measures, called "Anton Piller" orders (Cornish & Llewellyn, 5e, 2003: section 2-43, page 82):
As a measure of "privatisation" the order is remarkable: a non-State agency is employed in the direct infarction of personal liberties; and more than that, those executing the order act for the very person who can least be expected to preserve a measure of objectivity and sense of proportion...
Although the reassurance was at first given that the orders would be rare [22], the procedure is regularly used, and it has considerably increased the speed and effectiveness of civil process. Yet it raises the spectre which in former times made the court so fearful of the general warrant to search. The proceedings turn on the plaintiff's evidence alone, and they occur in camera. If a single judge is satisfied prima facie that there is an infringement and a likelihood of serious injury, the plaintiff through his solicitor is empowered to launch his own attack on the defendant. Those executing the order are likely to believe that right is on their side and that they must put on a show of aggression if they are to secure what their client needs and deserves. In the tensions generated by the surprise service of the order, a defendant will need considerable temerity if he is to seek legal advice and challenge the basis on which the order was made.
You misunderstood the case. This IPR Enforcement directive is NOT about the legal validity of software patents.
True, but there are still 30,000 granted software patents in Europe, and the industry giants are pushing very hard on the EU Council of Ministers to reverse the Parliament vote.
The implications are potentially very frightening, if the sort the enforcement measures proposed in this Directive become available for use by any agressive litigation company acting on the basis of a dodgy software patent.
So I think Aurix's comment connecting this with software patents is right on the mark.
As to FFII's position: FFII is strongly in favour of copyright, and FFII is strongly against piracy.
We think copyright is the right way to build a fair information infrastructure. We are vehemently against commercial software being ripped off and placed on warez networks; and we are vehemently against GPL software being ripped off by closed source companies.
But there has to be a balance between the interests of supposed rightsholders (often very powerful) and the interests of other creative companies against wrongful claims of infringement.
It's not just about patents. We're also very concerned about the possibility of these measures being used in disputed copyright cases, disputed trademark cases, disputed cases about claimed confidential information... etc, etc, etc. Under Article 2, the Directive could be applied to "any [claimed] infringement of intellectual property rights as provided for by Community law and/or by the national law of the Member State concerned" -- ie at the moment the whole of Intellectual Property law could be grounds for initiating these measures.
That is why we believe it would be safest if the directive were to be sharply reined in, and apply only to commercially organised, knowingly intentional copyright and trademark infringement.
So does the FFII believe that these kind of disputes should be left to the legal systems of individual member nations then or would the FFII prefer to have the EU draft some other directives to handle them uniformly? Or would any directive on this kind of dispute be too draconian according to the FFII?
Europe isn't entirely a legal black hole, you know.
It's worth noting Article 2.1:
the measures and procedures provided for by the Directive are to be
"without prejudice to the means which may be provided for in national legislation, in so far as these means may be more favourable for rights holders",
Recitals 4 and 5:
(4) At international level, all Member States, as well as the Community itself as regards matters within its competence, are bound by the Agreement on Trade-Related Aspects of Intellectual Property (the "TRIPS Agreement"), approved, as part of the multilateral negotiations of the Uruguay Round, by Council Decision 94/800/EC5 concluded in the framework of the World Trade Organisation.
(5) The TRIPS Agreement contains, in particular, provisions on the means of enforcing intellectual property rights which are common standards applicable at international level and implemented in all Member States. The provisions of this Directive should not affect Member States' international obligations, including those under the TRIPS Agreement.
and Article 20:
Without prejudice to the civil and administrative measures and procedures laid down by this Directive, Member States may apply other appropriate sanctions in cases where intellectual property rights have been infringed.
The point is, that this directive goes far beyond what was agreed at TRIPS in promoting the interests of supposed rightholders. (It is "TRIPS-plus" in the jargon, or "the DMCA on steroids", according to Ross Anderson in Cambridge).
The other point is that it is absolutely against any idea of good lawmaking for rightsholders to try to crash this through all its remaining Parliamentary stages in three weeks flat.
It's fine to oppose something on principle but the FFII's alert doesn't seem to be proposing any alternative solution, other than "not what you've got", which weakens their stance IMHO. If they want to make a stand on IP, then they should do so instead of just being naysayers.
You might like to look at the amendments being canvassed by Andreas Dietl of European Digital Rights (EDRi), which you can find on the FFII website.
This directive merely harmonises law across states: it does not increase it (except for those states that have poor IP law)... The "anton piller" orders (i.e. search and seizure) they get so upset about are already available in many jurisdictions (e.g. the UK), all the directive is doing is making sure that the same procedures can be used across all EU member states, otherwise copyright infringers are able to locate their activities in a low-enforcement-quality state and make copyright law ineffective elsewhere in the EU.
Strangely enough, this is just what Janelly Fourtou told us in Strasbourg last week, just after she tried to persuade MEPs that we were talking about an out-of-date draft of the Directive -- which we weren't; and just before she told us that she too has been trying to take patents out of the Directive -- which, according to minutes of recent 'Trialogue' meetings, she hasn't. (Mme Fourtou is the MEP who is steering this through the Europarliament, and entirely coincidentally happens to be the wife of the CEO of Vivendi-Universal)
It simply isn't true.
Anton Piller orders are currently only available in the UK and France ("saisi-contrefacon"). These secret court authorisations of raids for evidence carried out by the plaintiff's own agents are not available in any of the other states of the EU.
Furthermore, after very strong criticism from the most senior judges, in the UK a strict new code of practice was brought in in the early 90s which cut the number of applications granted by a factor of ten.
(See this page for references to the detailed cases).
The judge who led that criticism, Lord Scott, was subsequently head of civil justice for five years, and is now one of the Lords of Appeal in the House of Lords -- the most senior court in the UK.
He now chairs the House of Lords scrutiny committee which has refused to clear this legislation. If he is concerned about the detailed text, then we all should be.
We are talking about unannounced dawn raids by private security firms, piling in with legal authority and seizing entire computer systems and filing cabinets full of documents. That is a terrifying and destructive experience for a small firm.
That is why FFII is arguing that such measures should only be available in the most extreme circumstances, and where there is clear evidence of a deliberate knowing intent to infringe for commercial gain on a commercial scale.
Such measures are totally inappropriate where there is no such deliberate piracy, and no such emergency, in cases as complex as those in patent law and disputed ownership of confidential information/trade secrets, which routinely can take five years in court. Such measures should not become automatic standard procedure in all IP disputes.
Furthermore, we think it is simply insane to try to crash through such a major change in the civil justice system -- a truly massive change in the whole legal IP environment for most firms in Europe -- in three weeks flat from publication of the detailed text to final vote in Plenary, short circuiting all the normal three readings procedures of the Parliament, and before even first publication of the results of the UK consultation and the UK impact study.
I didn't think that europe had actually passed software patents yet but only those tied in with hardware that you can prove are entirely new processes and inventions - this does not count!
The European Patent Office has granted something like 30,000 software patents over the last 20 years.
But that has been done, without legislative approval, by the EPO re-interpreting the rules to mean diametrically opposite to what was originally intended.
It's applications like this one from Microsoft which make the current legislative battle in the European system, which will finally write the official law on this, so vitally important to win.
It looks as though a very similar patent application has been filed in Europe, EP 1376387.
Clicking on the 'Claims' button in the top line of the page reveals:
EP1376387. Word-processing document stored in a single XML file
Applicant(s): MICROSOFT CORP (US)
Claims:
1. A computer-readable medium having computer-executable components, comprising:
a first component for reading a word-processor document stored as a single XML file;
a second component that utilizes an XSD for interpreting the word-processor document, and
a third component for performing an action on the word-processor document.
...
and so on down to claim 31.
This is only a patent application, not (yet) a granted patent (in fact in the EU the patent application has only just been published, on 2 Jan this year).
From the 'priority number' (US20020187060 20020628) it looks as though the original application was for a US patent, filed some time in 2002. So that is the cut-off date for prior art.
The full paperwork file for the EPO patent application can also be viewed, at EPOline.
If it doesn't allow you to gather information, then don't.
I'm not sure that's the whole answer.
Many sites may not have a robots.txt file, yet may still value their copyright and/or database rights.
On the other hand, for some purposes it may be legitimate to take some amount of data (obviously not the whole site), even in contravention of the wishes of a robots.txt
So I think the question is deeper than just "look for robots.txt"
How much can you screen-scrape legally ?
on
Spidering Hacks
·
· Score: 4, Interesting
Question: how much screen-scraping can you do, before the legal questions start ?
In the USA, trading information that has cost somebody else time and money to build up can be caught under a doctrine of "misappropriation of trade values" or "unfair competition", dating from the INS case in 1918.
Meanwhile here in Europe, a collection of data has full authorial copyright (life + 70) under the EU Database Directive (1996), if the collecting involved personal intellectual creativity; or special database rights (last update + 15 years) if it did not.
I've done a little screen-scraping for a "one name" family history project. Presumably that is in the clear, as it was for personal non-commmercial research, or (at most) quite limited private circulation.
But where are the limits ?
How much screen-scraping can one do (or advertise), before legally it becomes a "significant taking" ?
The rights conferred by patents granted for inventions within the scope of
this Directive shall not affect acts permitted under Articles 5 and 6 of
Directive 91/250/EEC on the legal protection of computer programs by
copyright, in particular under the provisions thereof in respect of
decompilation and interoperability.
Article 6a
Member States shall ensure that, wherever the use of a patented technique
is needed for a significant purpose such as ensuring conversion of the
conventions used in two different computer systems or networks so as to allow
communication and exchange of data content between them, such use is not
considered to be a patent infringement.
Member States shall ensure that patent claims granted in respect of computer-implemented inventions include only the technical contribution which justifies the patent claim. A patent claim to a computer program, either on its own or on a carrier, shall not be allowed.
Article 5b
Member States shall ensure that the production, handling, processing,
distribution and publication of information, in whatever form, can never
constitute direct or indirect infringement of a patent, even when a technical
apparatus is used for that purpose.
Article 5c
Member States shall ensure that the use of a computer program for
purposes that do not belong to the scope of the patent cannot constitute a
direct or indirect patent infringement.
Article 5d
Member States shall ensure that whenever a patent claim names features
that imply the use of a computer program, a well-functioning and well
documented reference implementation of such a program is published as part of
the patent description without any restricting licensing terms.
Member States shall ensure that a computer-implemented invention may be
claimed only as a product, that is as a programmed device, or as a technical
production process.
1. A computer-implemented invention shall not be regarded as making a technical
contribution merely because it involves the use of a computer, network or
other programmable apparatus. Accordingly, inventions involving computer
programs which implement business, mathematical or other methods and do not
produce any technical effects beyond the normal physical interactions between
a program and the computer, network or other programmable apparatus in which
it is run shall not be patentable.
2. Member States shall ensure that computer-implemented solutions to technical
problems are not considered to be patentable inventions merely because they
improve efficiency in the use of resources within the data processing system.
1. In order to be patentable, a computer-implemented invention must be
susceptible of industrial application and new and involve an inventive step.
2. In order to involve an inventive step, a computer-implemented invention must
make a technical contribution.
3. The significant extent of the technical contribution shall be assessed by
consideration of the difference between the technical elements included in
the scope of the patent claim considered as a whole and the state of the art.
3(a) In determining whether a given computer-implemented invention makes a
technical contribution, the following test shall be used: whether it
constitutes a new teaching on cause-effect relations in the use of
controllable forces of natures and has an industrial application in the
strict sense of the expression, in terms of both method and result.
Member states shall ensure that data processing is not considered to be a
field of technology in the sense of patent law, and that innovations in the
field of data processing are not considered to be inventions in the sense of
patent law.
Directive on the patentability of computer-implemented inventions
Article 2: Definitions
a) "computer-implemented invention" means any invention in the sense of the
European Patent Convention the performance of which involves the use of a
computer, computer network or other programmable apparatus and having in its
implementations one or more non-technical features which are realised wholly
or partly by a computer program or computer programs, besides the technical
features that any invention must contribute;
(b) "technical contribution", also called "invention", means a contribution to
the state of the art in technical field which is not obvious to a person skilled in the art . The technical character of the
contribution is one of the four requirements for patentability. Additionally,
to deserve a patent, the technical contribution has to be new, non-obvious,
and susceptible of industrial application.
(c) "technical field" means an industrial application domain requiring the
use of controllable forces of nature to achieve predictable results.
"Technical" means "belonging to a technical field". The use
of natural forces to control physical effects beyond the digital
representation of information belongs to a technical field. The processing,
handling, and presentation of information do not belong to a technical field,
even where technical devices are employed for such purposes.
(d) "industry" in the sense of patent law means "automated production of
material goods";
To get an idea of just how many politicians 'got it', and the whole tone of the debate, read the transcript at http://swpat.ffii.org/papers/eubsa-swpat0202/plen0 309/deba/, or watch the recorded Real Player stream from the EU Parliament in any of a dozen languages (until it gets slashdotted).
BTW: sorry about the formatting, but it's not too bad if you narrow the page.
The speeches by Framm, Boussa, Andersson, Cappato, Gebhardt, Boogert, Courtrand and McCormick show just how strongly the message got through.
This is a real credit to all the people who contacted their MEPs -- read what McCormick (not actually McCarthy) has to say about you!
This isn't the final law yet. As Halo1 posted, it is only the first reading, and will now get negotiated between the parliament, commission and member states, before coming back to the parliament for second reading.
Article 6a, which the parliament voted for today, reads:
a) Member States shall ensure that wherever the use of a patented technique is needed for the sole purpose of ensuring conversion of the conventions used in two different computer systems or network so as to allow communication and exchange of data content between them, such use is not considered to be a patent infringement.
This text would apply to all patents, whether granted already or not.
This was the instant comment from slashdotter Halo1, who was in the Parliament all last night and this morning, on the spot as the vote happened:
Tino is sending a full list with results.
However, we got the full article 2
(2a and 2b from kauppi, PSE 69 + non-conflicting part from 55/97/108. We also
have the industrial definition!
Art 3 is deleted, not amended
Art 4 is the biggest loss: for 4.1 and 4.2, the commission proposal has been
voted. 4.3 is 110 somewhat amended ("compromise" Kauppi, but the compromise
does not change the meaning in any way).
Art 5 is 102/111 (and 18 killed).
Art 6a is 76(1), without 76(2), so we got interoperability.
We lost most recitals, except for deletion of recital 6 (so no modification
by NGL though) and also most other smaller amendments to the articles.
So all in all, we sort of crushed the backbone of the proposed directive. I
think we have a very strong start for the second reading.
Slashdot Mirror
FFII's lastest political situation report has some pretty choice things to say about about it:
and, later on:Actually, in the EU Antitrust case, it is.
The lead complaint was made by Sun, complaining that MS was deliberately locking out Sun file and print servers from interoperation with Windows PCs.
Note however that the information will be available only to those who pay for a licence; and who accept MS's terms and conditions.
"The draft decision says the US firm must share more of its protocols with rivals, charging a reasonable royalty. It will be left to Microsoft to work out the precise solution, with close oversight by the commission". -- Guardian
The number of IP licences taken out actually fell after they were mandated by the US judgment.
A better solution would be for to require the protocols be openly published, with their value for non-monopolistic use netted off against the fines the EU wants to impose.
The penalty for misusing protocol specifications to reinforce a monopoly should be the loss of control of those specifications.
Umm, no. That was more or less what the original EU Commission text proposed.
But if you re-read paragraph 1 carefully, it now says that the orders shall be granted "in particular" under those circumstances.
What it actually requires is that judges must be allowed to grant such orders at their own discretion, "on application by a party who has presented reasonably available evidence to support his claims that his intellectual property right has been infringed or is about to be infringed".
The directive in fact overrules existing safeguards which say that the orders can only be granted when there is irreparable harm or destruction of evidence likely.
In these countries the orders are indeed granted directly to the plaintiffs, in secret, without the defendants' case being put, authorising the plaintiffs themselves to go ahead.
Here's what a standard thousand-page textbook on UK Intellectual Property law has to say about such measures, called "Anton Piller" orders (Cornish & Llewellyn, 5e, 2003: section 2-43, page 82):
True, but there are still 30,000 granted software patents in Europe, and the industry giants are pushing very hard on the EU Council of Ministers to reverse the Parliament vote. The implications are potentially very frightening, if the sort the enforcement measures proposed in this Directive become available for use by any agressive litigation company acting on the basis of a dodgy software patent. So I think Aurix's comment connecting this with software patents is right on the mark.
As to FFII's position: FFII is strongly in favour of copyright, and FFII is strongly against piracy.
We think copyright is the right way to build a fair information infrastructure. We are vehemently against commercial software being ripped off and placed on warez networks; and we are vehemently against GPL software being ripped off by closed source companies. But there has to be a balance between the interests of supposed rightsholders (often very powerful) and the interests of other creative companies against wrongful claims of infringement.
It's not just about patents. We're also very concerned about the possibility of these measures being used in disputed copyright cases, disputed trademark cases, disputed cases about claimed confidential information... etc, etc, etc. Under Article 2, the Directive could be applied to "any [claimed] infringement of intellectual property rights as provided for by Community law and/or by the national law of the Member State concerned" -- ie at the moment the whole of Intellectual Property law could be grounds for initiating these measures.
That is why we believe it would be safest if the directive were to be sharply reined in, and apply only to commercially organised, knowingly intentional copyright and trademark infringement.
Europe isn't entirely a legal black hole, you know.
It's worth noting Article 2.1:
Recitals 4 and 5: and Article 20: The point is, that this directive goes far beyond what was agreed at TRIPS in promoting the interests of supposed rightholders. (It is "TRIPS-plus" in the jargon, or "the DMCA on steroids", according to Ross Anderson in Cambridge).The other point is that it is absolutely against any idea of good lawmaking for rightsholders to try to crash this through all its remaining Parliamentary stages in three weeks flat.
It's fine to oppose something on principle but the FFII's alert doesn't seem to be proposing any alternative solution, other than "not what you've got", which weakens their stance IMHO. If they want to make a stand on IP, then they should do so instead of just being naysayers.
You might like to look at the amendments being canvassed by Andreas Dietl of European Digital Rights (EDRi), which you can find on the FFII website.
Strangely enough, this is just what Janelly Fourtou told us in Strasbourg last week, just after she tried to persuade MEPs that we were talking about an out-of-date draft of the Directive -- which we weren't; and just before she told us that she too has been trying to take patents out of the Directive -- which, according to minutes of recent 'Trialogue' meetings, she hasn't. (Mme Fourtou is the MEP who is steering this through the Europarliament, and entirely coincidentally happens to be the wife of the CEO of Vivendi-Universal)
It simply isn't true.
Anton Piller orders are currently only available in the UK and France ("saisi-contrefacon"). These secret court authorisations of raids for evidence carried out by the plaintiff's own agents are not available in any of the other states of the EU.
Furthermore, after very strong criticism from the most senior judges, in the UK a strict new code of practice was brought in in the early 90s which cut the number of applications granted by a factor of ten. (See this page for references to the detailed cases). The judge who led that criticism, Lord Scott, was subsequently head of civil justice for five years, and is now one of the Lords of Appeal in the House of Lords -- the most senior court in the UK. He now chairs the House of Lords scrutiny committee which has refused to clear this legislation. If he is concerned about the detailed text, then we all should be.
We are talking about unannounced dawn raids by private security firms, piling in with legal authority and seizing entire computer systems and filing cabinets full of documents. That is a terrifying and destructive experience for a small firm.
That is why FFII is arguing that such measures should only be available in the most extreme circumstances, and where there is clear evidence of a deliberate knowing intent to infringe for commercial gain on a commercial scale. Such measures are totally inappropriate where there is no such deliberate piracy, and no such emergency, in cases as complex as those in patent law and disputed ownership of confidential information/trade secrets, which routinely can take five years in court. Such measures should not become automatic standard procedure in all IP disputes.
Furthermore, we think it is simply insane to try to crash through such a major change in the civil justice system -- a truly massive change in the whole legal IP environment for most firms in Europe -- in three weeks flat from publication of the detailed text to final vote in Plenary, short circuiting all the normal three readings procedures of the Parliament, and before even first publication of the results of the UK consultation and the UK impact study.
No, this is not just FUD.
The European Patent Office has granted something like 30,000 software patents over the last 20 years.
But that has been done, without legislative approval, by the EPO re-interpreting the rules to mean diametrically opposite to what was originally intended.
It's applications like this one from Microsoft which make the current legislative battle in the European system, which will finally write the official law on this, so vitally important to win.
This is only a patent application, not (yet) a granted patent (in fact in the EU the patent application has only just been published, on 2 Jan this year).
From the 'priority number' (US20020187060 20020628) it looks as though the original application was for a US patent, filed some time in 2002. So that is the cut-off date for prior art.
The full paperwork file for the EPO patent application can also be viewed, at EPOline.
"A Quantum Theory of Internet Value" by Andrew Orlowski
-- why librarians are better at finding the book you want than Google.
I'm not sure that's the whole answer.
Many sites may not have a robots.txt file, yet may still value their copyright and/or database rights.
On the other hand, for some purposes it may be legitimate to take some amount of data (obviously not the whole site), even in contravention of the wishes of a robots.txt
So I think the question is deeper than just "look for robots.txt"
In the USA, trading information that has cost somebody else time and money to build up can be caught under a doctrine of "misappropriation of trade values" or "unfair competition", dating from the INS case in 1918.
Meanwhile here in Europe, a collection of data has full authorial copyright (life + 70) under the EU Database Directive (1996), if the collecting involved personal intellectual creativity; or special database rights (last update + 15 years) if it did not.
I've done a little screen-scraping for a "one name" family history project. Presumably that is in the clear, as it was for personal non-commmercial research, or (at most) quite limited private circulation.
But where are the limits ?
How much screen-scraping can one do (or advertise), before legally it becomes a "significant taking" ?
Note: the forces of darkness are already circling around the amended directive and calling for it to be killed.
BTW: sorry about the formatting, but it's not too bad if you narrow the page.
The speeches by Framm, Boussa, Andersson, Cappato, Gebhardt, Boogert, Courtrand and McCormick show just how strongly the message got through.
This is a real credit to all the people who contacted their MEPs -- read what McCormick (not actually McCarthy) has to say about you!
Article 6a, which the parliament voted for today, reads:
This text would apply to all patents, whether granted already or not.Article 2 = Fundamental definition of "technical": what is patentable and what is not. OUR DEFINITION ACCEPTED.
Article 3 = All software by definition patentable. KILLED.
Article 4 = Detailed conditions for deciding patentability. AMENDED. Will now be re-negotiated between the Parliament, Commission and Member States.
Article 5 = Program Claims. KILLED.
Article 6a = Right to use of patented techniques, without authorisation or royalty, if needed solely to achieve software interoperability. UPHELD.
This was achieved against massive counter-lobbying from the BSA and other industry giants.