cpan.org · Domains · Slashdot Mirror

Re:Contradicts the terms of use by fanatic · 2002-04-06 14:28 · Score: 2 · on Google to Offer API

No Automated Querying: You may not send automated queries of any sort to Google's system without express permission in advance from Google.

The mean it. The version of WWW::Search::Google on CPAN sends a User-Agent header of "WWW::Search/2.33", which Google rejects, saying: "Your client does not have permission to get URL...". Changing the User-Agent to "lynx" allows the query to be answered.

Of course, the question of how their terms of service become binding upon us is yet to be answered. They say: By using Google's search engine services ("Google Search Services"), you agree to be bound by the following terms and conditions (the "Terms of Service"). Yeah, right. I don't accept that from microsloth, and I don't accept it from Google.

Re:Cool! by Shiny+Metal+S. · 2002-04-06 13:55 · Score: 2 · on Google to Offer API

For my high school senior project I wrote a Java program that made specific searches on google, and parsed the results. I spent 3/4 of my time perfecting the nasty string manipulation to strip out the HTML and isolate indivisual results, urls, etc. in my own databse.

I think your high school programming teacher was one of those Java programmers with a sense of humor. There are languages where you just have to write one line, and this is only when you want to reinvent the wheel.

m|^<p><a href="?([^">]+)"?>(.+?)</a>| and print "$2\n$1\n\n" while <>;

Those Java programmers really does have a sense of humor... Or maybe he was one of those CIPA guys, because if you had used Perl, you would have spent 3/4 of your time watching pr0n.

Re:This is the beginning of the revolution by Shiny+Metal+S. · 2002-04-06 12:40 · Score: 2 · on Google to Offer API

Yeah, but shouldn't those modules be re-written, or at least patched, every time Google decides to change the HTML format of the result page?

I don't remember any major HTML change since I started using Google few years ago. See the WWW::Search::Google change log, I thing that about one minor HTML change per year is not much.

I think that accessing the results using a standard API would be much better.

I didn't say it's not better, of course it is. I just said it's not any revolution. People could write their own programs searching Google using WWW::Search::Google since 1999, and before that using just a simple regex.

To demonstrate that it's not a rocket sciense to parse Google results, I just wrote this quick and dirty, nasty hack in the command line:

shiny@c10:~$ GET -H 'User-Agent: Mozilla' http://www.google.com/search?q=perl | perl -ne 'print "$2\n$1\n\n" if m|^<p>\s*<a href="?([^">]+)"?>(.+?)</a>|i' | sed 's/<[^>]*>//g' > perl-links.txt

(it's all in one line) And this is what I have in my new perl-links.txt file:

Perl.com: The Source for Perl -- perl development, perl ...
http://www.perl.com/

www.perl.com/perl/
http://www.perl.com/perl/

Perl Mongers
http://www.perl.org/

CPAN
http://www.cpan.org/

et cetera...

In 180-character command line I wrote a Google frontend, and the parser itself takes only 75 characters including "perl -ne", so it's a real one-liner. I'm sure I could optimize the whole damned thing to less than 100 characters. But why the hell I'm writing this?! I must be very tired... I guess, my point is, that I don't need no stinkin API! (Note to self: must decrease daily 1,3,7-trimethylxanthine dose below LD50 as soon as possible)