Google to Offer API

philipx writes "From the ruby-talk archives here's a little interesting snippet from a post you have to check out: "Here at Google, we're about to start offering an API to our search-engine, so that people can programmatically use Google through a clean and clearly defined interface, rather than have to resort to parsing HTML." It goes on talking about SOAP and I think this is utterly cool."

Comments

Cool, but....

[Patman]

This is very cool, but how long will it last? How will Google make many(and by extension, stay open) when you don't even have to visit their site?

Re:Cool, but....

[Hammerikaner]

Considering most of their money is made through licensing their search technology to other searching outfits and businesses, they shouldn't have any trouble making money.

Re:Cool, but....

[Restil]

That's the beautiful thing about a site that doesn't fund themselves through the use of banner ads. It doesn't MATTER if you access their content through the main website, an affiliate like yahoo, or an api interface.

-Restil

Re:Cool, but....

[NullStream]

If you look at that snippet of Ruby code there you can see that there is a field for a Key of some sort. I'm assuming google will sell you this service and provide you with a key in which you would use it. I know absolutely nothing about ruby (other than it's name) though this is the first thing that came to mind when I saw that code.

Re:Cool, but....

[cxgd]

I thought they made their money by selling the first few search result slots. So they will still make money regardless of how you access the data.

Re:Cool, but....

[jareds]

That's the beautiful thing about a site that doesn't fund themselves through the use of banner ads. It doesn't MATTER if you access their content through the main website, an affiliate like yahoo, or an api interface.

They sells text ads on their main site. Yahoo pays them a fee. So how will the API users pay for themselves?

Re:Cool, but....

[jayant_techguy]

ya, that is ok but what about the ads google has. if by chance someone service utilizes the API in such a way that google has its popularity decreasing then what? well i agree the probability is remote, but not 0 also.

Re:Cool, but....

[1Oman]

The search will probably still return their sponsored links first. So this could actually increase the value of paying for a sponsored link.

Re:Cool, but....

[Amit J. Patel]

No, they don't sell any of the search result slots. They sell ads above and to the right of the search results, but these do not affect the search results themselves.

- Amit

Re:Cool, but....

[inepom01]

If google returns paid links as well as non-paid ones, they would have to indicate it in the search results. Then, the program pulling in the search results could simply filter those out. Chances are, google would be able to make a legal clause for the use of the API but it would still be pretty damn hard to track. The other choice would be not to indicate which listings are paid and which aren't- but this would really confuse the people using the search results and wouldn't be too great at all.

Re:Cool, but....

[gewalker]

If you think Google is concerned about the loss of ad revenue from the API format, you have forgotten what Google said about the alternative of "parsing the html".

Don't know about you, but I doubt the ads silently dumped during in the html parser phase are doing the advertiser much good. Google could charge more for their ads since they would know that html parser "ad bypass" would not be occuring en masse, since any intelligent googler would write to the API.

Beside, why shouldn't the API support ads also, think about the API ads being interpreted as quality hits when the API specifically requests the ad content.

Google should also rack up savings on CPU utilization, especially for Google Games or GoogleWhacking.

Re:Cool, but....

[Paradise Pete]

They sells text ads on their main site. Yahoo pays them a fee. So how will the API users pay for themselves?

They'll get the placement ads like everybody else, and now Google will have people writing their own programs to retrieve those ads. This also helps Google to be the default search engine.

Re:Cool, but....

[yintercept]

If Google charges for the service, then the people who use the API will have to find some way to make money to pay the charge. The other alternative is for Google to compromise the integrity of content and deliver page links with ads. Of course, if Google doesn't charge then they go under. Anyone foolish enough to have written code around the API will now have broken, defunct pages.

Down with the capitalist pigs. Up with living in dank caves.

Re:Cool, but....

[ubernostrum]

Well, subscriptions to allow use of the API could work. There's also the possibility of *gasp* licensing fees - you don't really think Google's income all comes from text ads, do you? They've been licensing their search to people for a good long while...

Re:Cool, but....

[toriver]

In addition to what other posters have said about Google actually NOT being financed by ad revenue, there is probably support in SOAP for paid web services.

Re:Cool, but....

[babbage]

Ahh, but you *are* visiting their site. Think about it. People are already writing scripts to harvest data from search engines, news sites, Amazon, IMDB, Weather.com, etc. This is a useful enough idea that people are coming up with programs to do it even if subtle changes to the page layout breaks things & they have to rewrite the parsers. You can either deny this is happening -- and fight this silly little war of attrition against clever developers doing Interesting Things -- or you can give in, nurture this emerging technology, and oh yeah cut way down on your bandwidth costs because you're letting them slurp up only the content that they want. Win-win.

Yeah, it makes it hard/impossible to make money on it with current models (no ads etc), but it could evolve into a system where sites can exchange data for mutual benefit, giving them all data they need while cutting back on transmission overheads.

Think of it as being ahead of the curve. This is where MS is pushing with SOAP, and others are going with XML-RPC. It can lead us to the point where the "noosphere" of web info can be universally accessed without having to run a clunky old web browser to do it (cell phone, palm pilot, and other alternative access users of the world, rejoice! :).

Google is clever. They're trying to set themselves up as all purpose, all access information scroungers, and they're doing a hell of a good job at it while making a nice profit besides. I don't doubt that they'll be able to parlay this into a revenue stream if it takes off -- just give it a chance to flourish...

Cool feature

[Rock 'N' Troll]

Good idea. By the way, shouldn't /. have a specific "Google" topic?

Re:Cool feature

[Will Collins]

I agree with this suggestion. I suggest the logo be a 'G' from the google logo.

Re:Cool feature

[dodald]

I just quickly made a few. Here you go.

Re:Cool feature

[Rock 'N' Troll]

NOOOOO what have I done!!! I lost my sexy negative karma :(

Re:Cool feature

[Anonymous Coward]

you fail it!!

Re:Cool feature

[Anonymous Coward]

your skill is not enough.

see you next time.

BYE BYE

DoS Google?

[David Kennedy]

The only problem I can see with this is that there was a recent thread on here about Google blocking a lump of IP addresses as someone in there was automatically querying way too often and affecting their load.

With the exposed API I could see, by malice or sheer accident, floods of queries coming in...

Re:DoS Google?

[RatOmeter]

I wonder as well. I've used my own C/GUI front end for Google searches, mainly to automate finding and downloading PDF, gz and other binaries. But I use that much less frequently than I do normal browser based searches so I don't make the slightest spike on their BadBoy meter.

I wonder if they have a [business] plan for this feature or if it's just the brain child of their gurus in the back room. Actually, I'm quite intriqued by the possibilities, so I'm going to keep on eye on it...

-

Re:DoS Google?

[Otterley]

In order to DoS Google it doesn't really matter whether you bang on the front door or the back door.

In fact, an attack through the front door will be more likely to succeed because you're hitting the rendering engine, which takes a lot more CPU time (believe it or not) than the search engine.

OTOH the back door is lightweight and is as such advantageous for not only third parties but also Google itself to employ.

Besides, if you're being abused, if you don't want to use technological avenues to keep miscreants away, you can always use legal ones.

Re:DoS Google?

[Anonymous Coward]

"you're hitting the rendering engine, which takes a lot more CPU time (believe it or not) than the search engine."

Its essentially a database with relatively simple queries. Why would anyone be surprised that IO costs utterly dominate?

Re:DoS Google?

[0x0d0a]

I would be surprised.

Rendering a web page (particularly a Google web page) is pretty lightweight. Searching through the contents of a significant portion of the textual Web is another thing, regardless of what neat tricks and preindexing you use.

This is great.

[aozilla]

Text ads... Open standards for content distribution... If only certain other sites would follow...

Re:This is great.

[cpfeifer]

Open standards for content distribution... If only certain other sites...

Ah, but they do

Re:This is great.

[aozilla]

That's just the headlines... I want messages... moderations... articles... everything...

Re:This is great.

[carm$y$]

You can do the obvious thing, or you can use any of the filtering software available:

squid plus ad-zap (my choice) ;

junkbuster ;

proxomitron.

There may be others. Some tweaking required (not for subscribing, obviously :)

Re:This is great.

[aozilla]

I'd be perfectly willing to subscribe if I got XML data. It's not about the ads, as you've said there are plenty of easier ways to get rid of ads (I use mozilla). It's about being able to write a PHP script to show replies to my posts on my PDA. It's about coming up with my own arbitrary moderation scheme (automatically -1 posts with the word "unconstitutional" in it).

Re:This is great.

[Anonymous Coward]

Why don't you parse alterslash? You advertise it in your sig, so I guess you like it already...

Re:This is great.

[maxume]

Google isn't offering an open standard for content distribution. They would most likely offer this as a for fee service, and they certainly aren't saying that users of this service will be able to, for free, offer thier own frontend to the google database to the Public. The very idea of making this service analogous to what you are stealing from slashdot is ridiculous. Sure, you aren't trying to profit from it, but your page would not be possible without slashdot's services, and your actions are possibly reducing thier revenue, which is paramount to theft.

Re:This is great.

[Anonymous Coward]

Is there anything else you'd like with that, oh great master of all things? Perhaps a foot massage and a blow job on the side?

Re:This is great.

[stripes]

Sure, you aren't trying to profit from it, but your page would not be possible without slashdot's services, and your actions are possibly reducing thier revenue, which is paramount to theft.

No, theft is theft. This would be using the offered services. Much like buying food from the University of Maryland food services on your point card, and then giving it to charity for the homeless isn't theft either. (of corse the UofMD food services folks claimed it was though)

Reducing someone's revenue is not theft, even if it involves using someone else's services (so long as you use them within the bounds they have been offered). A Mobil gas station next to a Shell gas station is likely to reduce the Shell's revenue. Not theft. If the Shell is dumb enough to to sell the gas at half price to the Mobil, it still isn't theft. If the Shell sells it half price but with a no resale rider it is a contract violation though.

Re:This is great.

[shayne321]

Text ads... Open standards for content distribution... If only certain other sites would follow...

Apples and oranges... Google's bread and butter is their patented PageRank technology, which they license for what I'm sure is a lot of money. Slashdot, having made the decision to opensource slashcode do not have this option, therefore we're forced to endure banner ads and subscriptions as their only source of revenue. Ironic, eh? The people that screamed so loud about how long it took ./ to release the source for slash are now bitching about subscriptions and banner ads.. Like it or not, if slashcode was proprietary it could be sold and licensed and you wouldn't have to see ads here (or at least not the larger ones). Sourceforge figured this out too late, and are now trying to sell the SourceForge software as a source of income.

Hopefully ./ will wise up and figure out if they ever want to make any real money they'll have to offer a real service.. Like consulting to companies/webmasters to setup slashcode for customers (like MySQL AB does)... Too bad VA Linux went out of the hardware market. I think a pre-configured "Slash Appliance" (sort of like google's Search Appliance) would be cool as hell for companies needing an internal collaboration system. ./ has really missed the boat here, IMHO.

Shayne

Re:This is great.

[aozilla]

The very idea of making this service analogous to what you are stealing from slashdot is ridiculous.

I'm not stealing anything. The site I link to in my signature is not one I run, though it's one I support whole-heartedly.

Re:This is great.

[aozilla]

Apples and oranges... Google's bread and butter is their patented PageRank technology, which they license for what I'm sure is a lot of money.

And VA Software's bread and butter is SourceForge Enterprise Edition and SourceForge Portal Edition, which I'm sure they license for a lot of money.

The people that screamed so loud about how long it took ./ to release the source for slash are now bitching about subscriptions and banner ads..

Which people would that be? Name one. I certainly never bitched about slash. It's complete shit.

Hopefully ./ will wise up and figure out if they ever want to make any real money they'll have to offer a real service.

Exactly. Banner ads stuck on other people's content isn't going to make a company. Open content licensing the content would drastically reduce the costs. Slashdot is never going to be more than a loss leader, no matter how big the ads get. Unless they revert back to BBS style. 2 or 3 employees, relying on the volunteer help of the community. But it's probably too late for even that. Most of the user's know they sold out.

As John Lennon said, "the dream is over".

Re:This is great.

[Anonymous Coward]

Pray he finds something more useful to do with his time.

Re:This is great.

[maxume]

My bad. Sorry bout that. Whoever runs alterslash is a dirty thief though...

Re:This is great.

[maxume]

Right, but google says not to 'redistribute' or whatever, their service, so a self made front end would essentially be theft, if it was used by someone other than self...

Re:This is great.

[stripes]

Right, but google says not to 'redistribute' or whatever, their service, so a self made front end would essentially be theft, if it was used by someone other than self...

No, that is still not theft, it is a contract violation. Not everything bad is theft. Spray painting my house isn't theft, even if it lowers the property value of my house, or is done just before an open house and in fact costs me potential buyers.

Kicking my dog wouldn't be assault either, it would be animal cruelty (and if I catch you kicking my dog, you may get to file an assault charge against me).

Re:This is great.

[aozilla]

Whoever runs alterslash is a dirty thief though...

And I think it's a perfect example of fair use... The purpose and character is non-commercial news and education. The nature is a published document freely distributed over the internet. The amount and substantiality is minor; slashdot owns copyright in about 1% of the work. The effect of the use on the potential market is minor.

This is the beginning of the revolution

[astrashe]

This is really fantastic. I can already think of a dozen scripts or so that I'd like to write to take advantage of this. I love the fact that this is from a Ruby list, and it's about Google. It's not MSDN and MSN.

They'll need a business model of some sort -- without the ads, and with the potential this has to hammer their servers, they'll need to meter access to the API in some way. But I'll pay -- where do I sign up?

I'll bet that this is how they'll end up making most of their money a couple of years from now.

Re:This is the beginning of the revolution

[Carnage4Life]

It's not MSDN and MSN.

I'm curious as to whether people would actually want such functionality from MSDN. It's one thing to be able to do a Google search from a function call and get the results back as XML but do people want API docs and technical articles retrieved via getArticle() and getAPI() webmethods?

One place where it might be useful however is KnowledgeBase articles. Perhaps a web service that retrieves a KB article given the Q number (e.g. Q123456) might be useful.

Disclaimer: This post is my opinion on doesnot reflect the thoughts, strategies, intentions or opinions of my employer.

Re:This is the beginning of the revolution

[IamTheRealMike]

One way I'd consider of making money if I were google would be to merge the AdWord links with the top results so they are indistinguishable. If you have a paying account with them, you can opt to "clean up" the results.

Re:This is the beginning of the revolution

[Shiny Metal S.]

This is really fantastic. I can already think of a dozen scripts or so that I'd like to write to take advantage of this.

I see that most of people probably don't know that there are already Perl modules on CPAN to search Google and even to use Google cache:

• WWW::Search::Google by Jim Smyser
• WWW::Cache::Google by MIYAGAWA Tatsuhiko
• Apache::No404Proxy::Google by MIYAGAWA Tatsuhiko
• WWW::Search::Scraper::Google by Glenn Wood

See also:

WWW::Search::AOL::Classifieds::Employment, WWW::Search::AP, WWW::Search::AlltheWeb, WWW::Search::AltaVista, WWW::Search::AltaVista::AdvancedNews, WWW::Search::AltaVista::AdvancedWeb, WWW::Search::AltaVista::Careers, WWW::Search::AltaVista::Intranet, WWW::Search::AltaVista::Intranet3, WWW::Search::AltaVista::NL, WWW::Search::AltaVista::News, WWW::Search::AltaVista::Web, WWW::Search::Brassring, WWW::Search::CraigsList, WWW::Search::Crawler, WWW::Search::Dice, WWW::Search::Ebay, WWW::Search::Ebay::ByEndDate, WWW::Search::EuroSeek, WWW::Search::Excite::News, WWW::Search::ExciteForWebServers, WWW::Search::Fireball, WWW::Search::FirstGov, WWW::Search::FolioViews, WWW::Search::Go, WWW::Search::GoTo, WWW::Search::Google, WWW::Search::Gopher, WWW::Search::HeadHunter, WWW::Search::HotBot, WWW::Search::HotFiles, WWW::Search::HotJobs, WWW::Search::Livelink, WWW::Search::LookSmart, WWW::Search::Lycos, WWW::Search::MSIndexServer, WWW::Search::MetaCrawler, WWW::Search::Metapedia, WWW::Search::Monster, WWW::Search::NetFind, WWW::Search::Newsbytes, WWW::Search::Nomade, WWW::Search::NorthernLight, WWW::Search::Null, WWW::Search::OpenDirectory, WWW::Search::PLweb, WWW::Search::PRWire, WWW::Search::PubMed, WWW::Search::RpmFind, WWW::Search::SFgate, WWW::Search::Scraper, WWW::Search::Scraper::BAJobs, WWW::Search::Scraper::BayAreaHelpWanted, WWW::Search::Scraper::Beaucoup, WWW::Search::Scraper::Brainpower, WWW::Search::Scraper::CraigsList, WWW::Search::Scraper::Dice, WWW::Search::Scraper::Dogpile, WWW::Search::Scraper::FieldTranslation, WWW::Search::Scraper::FlipDog, WWW::Search::Scraper::Google, WWW::Search::Scraper::JustTechJobs, WWW::Search::Scraper::Lycos, WWW::Search::Scraper::Monster, WWW::Search::Scraper::NorthernLight, WWW::Search::Scraper::Request, WWW::Search::Scraper::Request::Job, WWW::Search::Scraper::Request::ZIPplus4, WWW::Search::Scraper::Response, WWW::Search::Scraper::Response::Auction, WWW::Search::Scraper::Response::Job, WWW::Search::Scraper::Response::Sherlock, WWW::Search::Scraper::Sherlock, WWW::Search::Scraper::TidyXML, WWW::Search::Scraper::ZIPplus4, WWW::Search::Scraper::apartments, WWW::Search::Scraper::computerjobs, WWW::Search::Scraper::eBay, WWW::Search::Scraper::techies, WWW::Search::Scraper::theWorksUSA, WWW::Search::Search97, WWW::Search::Simple, WWW::Search::Snap, WWW::Search::Test, WWW::Search::Translator, WWW::Search::Verity, WWW::Search::VoilaFr, WWW::Search::WashPost, WWW::Search::WashTech, WWW::Search::Yahoo, WWW::Search::Yahoo::Classifieds::Employment, WWW::Search::Yahoo::Japan::News, WWW::Search::Yahoo::Korea, WWW::Search::Yahoo::News::Advanced, WWW::Search::Yahoo::UK, WWW::Search::YahooNews, WWW::Search::ZDNet and WWW::SearchResult.

As you can see, it's already possible to use lots of different search engines in your own scripts. Of cource it's great news about Google API, but I wouldn't exactly call it a revolution. You could write a dozen of your scripts a very long time ago.

Re:This is the beginning of the revolution

[plaa]

They'll need a business model of some sort -- without the ads, and with the potential this has to hammer their servers, they'll need to meter access to the API in some way. But I'll pay -- where do I sign up?

Another option is to give better access to paying customers: a paying customer is given unlimited use of the search, while private individuals (distinguished via IP/registration/...) would be limited to, say, one search per 5 seconds. It would be great to be able to use this API for some small things without having the hassle of paying. A 5-10 second delay isn't very bad in a small home situation, but is out of the question for any larger-scale applications.

I'd say it would also be consistent with their current user-friendly business model, and give another jolt of good PR for them.

Re:This is the beginning of the revolution

[dytin]

That's a great idea! You should work for google.

Re:This is the beginning of the revolution

[at_18]

I see that most of people probably don't know that there are already Perl modules on CPAN [cpan.org] to search Google and even to use Google cache:

* WWW::Search::Google [cpan.org] by Jim Smyser [cpan.org]
* WWW::Cache::Google [cpan.org] by MIYAGAWA Tatsuhiko [cpan.org]
* Apache::No404Proxy::Google [cpan.org] by MIYAGAWA Tatsuhiko [cpan.org]
* WWW::Search::Scraper::Google [cpan.org] by Glenn Wood [cpan.org]


Yeah, but shouldn't those modules be re-written, or at least patched, every time Google decides to change the HTML format of the result page?
I think that accessing the results using a standard API would be much better.

Re:This is the beginning of the revolution

[Anonymous Coward]

One way I'd consider of making money if I were google would be to merge the AdWord links with the top results so they are indistinguishable. If you have a paying account with them, you can opt to "clean up" the results.

That's a good way to destroy your ad revenue stream. There are several other engines which are fairly good, and if Google sells their rankings, their search result quality will invariably decrease, and the non-paying customers will move to the competitors. Most people will not pay for a service when there are free alternatives that are almost as good.

Re:This is the beginning of the revolution

[Shiny Metal S.]

Yeah, but shouldn't those modules be re-written, or at least patched, every time Google decides to change the HTML format of the result page?

I don't remember any major HTML change since I started using Google few years ago. See the WWW::Search::Google change log, I thing that about one minor HTML change per year is not much.

I think that accessing the results using a standard API would be much better.

I didn't say it's not better, of course it is. I just said it's not any revolution. People could write their own programs searching Google using WWW::Search::Google since 1999, and before that using just a simple regex.

To demonstrate that it's not a rocket sciense to parse Google results, I just wrote this quick and dirty, nasty hack in the command line:

shiny@c10:~$ GET -H 'User-Agent: Mozilla' http://www.google.com/search?q=perl | perl -ne 'print "$2\n$1\n\n" if m|^<p>\s*<a href="?([^">]+)"?>(.+?)</a>|i' | sed 's/<[^>]*>//g' > perl-links.txt

(it's all in one line) And this is what I have in my new perl-links.txt file:

Perl.com: The Source for Perl -- perl development, perl ...
http://www.perl.com/

www.perl.com/perl/
http://www.perl.com/perl/

Perl Mongers
http://www.perl.org/

CPAN
http://www.cpan.org/

et cetera...

In 180-character command line I wrote a Google frontend, and the parser itself takes only 75 characters including "perl -ne", so it's a real one-liner. I'm sure I could optimize the whole damned thing to less than 100 characters. But why the hell I'm writing this?! I must be very tired... I guess, my point is, that I don't need no stinkin API! (Note to self: must decrease daily 1,3,7-trimethylxanthine dose below LD50 as soon as possible)

Re:This is the beginning of the revolution

[dillon_rinker]

technet.microsoft.com, perhaps? And not just the articles themselves, but the ones that reference them.

Re:This is the beginning of the revolution

[Anonymous Coward]

One place where it might be useful however is KnowledgeBase articles [microsoft.com]. Perhaps a web service that retrieves a KB article given the Q number (e.g. Q123456) might be useful.

You mean like how you can just type the following in IE to get Q123456?:

mskb Q123456

Re:This is the beginning of the revolution

[0x0d0a]

With the SOAP results, not the HTML ones, silly.

Re:This is the beginning of the revolution

[Anonymous Coward]

It seems that you probably don't know that using those modules is a violation of Google's Terms of Service unless you get their permission first. And using the Google API is presumably a lot more efficient and reliable since you'd be getting output that was designed to be parsed for search results, not scraping HTML that's really meant for browser/human consumption.

Cool!

[geddes]

For my high school senior project I wrote a Java program that made specific searches on google, and parsed the results. I spent 3/4 of my time perfecting the nasty string manipulation to strip out the HTML and isolate indivisual results, urls, etc. in my own databse. Had the API come out two years ago, I would have spent a lot less time on that thing. Way to go Google!

Could this be in response to the supposed competition from tokohma? open up thier results in some way to increase thier usage?

Re:Cool!

[Anonymous Coward]

Had the API come out two years ago, I would have spent a lot less time on that thing.

Had you enough sense to use Perl, you would have spent a lot less time on that thing. What crack smoking teacher would force someone to do string manipulation in Java...sadistic.

Re:Cool!

[Anonymous Coward]

I did a similar string manipulation script in Perl, and this guy is more than right. I was done in a matter of minutes. I was also 11 when I wrote the thing. *smile* The power and ease of Perl!

-James Skarzinskas.

Re:Cool!

[Anonymous Coward]

Java 1.4 supports regular expressions.

Re:Cool!

[Anonymous Coward]

whoopdefuckingdoo.

Re:Cool!

[XBL]

You could have ran the data through an HTML parser into a DOM tree. Much more easy to work with the data there!

Re:Cool!

[frisket]

Interesting to note that this is part of what HTML was originally intended to do: provide one possible standard way of identifying information (like <title> was supposed to hold the title of the document), with SGML being used to let people develop more specialist applications.

It got perverted along the way, but hey, it's only taken us a decade to get back on track. So when will /. be available as a Web Service? C'mon guys, drop the mod_perl stuff, just give us the XML...

///Peter

Re:Cool!

[Shiny Metal S.]

For my high school senior project I wrote a Java program that made specific searches on google, and parsed the results. I spent 3/4 of my time perfecting the nasty string manipulation to strip out the HTML and isolate indivisual results, urls, etc. in my own databse.

I think your high school programming teacher was one of those Java programmers with a sense of humor. There are languages where you just have to write one line, and this is only when you want to reinvent the wheel.

m|^<p><a href="?([^">]+)"?>(.+?)</a>| and print "$2\n$1\n\n" while <>;

Those Java programmers really does have a sense of humor... Or maybe he was one of those CIPA guys, because if you had used Perl, you would have spent 3/4 of your time watching pr0n.

Re:Cool!

[p3d0]

That's your fault man. Java is the wrong language for this kind of text processing. If you like OO, you should have tried Python.

Contradicts the terms of use

[Anonymous Coward]

Google Terms of Service. Some excerpts:

Personal Use Only [...] You may not take the results from a Google search and reformat and display them

No Automated Querying: You may not send automated queries of any sort to Google's system without express permission in advance from Google.

So how useful might that API be if you can't do anything with it...

Re:Contradicts the terms of use

[red_dragon]

I'd assume that the API would be subject to a different set of terms and conditions than those for the main site. Given that it'll probably be a pay-for-use service (as another poster hinted at), it'd most certainly be that way.

Re:Contradicts the terms of use

[Anonymous Coward]

You can get permission if you ask instead of just giving up =)

Re:Contradicts the terms of use

[LordNimon]

You could write a program that would use Google as its back-end search engine for the Internet. That program could be sold and anyone can use it on their computer. For instance, I can imagine Apple using that API for their Sherlock search application. Just because the software that uses it is distributed doesn't mean that it violates the license. As long as the results are not redistributed (i.e. in your own public web site), and the search is initiated only upon request from the user and it not some kind of cron job, then it's okay. Apple's Sherlock and Mozilla's search tools both conform.

Re:Contradicts the terms of use

[RatOmeter]

To quote my own post (above):
"I wonder if they have a [business] plan for this feature..."

The more I think about it, the less I think this is going to be a "free" feature. It seems to encourage "automated queries" and that doesn't fit with any business model I've thought of (for Google) and, as you've pointed out, it's against their TOS.

I'm guessing that, if this feature comes to fruition and stays, its usage will come under some sort of paid model.
-

Re:Contradicts the terms of use

[Anonymous Coward]

It seems to encourage "automated queries" and that doesn't fit with any business model I've thought of (for Google)

What about pay for search result placement (with the understanding that this would only be a "feature" of non-paying API users).

Re:Contradicts the terms of use

[augustz]

Um, doesn't it say you can do something with it, or did I miss a statement about personal use somehow?

I suspect that, despite the outcry and outrage from some quarters, they're not simply going to give away their entire search engine API connected to their search farm. Perhaps they'll limit it, meter it, and even charge for it. All would be more then fair.

Re:Contradicts the terms of use

[Decimal Dave]

No Automated Querying: You may not send automated queries of any sort to Google's system without express permission in advance from Google.

Great...there goes my plans to use Google's database as training data for the neural network in my trolling AI.

Re:Contradicts the terms of use

[motox]

Because you get free publicity on slashdot

Re:Contradicts the terms of use

[fanatic]

No Automated Querying: You may not send automated queries of any sort to Google's system without express permission in advance from Google.

The mean it. The version of WWW::Search::Google on CPAN sends a User-Agent header of "WWW::Search/2.33", which Google rejects, saying: "Your client does not have permission to get URL...". Changing the User-Agent to "lynx" allows the query to be answered.

Of course, the question of how their terms of service become binding upon us is yet to be answered. They say: By using Google's search engine services ("Google Search Services"), you agree to be bound by the following terms and conditions (the "Terms of Service"). Yeah, right. I don't accept that from microsloth, and I don't accept it from Google.

This sounds cool, but..

[Idimmu Xul]

Some unscrupulous players could surely abuse this by 'making their own' search engines that essentially rip off google without any hassle what so ever?
Ok, it can be done already, but this would make it possibly too easy...?

Also, this will miss out their ads etc that they get revenue from, I wonder what their long term stratagy is?

Re:This sounds cool, but..

[SerpentMage]

While it is possible practically. I doubt they could pull it off. Lets say you do create a search engine and all of the sudden a huge number of requests come through? What do you do?

Or lets say Google spikes the search request at some competitors to prove they are using Google.

So sure they could do it, but I doubt any popular site could get away with it for long.

Re:This sounds cool, but..

[aozilla]

Some unscrupulous players could surely abuse this by 'making their own' search engines that essentially rip off google without any hassle what so ever?

Yahoo hasn't had enough problems with it to take it down. It's really nice being able to make my own PHP script to display customized stock quotes on my PDA.

Re:This sounds cool, but..

[Kwikymart]

Google has no qualms with denying service to whole ISPs because of a few people from that specific provider abusing their search engine. I was reading somewhere that they denied everyone from Comcast for just that reason. The people at Google are pretty smart and I assume they will have a method for preventing abuse of their SOAP interface. I guess they will just deny anyone who abuses. Also, I assume they will have text adds or something (from seeing some of the other discussion in the /. stories). This move will no doubt allow Google to really entrench their position as the best search engine around.

Re:Cool, but.... They never said if was free!

[BrynM]

They could actually charge for a devkit or usage to break even on the project. Even if it did costsome money, I could see it being well worth the price, if it works well.

I just wonder how it will tie into my app. Will it open my browser? Will the Google Bar plugin be the foundation?

We'll just have to wait and see...

Since when do we like Google?

[Anonymous Coward]

Well?

Re:Since when do we like Google?

[Rayonic]

FYI:

We like them Monday, Wednesday, and Friday.
We love them Tuesday, Thursday, and Saturday.
And we alternate Sundays.

Get with the program.

Re:Since when do we like Google?

[Anonymous Coward]

The other poster said it better, but what the hell are you talking about? Everyone loves Google!

The app I'm working on could use this

[mikosullivan]

I've been writing a bookmarking application that directs the user to Google and later remembers the last Google search so you can resume it. This API will simplify the interface significantly and open up a whole new world of possibilities.

Explanation / Keep Supporting Google

[Anonymous Coward]

The first page I visit every morning
---

The following is the preliminary code that a particular Google sysadmin (ian@) is trying out. He'd prefer to have a single WSDL file do all of the configure (from Google's end to client), but he first needs to get some advice from an experienced Ruby hacker.

Also, let's keep in mind that this API will actually be decreasing Google pageviews and hits, which will in turn make their AdWords, AdWordsSelect, and textads less effective. So, it's our duty to continue to support Google and show them that the free/open source software people are behind them 100%. We know that Teoma just doesn't deliver, and Google's already got 3 billion pages indexed and cached.

Support Google today, because they're the future of information indexing on the Web!

--- begin code ---

#!/usr/bin/ruby

require 'soap/driver'

endpoint = 'http://api-ab.google.com/search/beta2'
ns = 'urn:GoogleSearch'
key = 'xxxxxxxxxxxxxxx'
service = 'file:GoogleSearch.wsdl'
query = ARGV.shift || 'foo'

soap = SOAP::Driver.new(nil, nil, ns, endpoint)

# uncomment the next line to dump the traffic on the wire
#
#soap.setWireDumpDev(STDERR)

soap.addMethodWithSOAPAction('doGoogleSearch', ns, 'key', 'q', 'start',
'maxResults', 'filter', 'restrict',
'safeSearch', 'lr', 'ie', 'oe')
r = soap.doGoogleSearch(key, query, 0, 10, false, nil, false, nil,
'latin1', 'latin1')

printf "Estimated number of results is %d.\n", r.estimatedTotalResultsCount
printf "Your query took %6f seconds.\n", r.searchTime

Re:Explanation / Keep Supporting Google

[quintessent]

let's keep in mind that this API will actually be decreasing Google pageviews and hits, which will in turn make their AdWords, AdWordsSelect, and textads less effective.

Maybe. However, you can already use screen-scraping to query Google today. The difference is that Google will better know the difference between a program and an actual user. So each ad-word view may be more effective. Of course, now that Google offers pay-per-click (in place of pay-per-impression), that might not be an issue anyway.

Will the number of actual ad impressions go down? Could be. But maybe by being a better product, Google will gain popularity and increase impression counts.

As for supporting Google: Definitely; they make an unbeatable product.

Is this a subscription service?

[Idimmu Xul]

key = 'xxxxxxxxxxxxxxx'

I havent tried to get it to work yet, due to not having ruby installed, but does this imply some sort of subscription service?

Possibly a new way for them to raise revenue? Im assuming that the bold line means the authors key has been blanked out so other people cant abuse this service for free?

Lameness filter encountered. Post aborted! Reason: Too much repetition. :/

Re:Is this a subscription service?

[teejie]

I installed ruby just to test this out and got this:

# ./google.ry
#: Exception from service object: Invalid authorization key: xxxxxxxxxxxxxxx (SOAP::FaultError)

So it's just to keep us from using it just yet I guess.

Re:Is this a subscription service?

[Anonymous Coward]

Maybe they aren't offering the service to the public yet, since its still in beta?

Re:Is this a subscription service?

[goolo]

you should replace these xxxs with something like 'pay-dollars-per-1k-search' or 'pay-erver-month'...

but maybe they'll have some more keys:
- 'pay-dollars-for-adfreesearch'
- 'pay-less-for-some-ads'
- 'free-but-only-ads'
- 'pay-or-p155off' - this key will always 'work'

I think this SOAP stuff is a good idea to build you're own search engine. You could refinance this with 'Pay-for-Placement'.

BUT:
1. maybe Overture sues you too (like Overture is doing it with Google)
2. what great idea could you have, that google wouldn't have? - to make people move to you're search engine

OpenGoogle?

[VistaBoy]

Are they going to release the source code to the search engine itself? That would be REALLY cool...

We can finally find out how to implement their PigeonRank system...

Re:OpenGoogle?

[0xB]

You don't need to source code for that, do you? You need the Pigeon genome

Re:OpenGoogle?

[RatOmeter]

"We can finally find out how to implement their PigeonRank system..."

They already described the machanisms fairly well on their site. What you might not have access to is the programming language that's optimized to the task: HP-PGL (High Performance Pigeon Goading Language).

-

Re:OpenGoogle?

[p3d0]

I doubt it. That's their revenue model. If they give away the engine source, what do they have left?

Re:OpenGoogle?

[Inthewire]

Couldn't they sell support?
Oh, I crack me up.

Google topic icon

[Mr. Neutron]

Perhaps a farmer picking apart a haystack, one piece at a time.

Re:Google topic icon

[sfraggle]

or maybe a pigeon :)

Re:Contradicts the terms of use (Moron)

[Anonymous Coward]

You don't suppose they'll modify their terms of service to accommodate the new API do you?

Think, people.

Good to see this idea return

[drok]

Last year Google temporarily had an XML interface available using a query like: http://www.google.com/xml?q=slashdot

Of course, now it's just forbidden. I am surprised they would go back to such a service, it would seem to wind up losing revenue for them depending upon whether or not people are good about passing along whatever Ad-words Google returns. They could expect the traffic to be low enough to not matter compared to the continued word-of-mouth benefit. Or access to the SOAP interface could be offered as a subscription model (pure speculation on my part).

-Robert

Re:Good to see this idea return

[lkaos]

See, with Google, the Ad's are really links that are for the most part relavant to the topic being searched for.

If the results are returned using SOAP, then the backend surely would want to display the ads because a lot of the time, they are what the user is looking for.

I know if I am looking to buy something search Google for vendors, I am more likely to choose a vendor from the Ads on the side. I figure it is a bit safer since these people actually have something invested in it.

The only reason I can think that someone would filter out the ads is simply because they want to hurt Google. Who wants to hurt Google though?

The click through rate is probably going to make things hard since there is no way to tell if a user clicked an ad. That just means a different guage...

Ode to Google :)

[Khalid]

Google has been an enchantment for me since it's beginning !

They have always made the right decision ! they have offered internet users an incredible asset ! and I was so much grateful when they decided to rescue Deja, a site something I just don't know how I can leave without !

I view them as the most "honest and fair" site on the Net ! and without any doubt the most useful too.

Go Google ! you are showing the right way ! to all these stupid-crapy-portal sites which have invaded the net, I just hope you manage to stay in business and prosper for a loooooong, looooong time :)

Re:Ode to Google :)

[ChrisCampbell47]

I Kiss You!

[there goes my karma]

Re:Ode to Google :)

[Anonymous Coward]

Ironically, I'm going to use Google's API to write a program to make a list of the 10,000 articles that I wrote on Usenet when I was younger and stupider, and have them all removed from the database.

Thanks Google!

Re:Ode to Google :)

[Anonymous Coward]

post the code? i'd like to do the same

Very Useful

[DrNibbler]

Using the API and a dictionary I could find the most google smacks.

Re:Cool, but.... They never said if was free!

[Carnage4Life]

They could actually charge for a devkit or usage to break even on the project. Even if it did costsome money, I could see it being well worth the price, if it works well.

I just wonder how it will tie into my app. Will it open my browser? Will the Google Bar plugin be the foundation?

The post describes a SOAP web service which in most cases is an RPC call in your application of choice. However unlike RPC in days of yore using SOAP to do RPC in applications is relatively easy. If you want to learn more about SOAP I suggest reading A GEntle Introduction To SOAP by Sam Ruby for an overview of the protocol and A Busy Developer's Guide to WSDL 1.1 to see how one could go from defining a WSDL file (as the Google sys admin is trying to do) to actually accessing the web service remotely from a Java application.

There is also a grab bag of resources on XML webservices at the .NET Framework community website.

To answer your question, if the Google API is available as a web service then it can be intergrated into any application at all from command line to dynamic web page to GUI application as long as there is network availability on the host machine.

Interesting URL there taco...

[Anonymous Coward]

http://www.ruby-talk.org/cgi-bin/scat.rb/ruby/ruby -talk/37623

SCAT.rb ??? ;)

Re:Interesting URL there taco...

[Anonymous Coward]

.rb is a Ruby script's suffix. this cgi is written in Ruby.

wow -- wish i'd had this earlier

[macsox]

it would have made the creation of my random google searcher a bit easier, and faster.

Pay-per-placement will pay for this...

[Anonymous Coward]

Text ads shmext ads. They can easily be ignored. The thing that will pay for this kind of access is actually Google's pay-per-placement plan. Advertisers will pay for their sites being ranked high, not for their banners being shown to us. Any application that uses this API to search google will return those sponsored results, which is as good as a banner view. Actually, if it's targetted (only sponsored sites that are relevant to your search will be shown), both users and sponsors will be pleased.

Re:Pay-per-placement will pay for this...

[ipsuid]

Google does not have a pay for placement plan - if you are making reference to the practice of changing the order of search results based on advertiser dollars.

That was the very thing that turned people onto Google. I very much doubt that they would change that.

Re:Cool feature GREAT IDEA!

[phoxix]

seriously,

all of us do nothing but rave about google day and night

for it is a search engine we love, with a company many of us have come to love

I for one would love to see google have its own slashdot icon

Come to think of it, there are plenty of USELESS icons none of us give a damn about

the following are a few:

Beanies
E+(huh?)
OS 9

Heres hoping for a new google icon!!

Just my two cents, all taxes included

Sunny Dubey

Re:Things To Do Today

[Anonymous Coward]

3. Do a shit

4. Eat a shit.

uh oh

[terradyn]

They'd better make sure a "clean and clearly defined interface" doesn't violate one of Overture's patents or they may have another suit on their hands...

what about the law suit

[JDizzy]

IS this API going to have A system and method for enabling information providers using a computer network such as the Internet to influence a position for a search listing within a search result list generated by an Internet search engine, because that is what google is being sued for at the momment. Interesting they choose now to release the API. Almost as if they can show that the function is an intrigal part of a different system (by way of this new API), that they have a chance in the courts. I'll let you be the judge!

Soap is good

[Anonymous Coward]

Yes, it would be a very good thing if Google employees used soap. That's pretty disgusting if they don't.

Well duh!

[danrees]

Of course you're going to have clean code if you use a SOAP interface...

Command line google is already here!

[Anonymous Coward]

I thought this was funny as hell.

A hack and a half, but it actually works!

Re:Command line google is already here!

[Anonymous Coward]

Try Surfraw; it does this and much more.

Re:Command line google is already here!

[Kirruth]

Try Surfraw [http://surfraw.sourceforge.net/]; it does this and much more.

Very cool. Makes it very straightforward to automate navigation/downloading of web pages based on a query, a la the Matrix. I need to fish out my dark glasses.

Not publically accessible

[angio]

com.google.soap.search.GoogleSearchFault: Invalid authorization key: xxxxxxxxxxxxxxx
at com.google.soap.search.QueryLimits.lookUpAndLoadFr omINSIfNeedBe(Query
...

Alas, looks like the rest of us won't be able
to play with Google's beta SOAP service. Which makes quite a bit of sense - this would be a great way for Google to allow people to resell Google in a standardized way, be it from inside a program (scary, too easy to reverse engineer) or from some other web service (less scary. :-). It doesn't make much sense for Google to say, "Hey, world, come and use our search services for free without our ads."

Barter worked for a long time...

[Hooya]

..and may work again. I think google is in a unique position where they could make a value proposition by using a combination of a barter system and of course the monetery system. consider this: google could have one of two modes of payment:

• 1) you pay a subscription fee ($/query, or flat fee, a combination of both like the utilities... whatever works best).
• 2) for people that don't want to pay, or cannot afford to pay... put up a barter system. the way that would work is, the subscriber gives up clock cycles (in the SETI@home fashion to build up 'karma' or 'virtual money' that can be used to pay for the subscription system mentioned previously. That way people who cannot or don't want to pay for the service directly in real-dollars can keep their computers on and earn google-dollars to later redeem it.

  • Google benefits from the monetery system in an obvious way. They also benefit from the barter system by vastly adding to the crunch power which hopefully improves their indexing/grading system. Unused clock cycles which would otherwise be wasted can now earn some value for the users and at the same time give google the 'value' for providing their service.

    So their 'open' system if presented in the form of barter could actually work for the advantage of both parties involved.

Re:Barter worked for a long time...

[mrmag00]

I don't think google would want external sources 'improving' on their indexing system. it requires too much overhead on their end for managing all the results, requires putting their software out for public consumption, and they would risk of having their database cluttered by people who fake responses.

On the other hand, providing the API doesn't seem like they are losing out on much. If yahoo offered such a service I would use a quick frontend because I hate their website (and no longer visit it). But why bother, google has a few pictures VERY fast results and non-intrusive ads. the worst someone can do is take away the text ads and cut the size of the page down by a few hundred bytes.

I think they are doing this for either subscription based service in the future or are offering it because they know people will just parse the HTML back into the raw data and reuse it, and this saves them spare CPU cycles of converting to HTML in the first place.

Re:Barter worked for a long time...

[Anonymous Coward]

Google Toolbar will soon incorporate the share of Clock Cycles by the host's system...

No contradiction

[fm6]

The "Personal Use" restriction means that you can't download results for Google than pass them on as your own product. There's no restriction on downloading and reformarting results for your own use. Nor on applications that help you do it. There are already a lot of products that do this -- including plugins for all the major browsers.

On the other hand, Google would obviously not want you to set up your own search site that passes queries to their engine, harvests the results, and presents them on your own site. That is the obvious target of the "Personal Use" restriction.

As for the "Automated Query" restriction -- well, what do you think they mean by "Automated"? Programmatic access to their engine? They couldn't prevent that even if they wanted to. "Automated" obviously means programs that issue hundreds of queries for data mining purpose. Example: crawling the Groups archives to harvest email addresses.

(This was a matter of some concern to me, when I noticed that the Google Usenet archives included all my company's private groups. I'd innocently used by real corporate email, innocently thinking that the groups weren't accessible outside the company. But the spam volume is still very low. Their bot detection software must be quite good.)

Note that making a simple API available doesn't enable any new kind of access to the Google engine. A clever programmer can already parse the HTML results. The API just makes it easier -- and gives Google another product they can sell licenses for.

Re:No contradiction

[LS]

Yes they can prevent it. I wrote a script to browse google's cache, and they somehow detected it and disallow my IP from running it. But I can still query through the browser.

Re:No contradiction

[Anonymous Coward]

Did you fake the user-agent?

Re:No contradiction

[passion]

ote that making a simple API available doesn't enable any new kind of access to the Google engine. A clever programmer can already parse the HTML results. The API just makes it easier -- and gives Google another product they can sell licenses for.

I've written a script to pull from google's search results so that I could have a site-only search function built into my page. I've had to rework it about 3-4 times as they keep adjusting the way to connect.

This is definitely about money, and I don't blame them. As long as their pricing model is decent, then I think they'll make a killing. Information is power, being able to find that information is even more power, and when you hold a monopoly because you're so much fucking better than everyone else... you can command money for your services.

I'll probably buy into it if they come up with a decent pricing scheme. I'd like to see something relating hits to cash, but also have some cut-off switches (so you can protect yourself against runs and slashdottings).

Re:No contradiction

[yintercept]

This is not a complaint. I just want to point out that the whole Google Concept is built on a subtle contradicitons or, dare I say, hypocracy. From the User Agreement:

The search results that appear from Google's indices are indexed by Google's automated machinery and computers

The User Agreement precludes you from automatically querying their site:

You may not send automated queries of any sort to Google's system without express permission in advance from Google.

The google agreement demands express permission to automatically scan its site, while Google assumes the permission to index all sites on the net.

Google also pretty much make the claim that if it is on the Internet they will index it. Their terms of service states that the only way not to have a site indexed is remove it from the net:

For each web site reflected in Google's indices, if either (i) a site owner restricts access to his or her web site or (ii) a site is taken down from the web, then, upon receipt of a request by the site owner or a third party in the second instance, Google would consider on a case-by-case basis requests to remove the link to that site from its indices.

I think Google is providing a great service, but I hope you can see the subtle contradictions in their product. They basically are saying that anything on the web is fair game for Google. Yet, Google is on the web, it is not fair game for other organizations. This is a very blatant double standard.

Google is a derivative work. The product model of Google is to determine expert sites by aggregating the link lists on other expert sites. In other words, they are taking other people's work, aggregating it and providing the results. Google's aggregation program is a derivative work. Not only that, they fail to give any compensation to the expert sites.

As for the issue of intergallactic karma, they actually expect the expert site to pay for the bandwidth needed by Google to aggregate the site. They then use this information to draw human traffic from the expert site.

Again, to the Google worshippers, I am not complaining or flaming Google, but simply pointing out a logical contradiction. Jack's Expert Site is harmed by Google in two ways: The googlebots take up a great deal of bandwidth that Jack pays for. Google then uses this information to draw actual human traffic from Jack's Expert Site. From this vantage Google is a big guy stomping on the small guy. When Microsoft does this type of stuff, we call it evil.

Re:No contradiction

[lexi]

The googlebots take up a great deal of bandwidth that Jack pays for

The googlebot takes up exactly the same amount of bandwith as one user browsing the whole site, once per index cycle, and it doesn't fetch images. How is this a 'great deal' of bandwidth?

Google then uses this information to draw actual human traffic from Jack's Expert Site

Um, no. Google then uses this information to drive actual human traffic to Jack's Expert Site. How exactly were users going to find Jack's Site if Google didn't index it? By word of mouth? By spam email? Google is providing a public service - both to Jack and the viewers looking for his site - for free. Jack should be really freaking happy that Google bothers to visit his site and drive traffic there, because the alternative is paying Yahoo for inclusion in their search results.

Re:No contradiction

[Anonymous Coward]

Um ... you may be doing this the hard way - the Jargon file simply adds the 'site:' modifier to the query, and gets site-specific results back. No HTML parsing required. Description is here.

Re:No contradiction

[yintercept]

First off, I should not have used the words "Great Deal." According to my logs, Google reads a little under 100MB from my sites each month. That's like a dollar each month.

I am not unhappy with Google. I am just pointing out a contradiction. My argument is that in some cases Google will incur a bandwidth cost on a site, and have the affect of taking traffic from the site.

In the Google paradigm there are two types of sites. There are expert and content sites. An expert site is a site with a bunch of links. Content sites have articles.

Google is the ultimate expert site. It aggregates other expert sites. As a result, people use Google rather than the independent expert site.

Remember an expert site is a link site. The goal of the expert site to direct traffic...not to deliver content. Google hits the expert site with a bandwidth charge. They use the information from the expert site to take traffic from the expert site.

BTW, this is precisely the reason that Google prohibits you from indexing the Google Site. They do not want you to use the information they gathered to take traffic from Google.

I am not saying this is bad, just pointing out contradictions...It's a Google eat goto world out there.

protophoto

Re:No contradiction

[Tony-A]

Private librarian where the library is the entire web and the library is also the public library. There is definately value there, and if I were a Fortune nnn company, I think I'd be stupid not to buy into a piece of it.
Symbiosis can be viewed as mutual parasitism. There are good reasons for corporate shirts to pay good money for what hackers download for free.

Re:No contradiction

[Anonymous Coward]

...and have the affect of taking traffic from the site.

You meant to say: effect.

Re:No contradiction

[Anonymous Coward]

I can't fathom how you could possibly think "There's no restriction on downloading and reformating results for your own use". Google's ToS explicitly says: "You may not take the results from a Google search and reformat and display them." How much clearer can they get?

Re:No contradiction

[Anonymous Coward]

You seem to be an idiot. Google adheres scrupulously to the robots.txt standard; if you don't want them to crawl your stinking website, there is a well-known standardized way to prevent them (and other well-behaved crawlers) from doing so.

On the other side of the fence, Google's servers have a robots.txt file that indicates that robots should *not* crawl their search results. And this jibes precisely with their ToS.

These guys drive me crazy.

[fm6]

The keep adding groundbreaking features to their products and throwing them out as if it were no big deal. Don't they know they're supposed to beat the PR drum every time one of their engineers burps? Bunch of commies!

Re:These guys drive me crazy.

[david_g]

Yeah, I wonder if these guys have been reading The Cluetrain Manifesto.

Re:Ron Hubbard's role in this

[erroneus]

Oh, I'm convinced now... where do I sign up?

Woody and API

[Anonymous Coward]

I run on the WOODY API. Advanced Penile Interface.

Bin Laden too, Afghan Pussies, Inc. (Woodyless of course ;) )

Re:Things To Do Today

[Anonymous Coward]

OK, I'll ask: what's #1 and #2 ?

Already something like this...

[Anonymous Coward]

They do an output without HTML already, but it looks like they've restricted access to it. Compare
http://www.google.com/search?hl=en&q=blah&output =p rotocol4
with
http://www.google.com/search?hl=en&q=blah&output =w ashingtonpost
with
http://www.google.com/search? hl=en&q=blah&output=u nclesam

-nonymous

Why not?

[Anonymous Coward]

In a nutshell, Google wants every query to be triggered by a human typing some words in a text field, and wants the results to be used for what they are: search results.

But there is already a pretty good interface to do just that. You might want to somehow adapt the interface, but that's a pretty boring project. Anything else a programmer can think of is probably infringing. In other words, the API can't be used for anything imaginative. That ruins the fun for most people.

API for Biz Partners ($$$) Only ???

[wka]

First, here's a link to a current XML API for accessing Google:

http://www.google.com/xml?q=slashdot

You'll (probably) get an error page.

I read about this on Scripting News in February:

Dave Winer made an inquiry to Google about accessing this XML API.

Their initial response was not very helpful, asking for the link to be removed, and saying that the link is "obviously reserved for Google partners." Eventually, Google let Dave access the API. Now, he sounds like he's under NDA about this.

i'd settle

[Anonymous Coward]

for an interface to the MSDN KB that actually works. And by works, I mean, returns useful hits on queries. I almost always resort to searching the MSDN KB using Google, usually with quicker, more accurate results. Tell your employers that they should spend some time and money making their online support tools less shitty.

Re:i'd settle

[Carnage4Life]

an interface to the MSDN KB that actually works. And by works, I mean, returns useful hits on queries. I almost always resort to searching the MSDN KB using Google, usually with quicker, more accurate results. Tell your employers that they should spend some time and money making their online support tools less shitty.

Believe me, the folks at MSDN are well aware of how unsatisfied some people are by their search results. If and when any drastic change will occur is in doubt so for now I suggest using the Microsoft-related search on Google

Re:i'd settle

[Anonymous Coward]

Considering that MSDN and PSS search has sucked shit through a hose since they went online, oh, 7 years ago, I'm glad someone at MS finally noticed.

I always figured it was because MS was late to the full text search patent party, and ended up using some home-grown POS instead of buying something that worked (see also Index Server).

Sounds less like an API...

[evilpaul13]

and more like an XML Web service. That means someone could fairly easily make a page that'll let you search google from your cellphone :)

Problems...

[KarmaBitch]

The only problem I can see with this is that there was a recent thread on here about Google blocking a lump of IP addresses as someone in there was automatically querying way too often and affecting their load.

With the exposed API I could see, by malice or sheer accident, floods of queries coming in...

Google's Ad Policy

[Anonymous Coward]

FYI for all of the google lovers...

Google will refuse to do any business with anyone who has trades guns or knives - even if the advertisement you purchase is not related to guns or knives.

See http://www.bowmansbrigade.com/google1.htm

It could be an improvement for them!

[Christopher B. Brown]

It means that they don't have to render pretty HTML for you; they potentially pass you a smaller amount of data, which may lead to their having to do less work.

And as for "not having to visit their site," remember that they're not doing huge amounts of banner ads. It's not totally evident that this "destroys" any of their business.

They still get to collect statistics on what queries come from where on what, which doesn't change terribly much whether they're receiving queries as HTML FORMs or XML SCHEMAs, and there's only a little reason for them to care about folks receiving back HTML versus XML

Re:It could be an improvement for them!

[Anonymous Coward]

Google's HTML is very simple and compact. SOAP is very verbose and large.

It's probably about the same amount of data either way.

Actually, it works quite OK.

[Christopher B. Brown]

If you look at the way it works right now, using the interface requires an authorization key.

If you run the Ruby script, as is, the result is thus:

#: Exception from service object: Invalid authorization key: xxxxxxxxxxxxxxx (SOAP::FaultError)

If somebody starts abusing a particular key, it's a no-brainer for Google to shut the key off.

Finally..

[egeorge]

Somebody is beginning to see what web services are really for.

Damn those exclamation points

[Anonymous Coward]

Sorry, but you use way too many of them, and the fact that you put a space before each one makes it look like the word 'I'.

No space before exclamation points, please, and use them wisely.

Re:Damn those exclamation points

[Anonymous Coward]

Give the guy a break. English obviously isn't his first language.

Another GoogleSearch SOAP API

[Shimmer]

Here

Haven't tried it yet, so I can't say if it works.

-- Brian

been done...

[john_heidemann]

CPAN already contains the WWW::Search API to many search engines (including Google until [I am told] they requested it be removed). Yes, internally, it works by parsing HTML, but it exports a (Perl) API.

Re:been done...

[PhuCknuT]

Yeah, but if it didn't need to parse html, it would be better for the client AND google in terms of cpu and bandwidth usage.

google doesn't even fucking work properly

[Anonymous Coward]

I help admin a web site. It refuses to index all but two pages (which are two of hundreds of static pages). If I do a search for words plus site:the-url no matches appear at all, even though it works with the words alone. And even then, only some of the words on the pages seem to result in retrieval.

Frankly, I wouldn't be surprised if they didn't happily take money for better/more complete indexing, they just didn't admit it. "Do you?" "No." "Here's some money." "Oh, all right then, since you asked so nicely."

Terms of service

[Xepo]

Rather than making the API something ya gotta pay for, couldn't they simply put it into the terms of service that the ads have to be shown in any software that uses the API? They could possibly offer different types of ads(text, pictures, etc.) so that you could even develop a text based app to use it and still stay within the terms of service. Have a nice little "Report a program not following the terms of service" link on the main page, and have all those people who love google help them out by reporting any programs they find that don't show the ads. Oh, and then also offer a pay-for service if they want so that the program dosen't have to show the ads.

Re:Terms of service

[gorilla]

Not all programs output in such a way that adverts would make sense. For example, I can imagine that someone may wish to run a program which analyzes how well ranked their site is based upon various words over time. The natural output for this could be a chart. What do you do with the adverts?

Re:Terms of service

[Xepo]

They do output stuff though. They could output the chart with the ad on it, maybe in the background or something like that. Yea, the user could cut it out, but the person who used the program still saw the ad, and google should still get credit for that ad. Heck, the program could just display it on the splash screen, or on the main screen of the program, and not even mess with putting ads on the chart.

I think, based on what google has done in the past, google isn't out to make a billion bucks by releasing this programming thing, they just don't wanna lose money by doing it. This would be one possible way for them to do it.

Re:Terms of service

[gorilla]

That works if you have 1, but if you've got 10 keywords a week, every week for a year, that's 520 adverts to display!

SOAP must be evil though!

[Anonymous Coward]

Guys... you do all realize that Microsoft *invented* SOAP... so it MUST be evil.

We should all boycot Google, right?

Hello?

Re:SOAP must be evil though!

[Anonymous Coward]

Well, RMS didn't invent SOAP, that's for sure.

Ob20SecondTimeoutBloatLine

Re:SOAP must be evil though!

[Anonymous Coward]

In fact, that's probably why he doesn't use SOAP -- it's not Free.

business model

[rmadhuram]

I think this idea will work well for their google search appliance, if not for the internet. Imagine SOAP enabling the local google search engine. Now you can add search capabilities from any of the enterprise application (no matter what o/s, language) by connecting to the search appliance. That would be wonderful!

Re:business model

[0x0d0a]

That's what I assumed they were talking about at first. Heck, wouldn't surprise me if it's already SOAP-enabled.

"Search" as a network service...

[overturf]

So let me get this straight:

Google provides the ability to bounce searches off of their search application over the Internet using an API.

Microsoft writes ".NET" to allow folks to utilize application services over the Internet using an API.

They seem pretty similar in concept. Perhaps I'm missing something here, but isn't what Google is doing essentially the same thing as what a .Net service would allow them to provide?

Shoot, I wonder if they won't just set it up as a .Net service once .Net is available for Linux (wouldn't this allow them to do authentication and billing?)...

Re:"Search" as a network service...

[toriver]

Microsoft writes ".NET" to allow folks to utilize application services over the Internet using an API.

No, Microsoft and a bunch of others, through the W3 Consortium, make SOAP and WSDL to create platform-"neutral" protocols for web services. Whether they have .Net or J2EE sitting behind that SOAP interface is irrelevant, just as it's irrelevant what language you use to access the service: Everything is exchanged using XML documents anyway.

Slash could use this.

[Perdo]

Slash either needs to get a Google box or use these APIs to fix their search feature. There is so much haystack data compared to good needles on Slashdot and the search is so bad that most of the great gems of knowlege that Slashdot has generated might as well have never existed. It can take an hour to find even a popular poster's comments.

Need to reference John Carmack's comments? Sorting him out of the masses is next to impossible. Even a comment poster as prolific as Signal 11 (arguabley slashdots first and greatest Karma Whore) is nearly impossible to find. First 30 matches of how many? You want to sort through jeffy124's 700+ comments and 24 submitted stories just to find the pertinate one I need by hand? Not to mention the benefit to Slashdot's editors, being able to follow a clear history of articles on a given subject to look for repeats and make more informed editorial commentary. If 90% of readers never read the comments, the editors owe that 90% the sort of editorial commentary attached to each story that only good research can provide.

In fact, the editors could try it on an interim basis immediately, and provide the service to readers only if they had the resources. I sort of get the feeling that the editors are still thinking of slashdot as a small time blog run out of their apartment closet server.

Run google on slashdot now and you get the news from three weeks ago. Incorperate a google box or google APIs into Slash so I could search today's news and I would Pay 10 cents of subscription funds per search in a heartbeat.

Editors: look at the number of hits to your current broken search engine. Double that number because a dedicated google box would be so much better it would get used a whole lot more. Multiply that by 10 cents per search. See if the numbers work to afford the initial expenditure to get a nice yellow rack mount google box. Slashdot is sitting on a goldmine of data and no one can search it and Slashdot cannot profit from it without a nice pay per search subscription using the best engine available.

Re:Slash could use this.

[big.ears]

Many years from now, you will be considered "in the know" if you remember Signal11, and elite if you got personally trolled by him.

Re:Slash could use this.

[Perdo]

We had many fine discussions. I told him to go use his charismatic persuasion ability to become a politician. He left shortly after, giving his 500+ karma account to some troll to burn up posting goatse.cx, penis bird links, and refrences to natalie portman naked and petrified hot grits posts.

I don't know what became of him. I shure wish he (god, for that matter signal might be a she) would make a slashdot submission story letting everyone know how he's doing.

To give the kids an idea of his abilities, he would have at least one 5 post on every single story. It's like he was able to prepare posts for stories before they were even posted. He always had the first informative or insightfull comment in every single article.

Signal 11 has posted 2099 comments.

He was the Karma Whore for which the term was invented. He has HUNDREDS of imposters and may have had the first imposter, Signail11, a troll that knew moderators would get confused and occasionally mod him up. The man never slept and his insistance on posting early and often made slashdot consider headline beepers.

One of the things I miss the most was his ability to deep link to a NY Times article. No one ever had to register as long as he was around.

Re:Slash could use this.

[p3d0]

Just go to Google and add site:slashdot.org to your search.

Re:Slash could use this.

[Perdo]

Searched pages from slashdot.org for p3d0 .

Results 1 - 10 of about 51. Search took 0.12 seconds.

51 of 621 is no where close to good enough. The entire database must be indexed and parsed. Magic Google algorithms must dance upon the text and context.

I want a search to say:

results 1-50 of 621.
sort by relavance.
sort by score.
sort by date.
sort by number of replies.
sort by total moderation done to comment.
sort by the number of author's posts in story.

So I can say, "gee, I remember p3d0 had at least 5 comments on that story and two of them were at least score 5" and the search engine finds all data sets that match, sorted by how closely they match. Slashdot is a convoluted enough system that the Slash crew may have to extend even Google a bit. But they are bright lads and if not too busy with day to day editing would find such a task within their abilities.

Re:Slash could use this.

[p3d0]

Good point. Also, Google is, by its nature, always a little out of date.

Google returns 2 distinct sets of results

[yerricde]

Google does not have a pay for placement plan - if you are making reference to the practice of changing the order of search results based on advertiser dollars.

Yes it does. When you search Google, it displays two distinct sets of results side-by-side. One set is based solely on PageRank values; the other (clearly marked "Sponsored Links") is based on advertising dollars. The problem with GoTo was that you had to scroll and click past pages and pages of sponsored links to get to the results scrolled by relevance.

Re:Google returns 2 distinct sets of results

[ipsuid]

When I see the word's "Pay for Placement" I take it to mean the practice of altering the actual search results based on who has paid and who has not. The sponsored links on Google are not part of the search results. So if there is an API to search Google without going to the website, then you would be getting back either 1) The normal search results, 2) The normal search results plus the sponsored links, or 3) The sponsored links masquerading as part of the search results. Since the whole point of having an API is to be able to search from a script, and scripts don't read advertisements, the only way you could support this via Pay for Placement would be to use option 3. I don't see Google doing this, since it is one of their big selling points that they are concerned with the integrity of their search results. More likely they will either charge for the ability to search via this API - or will count on the API to boost the sales of their hardware search solutions by giving companies a taste of what they could do with one.

Google Hosted SiteSearch

[yerricde]

They admit it: see Google Hosted SiteSearch. Pay Google the $$$, and Google will index all your site's pages as often as you want.

Google's Custom SiteSearch means that the pages you wish indexed from your website are crawled and indexed. Crawls can occur daily or weekly, depending on your frequency of updates.

It's primarily intended for webmasters of commercial sites who want to outsource their "Search This Site" functionality, but the index does leak into the general web search database.

Re:Google Hosted SiteSearch

[pipingguy]

Doesn't Atomz do this also?

Re:Google Hosted SiteSearch

[mpthompson]

Yes, Atomz currently hosts site-search for more than 55,000 web sites. Unlike Google, we don't offer Internet wide search, but we certainly have an extensive list of customers for our hosted site-search and content management services.

The post googolplex era

[Anonymous Coward]

I was using Googolplex to do it. Now I will migrate!

Programmatic interfaces - the way forward

[Anonymous Coward]

Programmatic interfaces to web content and functionality really are the way forward. Anyone know why this isn't taking off more? The applications are just limitless. Not just things for things like syndication but just some of things like this:

• Cinema times: write your own program to integrate with your personal organiser to find out when you're free to go to the cinema. Integrate with a movie reviews programmatic interface to see if you'll like the film.
• Airport arrivals and departures: write your own program to keep track of your partner's flight, and get it to send you a text message to your phone when it arrives (through another programmatic interface).
• Real content distribution and integration. None of this IFRAME or remote JavaScript trash. Imagine being able to license content and use SOAP-based RMI to integrate it into your site.

I really love the idea of programmatic interfaces. It's fascinating, because the problem we have now is not a lack of information, but a lack of coherent *integration* of this information.

I also think that once people realise the limitless applications and emergent properties that arrise when different sources of information can be easily integrated, a whole new generation of people will be introduced to programming, especially with easy interfaces to SOAP through Perl and the SOAP::Lite module.

php

[slug359]

google already block php from querying, try wget http://www.google.com/search?q=moo --user-agent=PHP opposed to: wget http://www.google.com/search?q=moo --user-agent=Mozilla it stops fopen and file working, you can either modify the source or use fsockopen though. if they don't want php scripts parsing google, why would they want this? :)

Googleistic

[tbx2000]

Personally, I tend to think that the reason that Google would launch something like this is to support their new 'Overture-style' auctioned keywords... You'll probably get no-cost search results, as long as someone is paying them to rig the first couple.

To moderators

[Shiny Metal S.]

Redundant?! *Please* answer and tell me how the hell was it redundant while 95 minutes later a comment duplicating a subset of mine was suddenly interesting. Oh, I get it, you mean redundant in a sense, that others said the same later? I see, it is redundant for some reversed definition of redundancy.

I don't care about the stupid karma, but I do care about people reading and replying to my comments. Now I have karma 50 again, thanks to my other post, so you can safely mod my post up as +2 informative, +1 insightful and +1 interesting, not being affraid that I might get some karma. Thank you for your attention. I'm sure no one will read it now with Score: 1. Of course I'm currently not eligible to Meta Moderate so I can't even complain.

Re:To moderators

[Anonymous Coward]

You honestly give a rat's ass what these people think of you? Don't.

it will also save them lots of money

[zer0*ryok0]

a more clean method for people who want to use google (outside of the normal web interface) means less to transfer from the server, which means less $$$ spent on bandwidth bills.

it a common sensicle

sensible bandwidthwise

[ddent]

People can already access google through an API (perl modules exist already for instance) by making an API that parses the html. There is thus no reason for them to *not* make things available. It costs less to transmit the results without formatting information, its useful to people, .... Don't forget, google is pretty focused on usability; something which is one of the main things which has made them so popular IMO.

Re:sensible bandwidthwise

[Anonymous Coward]

In Python too, just Googolplex

robots.txt?

[_Bean_]

Ever heard of it?

*drool*

[potnoodle]

Think of the Multiplayer Online games with access to this kind of database for content and using it to port parts of it to a game universe... Better extend those schizo wards now , cause it's gonna be rough on some people :)

Re:Cool, but.... They never said if was free!

[aminorex]

> /.CONTRADICTION:
Dissing the rule of Satan, only to advocate the
rule of a loving God.

reductio ad absurdum

Re:Cool, but.... They never said if was free!

[Anonymous Coward]

>Dissing the rule of Satan, only to advocate the rule of a loving God.

You must be a Sun employee to call Scott McNeally's brainchild the reign of a loving God.

I got your Google API right here!

[Blue Lozenge]

char *google_search(char *pattern);

:)

why is this modded as troll?

[Anonymous Coward]

makes no sense.

is my min char limit reached yet?

Google database not updated in 3 months

[Korth]

Google doesn't seem to have been updated for about 3 months.

e.g. a search for "GXP120" on Google gets me 29 results.
The same search at www.alltheweb.com gets me 1012 results.

human shit, not ruby, techie wanker!

[Anonymous Coward]

scat..... the act of deriving sexual pleasure from human shit.

i wasnt commenting on the lang they used. prat

SOAP me up baby !

[beanerspace]

I was just thinking about incorporateing a Google search on my websites after an impressive experience with a few websites that employed their Free WebSearch plus SiteSearch feature.

This is even better. With this feature, I'll be able to SSI and/or push results using something as simple as SoapLite to get the job done.

I sure hope other content providers are taking note. Imagine how useful (not to much fun) it would be to snap up stuff from places like MoreOver.Com?

Not a problem!

[Anonymous Coward]

Check out Jakarta ORO, a Perl 5 Compliant Regular Expression Library for Java at http://jakarta.apache.org/oro/

I've been using this on my Java projects for years, works great! (Of course you have to know how to write Perl 5 regexes, an art in itself!)

Also Java 1.4 is FINALLY bundling some sort of regex library in it's core API, but I haven't messed with it yet.

Re:Cool, but.... They never said if was free!

[aminorex]

Actually, I'm a disgruntled ex-employee.
I think of Scott as more like Jim Barksdale
on acid than God.

.net on tux is a vaporware !

[Anonymous Coward]

Of course you may get any CLR port on whatever OS you want, but you will never get a real .net complient OS on thing but WinXX OS !

MS gents are maybe idots in communication but their are not stupid in strategy ;-)

This is not Java babe :o)

-4R34'.

slightly off-off topic

[Anonymous Coward]

My favorite "feature" on the full text search implementation in SQL Server 7.0 was that you couldn't pass variables into the searching functions. You had to pass in string literals.
Thus, you either had to allow selects on the table/view and run them from the client app, or allow selects on the table/view and write a nasty ass stored proc to dynamically create SQL and execute that.

They want to be taken as a serious enterprise player, as far as databases go, but they insist on introducing half-half baked features at every iteration.