Scroogle Has Been Blocked

An anonymous reader writes "Scroogle, the secure third-party Google search interface, has been blocked by Google. Scroogle was an SSL-based search proxy that enabled one to search for and receive Google results over an SSL connection in a pseudo-anonymous manner."

Comments

Scroogle

[sopssa]

While I would love to see a good rant towards Google and while I also myself use Scroogle, the summary isn't really being truthful. Google hasn't blocked anything, they just changed the page that Scroogle scrapes and they're throwing a hissy fit about it.

From the Scroogle announcement:

We regret to announce that our Google scraper may have to be permanently retired, thanks to a change at Google.

That interface was at www.google.com/ie but on May 10, 2010 they took it down and inserted a redirect to /toolbar/ie8/sidebar.html. It used to have a search box, and the results it showed were generic during that entire time.

Now that interface is gone. It is not possible to continue Scroogle unless we have a simple interface that is stable. Google's main consumer-oriented interface that they want everyone to use is too complex, and changes too frequently, to make our scraping operation possible.

Google changing something isn't exactly "blocking" a third party service. Even more so, it's just a few lines of code to get the results from main Google search too. All the search results and links have approciate html ID's associated to them and it's been the same for years already.

I have no idea why Scroogle is bitching about this.

Oh well. I changed to use ixquick, which also has the added benefit of being located in the Germany rather than US and a lot better and useful interface.

-sopssa

Re:Scroogle

[longacre]

What's the benefit of being in Germany?

Re:Scroogle

[Registered Coward v2]

What's the benefit of being in Germany?

Maybe:

Ve hav vays of making you benefit

or,

Ve no nutting, nutting,

Re:Scroogle

[sopssa]

From the FAQ:

European Privacy Seal
On July 14th 2008 Ixquick received the first European Privacy Seal from European Data Protection Supervisor Mr. Peter Hustinx. The Seal officially confirms the privacy promises we make to our users. It makes Ixquick the first and only EU-approved search engine. Both EU Commissioner Viviane Reding and Dr.Thilo Weichert, German Privacy Commissioner complemented Ixquick on its privacy achievements.
You can find the press release here.

Since I am in EU, it also means US can't just randomly get data that doesn't belong to them, ie. for people from other countries. Frankly, EU and European countries take privacy a lot more seriously, for historical reasons too.

Re:Scroogle

[Anonymous Coward]

real beer, excellent food, beautiful landscape

Re:Scroogle

[Jer]

What's more, the link they were scraping off of [www.google.com/ie] seems to be related to Google's support of Internet Explorer. Since it's been replaced with a "go get IE 8" page, it's probably been dumped to encourage people to dump their older versions of IE and get something newer.

Re:Scroogle

[cmiller173]

You sir are absolutely correct. Anyone who writes an application based on screen scraping should expect changes to happen and not act surprised when they do. Besides doesn't Google have a freaking search API? http://code.google.com/apis/ajaxsearch/

Re:Scroogle

[bsDaemon]

and less hoops for INTERPOL, German Federal police, etc. Also, there is then no protection, even in theory, from NSA or CIA operations as that becomes "over-seas" and thus fertile ground. I guess it all depends on who you're trying to slightly inconvenience, though.

Re:Scroogle

[JustinOpinion]

Indeed. Scroogle sounds like a good idea... but it's a service that exists parasitically to Google proper. I'm not trying to imply anything unethical by using the word "parasite", but this really is a situation where Scroogle uses Google's capabilities/services without contributing anything back to Google. This is fine to the extent that Google tolerates it. But they are under no obligation to make accommodations to keep these third-party services running smoothly. TFA says "It's not as if Google needs the money" which seems rather uncharitable given that Google has put up with Scroogle's operations for many years now without any complaints or blocking attempts (that I'm aware of). And Google does need some money (they would have to shut down if everyone used their services through Scroogle...).

Scroogle needs to either adjust their service to keep up with Google's changes, or make a business case to Google for why it is in their best interest to provide a stable interface/API for third-party redistributors like them. The implication in TFA that they are somehow entitled to this interface/API/access is really silly.

Re:Scroogle

[Anonymous Coward]

A couple fun facts about Scroogle:
1. Since Scroogle hit multiple Google IPs, it used to be possible to search the same keywords 5 or 6 times in a row and see the variation in page rank. Great for web site owners to see how they ranked.
2. Scroogle dot COM is NSFW at all, so when telling people about Scroogle it was usually CRUCIAL to emphasize the dot ORG part of the domain. At a previous job I made the mistake of telling my boss about it without emphasizing the dot ORG part and, well... he got an eyeful of the wrong type of "org"...

Re:Scroogle

[Jer]

The linked article does use the title "Scroogle has been blocked" when, really, they haven't been blocked at all. They're free to change their code to work with the various other methods of accessing Google - like perhaps using the publicly available API that Google provides. Since I've never used the API I'm not sure exactly what technical limitations it imposes that make screen scraping a better alternative to the API for privacy concerns. Anyone have an idea why they would need to use a screen scraper to anonymize connections instead of using the API?

Re:Scroogle

[somersault]

You could try following the links, dumbass

https://www.european-privacy-seal.eu/awarded-seals

Now, it doesn't mean there will never be a data breach (by a disgruntled employee or whatever), but the fact that they have actually bothered to get a certification says to me that these guys are more committed to data privacy than your average website that collects and distribute personal information.

Re:Scroogle

[geekoid]

The US can't do that in the US either. Just an FYI.

" Frankly, EU and European countries take privacy a lot more seriously,"
Care to back that up? I mean when you can take time away from being on public video, told what you can and can not say, carrying papers,

IT would be more correct to say it treats privacy different;which makes sense because what it considered ''privacy' is different. For example, what you do in public can be considered 'private' in some countries.

Of course it's such a patch work in the EU, it's almost nonsense to say to use the EU as a generally statement concerning privacy.

Re:Scroogle

[Anonymous Coward]

but it's a service that exists parasitically to Google proper. I'm not trying to imply anything unethical by using the word "parasite", but this really is a situation where Scroogle uses Google's capabilities/services without contributing anything back to Google.

The word you're looking for is commensalism. Although I think in this case it is closer to parasitic since it does use some of Google's resources without giving back much or any value to Google itself.

Re:Scroogle

[logjon]

Not sure which US you live in, but here in the US I'm a citizen of, the government has unfettered access to communications, digital and otherwise. The patriot act took the last of American privacy, and with a hearty chuckle, wiped its ass with the remainder of the fourth amendment.

Re:Scroogle

[_xeno_]

Because if they did that, they'd be forced to abide by the search Terms of Service. And they appear to be violating Section 1.4.

By using the generic web robot approach, they're allowed to scrape Google based on the same concepts that allow Google to scrape third party web pages in the first place.

From Google's robots.txt:
User-agent: *
[snip]
Disallow: /ie?

Well, OK, so they're not obeying robots.txt in the first place. But ignoring that one pesky fact, uh...

Re:Scroogle

[negRo_slim]

What's the benefit of being in Germany?

I may be mistaken but I believe they have stronger privacy laws.

Re:Scroogle

[Smauler]

The Data Protection Act in the UK was as a result of the Data Protection Directive from the EU. This severely limits what people and/or companies and/or governmental agencies (with some exceptions to the last) can do with information about you. I'm not aware of legislation as strong as this in the US, but I may be wrong.

I do agree that different countries treat privacy differently - I personally believe that anything I do in public is basically that - public. I won't ever carry papers in my own country, so if somehow the ID card in the UK goes through (looking very very unlikely at the moment), I'll just lose mine every time I get a new one, and reapply. Some people don't have a problem with such things, but I do. The EU is a very diverse place, but that data protection directive means that all EU countries have similar laws with regards to data protection AFAIK.

Re:Scroogle

[icebraining]

Except each search query Scroogle did cost some cents to Google, while a filesharer copying a song/album/movie to another person costs the record companies exactly zero.

They may had no right to copy it, and maybe they shouldn't have done it, but claiming each P2P copy costs real money to the companies is ludicrous. All the costs (bandwidth and electricity) are paid by the two sharers.

Re:Scroogle

[MikeBabcock]

As in they don't have a leg to stand on, yeah.

"Hey we're scraping a page you told us not to scrape as a robot and you moved it" - "We have a public API" - "We don't wanna follow your licensing terms"

Re:Scroogle

[Myopic]

Nice. How's the weather there? The rest of us are stuck here in the United States of Reality.

Re:Scroogle

[clone53421]

It’s not a fully-automated tool, therefore not a robot. It scrapes the page only once at a real user’s direct request.

As such it is no different from a browser (which also scrapes – downloads, parses, and translates into useful format – a page once at a user’s direct request)...

or, for that matter, any different from the IE search interface that the /ie path was meant to support!

Re:Scroogle

[slaad]

Since I am in EU, it also means US can't just randomly get data that doesn't belong to them, ie. for people from other countries. Frankly, EU and European countries take privacy a lot more seriously, for historical reasons too.

I'm pretty sure that the NSA doesn't much care about the European Privacy Seal.

Re:Scroogle

[insertwackynamehere]

Why is everyone so willfully ignorant to theft? By downloading media without paying for it, there is no tangible product that costs anyone any money except for bandwidth, but that is not what pisses off record companies. What pisses them off is that they release a product for sale with a value attached to it. This value factors in attempts at profits and attempts at covering the cost of production/paying the artist/etc. When music is illegally downloaded, the company loses money because someone now owns the song without paying for it. Now maybe that person wouldn't have paid in the first place and unlike shoplifting, there was no physical loss or diminishing of goods. However, the fact is that the person downloading the song is still taking something of value without purchasing it. This is theft. And media can have intrinsic value..it costs money to make good music and songs, money spent with the expectation of high returns in sales. If everyone downloads music, then the free market will indeed win as many Slashdotters smugly wish for, hoping to prove the things people make for them that they like aren't actually worth anything. However this free market win will be a ceasing of production for movies and songs..no one can profit so no need to fill the void.

Re:Scroogle

[al0ha]

Actually IxQuick is located in The Netherlands which is certainly not Germany and darn well good that it is not because Holland does actually concern itself with privacy protection, Germany on the other hand, not so much.

Re:Scroogle

[Myopic]

Yes exactly. It's the one where the Supreme Court rules that violations of any amendment are okay as long as they're really, really important. That, again, is called reality. Here in reality, we know that sometimes rules need to be broken.

In the case you cite, though, I don't like the way that rule is broken. I wish it were different, here in reality. There are lots of things I wish were different, here in reality, but I don't pretend that limited exceptions to constitutional provisions amount to (quoting you) "unfettered access to communications". In this case, the operative word is "unfettered". Here in reality, the government has *fettered* access to communications, although sometimes I wish they were slightly more fettered.

Re:Scroogle

[operagost]

I'm glad that Eurasia has certified that we are safe from the prying eyes of Oceania.

Re:Scroogle

[dAzED1]

Let's look at what I was replying to:

I mean when you can take time away from being on public video, told what you can and can not say, carrying papers

Now, how am I taking the current AZ situation out of context? Is racial profiling not occurring, with people being told to show papers? In fact, that's exactly what is occurring. Which means, it is no longer valid to use Europe's habit of asking for papers as an indication that we have more liberties here - since that is now occurring here.

Note also that I didn't really indicate how I felt about the new law, either - I just said that it makes it silly to say we're different in that regard, since we're no longer different.

Thoreau, after being jailed for refusing to pay taxes due to his stance against the Mexican war (oddly appropriate...), wrote a dialogue containing these lines:

Under a government which imprisons any unjustly, the true place for a just man is also in prison. The proper place to-day, the only place which Massachusetts has provided for her freer and less desponding spirits, is in her prisons, to be put out and locked out of the State by her own act, as they have already put themselves out by their principles. It is there that the fugitive slave, and the Mexican prisoner on parole, and the Indian come to plead the wrongs of his race, should find them; on that separate, but more free and honorable ground, where the State places those who are not with her, but against her,--the only house in a slave-state in which a free man can abide with honor.

So a person with dark skin is walking along, and a police officer spots them and asks them for their papers. Note that can be the primary cause for the contact; the person doesn't have to have been doing anything wrong, they merely need to be a victim of racism. If the person doesn't have paperwork on them, they are jailed, and have to prove they are allowed to be here. Would I run that risk, being white? Nope. And thus, the issue that many people have with the new law.

Re:Scroogle

[kiwimate]

Copyright infringement is not theft. Please look up the definitions of the two.

Okay, I'll do that.

Theft: the act of stealing

Stealing: 1 a : to take or appropriate without right or leave and with intent to keep or make use of wrongfully b : to take away by force or unjust means c : to take surreptitiously or without permission d : to appropriate to oneself or beyond one's proper share : make oneself the focus of

Copyright infringement is taking something without permission. Equals stealing. Equals theft. And by the way, why does it make things ANY better when slashdotters self-righteously proclaim "it's not theft or stealing, it's copyright infringement"? Nice one, all you're doing is muddying the argument over semantics; as is frequently pointed out, it's still illegal.

Next time you decide to pontificate, make sure you know what you're getting into.

They didn't block it...

[stagg]

If you RTFA you'll notice that Google didn't block Scroogle, they just upgraded without consideration to its functionality. As soon as someone can explain why Google WOULD have Scroogle on a dependency chart we can all put our conspiracy hats back on.

Re:They didn't block it...

[enaso1970]

They WOULD because if you change some letters in Roswell and add one, you get Scroogle! Isn't it obvious???! It should be to the aliens who run Google. Clearly the government has you in their clutches - only an upgrade to TinFoilHat 2.3 will save you.

Re:They didn't block it...

[Monkeedude1212]

Are you trying to tell me that a Private Corporation such as Google doesn't regularily consider the functionality of other companies who slam it on a regular basis, such as Scroogle?

Re:They didn't block it...

[icebraining]

Scroogle is owned by Public Information Research, Inc. Their board of directors is in Scroogle: http://www.scroogle.org/staffsc.html

The Summary Lies!

[dancingmilk]

What a horrible summary. Google didn't block anything, they just changed the page that Scroogle scrapes off of. Scroogle claims that they need a "simple" interface to scrape off of. Sounds to me like they are too lazy to adjust their service.

Re:The Summary Lies!

[idontgno]

break scraping.

Scraping is inherently unreliable. Particularly if you're scraping without the data source's permission or cooperation. It's what you do with the bottom of the barrel.

If you want reliable, you won't be doing any scraping. If you're doing scraping, don't get bent out of shape with it suddenly stops working. By choosing a scraping solution, you've committed yourself to intermittent service and a continual race to keep up with target interface changes.

Of you can use the provided API? Yes, it has limitations. But one of them isn't "brittle, unreliable, and subject to complete failure without notice".

Re:The Summary Lies!

[natehoy]

Scroogle has the absolute right to a refund for any and all money that they have paid Google because Google isn't living up to the contract where Scroogle pays Google for a stable connec... wait, what was that? Oh, I see. Never mind.

Scroogle may be providing a service that people value, but they are still using Google to do it, and not paying Google for that access. Google is tolerating this, which is all well and good, but they are under absolutely no obligation to make sure the connection is unchanged. Sites change all the time, and anyone who employs scraping technology as part of their technological solution should not be surprised when they do.

Duck Duck Go

[Anonymous Coward]

I used to use Scroogle for privacy reasons, but switched to Duck Duck Go a few weeks ago. It is quickly becoming a great privacy-respecting alternative to Google and often gives more relevant results than Google.

Re:Duck Duck Go

[compumike]

I too have been trying Duck Duck Go (link to encrypted version) for the last several weeks and have been impressed.

Furthermore, check out their privacy policy, as well as a recent blog post about search privacy that explains why it "might be the most private place to search the Internet". No IPs logged, no cookies, no contractors.

There are also a large set of convenient "bang commands" such as searching "!slashdot foo".

And finally, searching over (encrypted) HTTPS just works "out of the box".

Give it a try for a few weeks!

Re:Duck Duck Go

[base3]

Yup -- not turning on JavsScript for that--no reason it should need it.

Umm

[mindstrm]

When google wants them to stop, they'll be hearing from lawyers........ not just finding that google changed their page layout.

Re:Optimize Google Firefox Extension

[Anonymous Coward]

I have been using this extension for Firefox called Optimize Google. http://www.optimizegoogle.com/ [optimizegoogle.com] It has the ability to disable click tracking and Google's ad services as well as a bunch of other features. Whether it works is up for debate.

You use it, but don't know if it works?

Re:Optimize Google Firefox Extension

[0100010001010011]

The point isn't that Google or Scroogle can see what you're searching for, with SSL no one in between can see.

Say you search for "How to kill your wife and hide the body". With Google, every ISP that transfers packets between you and them has a record of it. With Scroogle only Scroogle knows what you searched for. (Not sure if they keep logs to subpoena).

Ah, Don't be evil?

[GNUALMAFUERTE]

They are being Evil. They have a perpetual obligation to keep every single feature in a time-freeze so that third parties can use them as they see fit!

Ah, wait, no they don't.

There is an assload of meta-search engines out there. Scroogle seems to be the only one that has been affected. That's because they were saving bandwidth, processor usage, and programmer's time by using the same fucking simple interface for the last 5 years. So, they've been using an old interface that existed for the SOLE PURPOSE of being compatible with shitty old IE versions .... now that google pulls it out, they bitch about it? Come on ...

Here is what I hate: Everyone is complaining about the privacy concerns with many services, but nobody stops using them! Everyone feels they have the right for every service to work they way they want it to. Guess what, you don't. You don't like google? Stop using it!. I don't like microsoft. I Don't like anything from them. So, I don't use ANYTHING FROM THEM. Not their software, nor their services, nothing. On the other hand, we have people cracking their software and complaining when they are evil. They ARE evil? stay the hell away from it.

I'm really tired of this privacy-concerns constant circle-jerking. Stop using the shit you don't like. Simple, huh?

Need for anonymous search engine

[Enderandrew]

Is this the same company that started anonymizing search logs sooner and refused to hand over search data to the US?

Is there a reason why you NEED a more anonymous search engine? And can you trust the other party you're going through isn't logging your search inquiries?

Ultimately it comes down to who you trust more. I just don't understand why no one trusts Google when they have the cleanest track record out there.

Re:Need for anonymous search engine

[geekoid]

And yet, some people will trust a site based on the pure presentation of that site not trusting google.

As if some in between site is more or less likely to sell your data.

Re:Need for anonymous search engine

[Enderandrew]

I just think Google is smarter than that. Facebook is king of the internet right now, but they are alienating their users at every turn.

They forget that Myspace was once the largest social networking company out there. Facebook could fall to second banana just as easily the second a better alternative shows up.

Google however is smart enough to realize that their entire business model is about getting users to use their products so they can serve up contextual ads. If they piss their users off, it destroys their business model.

Their April Fool's prank was to name themselves Topeka for a day. They were basically saying "we love you too". In fact, they've directly said it to me.

They had award nominations for open source contributers. I nominated Andrew Morton, not realizing at the time that Google was paying him to work on Linux full time. I got an email back from Google explaing that it made Morton ineligible for the award.

I responded saying that it was just one more reason I loved Google, because they pay guys like Andrew Morton. I literally got an email back from Google saying "We love you too."

The world is full of shitty companies that don't care about their customers/users. They just don't get it. I really think Google is the exception to the rule.

Seriously, change the header

[valadaar]

No other comment - this is simply factually wrong. Let me know when Scroogle can't even resolve Google servers, then they are truly blocked.

I don't get it - why "scrape" at all?

[NotQuiteReal]

I don't get it - why "scrape" at all? Google has a real search API, do they not?

Re:I don't get it - why "scrape" at all?

[Anonymous Coward]

Because they were too lazy/lacked the skill. I shed no tears for an operation called Screw-gle that can no longer suckle from the teat of Goo-gle.

Re:I don't get it - why "scrape" at all?

[MartinSchou]

Doesn't that one require a key that only supports 1,000 searches per day?

That kind of thing would make Scroogle useless. And since Scroogle has no interest in paying Google for the results, they aren't going to purchase the kind of access they'd like to have.

Re:I don't get it - why "scrape" at all?

[Lunix Nutcase]

So basically Scroogle is a leech and they got what they deserved.

Re:Real API

[jandrese]

In fact they do. It's not clear why Scroogle has such a hard on for screen scraping.

www.google.com/ie gone -- also used by the blind.

[bpechter]

The wife used the www.google.com/ie interface for accessability reasons. It worked much better for her with her screen reader. She's totally blind. She'll miss the interface and I know there were others using it for the same reason.

Stick it to google....

[jtcampbell]

......by using a different search engine.

Oh wait - you're weren't generating any revenue for them and were actually costing them bandwidth.

That will really show them!

Re:Optimize Google Firefox Extension

[clone53421]

Not sure if they keep logs to subpoena

“We don’t use cookies, we don’t save search terms, and logs are deleted within 48 hours.” – graphic on their homepage.

Re:Why should i trust scroogle more than google?

[0xdeadbeef]

Trustworthy people usually don't have such a whiny sense of entitlement. They've probably been a honeypot all along.

Re:Distributed privacy?

[gorzek]

That's pretty much what Tor does, only it can be used for any kind of traffic, not just searching.

Re:Google search API not applicable

[clone53421]

If the connection was performed by sockets between Scroogle’s own servers and Google’s (which is what they were doing with their SSL searches to screen-scrape the results from the old /ie interface previously) it would be the same level of anonymity as before. AJAX is just a Javascript interface to open sockets and make HTTP/HTTPS requests.

It’s just a matter of server side vs. client side. The primary reason that an AJAX search is done by your browser rather than your own webpage is because it saves your server the bandwidth and time (and saves the visitor the time too) that would have been required if it was done server-side. It could be done using server-side scripting.

Re:www.google.com/ie gone -- also used by the blin

[mea37]

I'm legally blind (but not to the extent that I require a screen-reader) and certainly I advocate for accessability features. But, just like the /ie interface wasn't intended to be a stable screen-scraping interface for Scroogle, it wasn't intended to be an accessability feature. That's the problem with using things in unsupported ways. Sure, they may work now - but you have no assurances going forward.

I'd suggest your wife, and anyone else who finds Google's support for low-vision users lacking, contact them and start lobbying for a proper solution that they will then have proper knowledge of and reason to support.

No, Google doesn't have a real search API.

[Animats]

Google once had a real search API. It was SOAP-based. But they discontinued it years ago.

Google's AJAX search API is, by design, very limited. All you can really do is create a little search widget, and perhaps add some fields of your own. The term prohibits doing much beyond that. "You are allowed to use the API only to display, and to make such uses as are necessary for You to display, Google Search Results on your Property. The API does not provide You with the ability to access, and You are not allowed to access, other underlying Google Services or data. Subject to the limitations and conditions described below, " ... "You agree that You will not, and You will not permit your users or other third parties to: (a) modify or replace the text, images, or other content of the Google Search Results, including by (i) changing the order in which the Google Search Results appear, (ii) intermixing Search Results from sources other than Google, or (iii) intermixing other content such that it appears to be part of the Google Search Results; or (b) modify, replace, obscure, or otherwise hinder the functioning of links to Google or third party websites provided in the Google Search Results. " Given those restrictions, you can't write Scroogle using that API.

We have a SiteTruth search page which uses the Google AJAX API. We're prohibited from re-ordering the entries or removing any of them. Since the whole point of SiteTruth is to re-order search results by business legitimacy, and we don't do that for the Google results, the Google results are inferior to the ones from other search engines. So our primary search page uses Yahoo/Bing.

Re:Why should i trust scroogle more than google?

[slyborg]

>Scroogle has access to the exact same information Google would have had you used Google

Wrong. The reason people that don't just walk around cluelessly believing everything people tell them are concerned about Google is that it continuously is trying to be the Panopticon. It's not just your search history, that's a tiny part of what they do now. It's Google Analytics, it's watching what you're watching on YouTube, their (pathetic) attempt to muscle into social networking with Buzz, the emails they have full access to via Gmail, etc. It's not because they want to "improve your search experience". They want a full profile on you so they can sell you as a package to advertisers.

Even setting aside concerns about oppressive governments getting their hands on this data, do you really want advertisers to have this data in detail? For example, Amazon already uses differential pricing. If a retailer knows you are super keen on a particular genre, they may provide you with *higher* pricing than other people because they are reasonably sure you'll pay. I don't want to hand over negotiating leverage to a party that already has way more information than I do.

Does this mean scroogle is...

[croftj]

scroodled?