geekfiend writes "Today Google updated their website to indicate over eight billion pages crawled, cached and indexed. They've also added an entry to their blog explaining that they still have tons of work to do."
"Google Index Nearly Doubles" is in the url and the first header. Look at how they do thinks... and your google traffic will increase.
More pages v.s more relevant pages
by
xiando
·
· Score: 5, Insightful
Personally I find that the lack of relevant pages if the biggest problem with search engines, not the lack of pages with information. It seems I always find what I'm looking for eventually, what I need improved is the time I spend looking though spam-bomb pages before I find a page with the correct information.
These spam-pages seem to be increasing; I mean those pages with just a buch of keywords or the output of some search system.
Re:More pages v.s more relevant pages
by
Kithraya
·
· Score: 5, Insightful
I'm especially irritated by the increasing number of highly-ranked pages that are nothing more than another search engine's results. If Google could find some way to identify and remove these from my result set, Google's usefulness to me would increase 10 times over.
Re:More pages v.s more relevant pages
by
metlin
·
· Score: 2, Interesting
Google has a problem with this because some of those searches are actually useful.
For instance, when I search for something technical, I often run into search results from DBLP, arXiv, CiteSeer and the like -- although these are really search results within themselves, they're immensely useful to me.
Since we both effectively have a conflict of interest - Google would need to figure out a way to strike a balance.
Re:More pages v.s more relevant pages
by
__aahlyu4518
·
· Score: 2, Informative
Personally I find that the lack of relevant pages if the biggest problem with search engines, not the lack of pages with information.
Actually.... information IS relevant data. If it's not relevant to what you want, then it is just data...
Re:More pages v.s more relevant pages
by
corrie
·
· Score: 2, Interesting
However, results from places like Starware Search are not useful, and elevates my blood pressure with all the attempts at spamming me.
Just because I use Firefox and Adblock doesn't mean I now want to visit all possible spam sites in existence.
I don't care if Starware and friends make their money from advertising or not. The point is that Google is ALREADY a search engine, and a pretty good one at that. What is the point of returning results from another search engine, especially if the other one does not even have specialised domain?
Re:More pages v.s more relevant pages
by
jez9999
·
· Score: 4, Interesting
One thing that would really help me sometimes would be if Google allowed you to do an 'exact match' search. No, I don't mean enclosing something in double quotes, that still ignores capitalization, whitespace, and most non-letter characters. I'd like to be able to search for pages that have the EXACT string '#windows EFNET', for example, or '/usr/bin/' or whatever. '/Usr/biN' wouldn't match, and nor would '#windows^^EFNET' (where ^ is equal to a space:-) ).
I sent an e-mail to Google about this and the guy who replied didn't seem to think it was possible... anyone know if it is?
Re:More pages v.s more relevant pages
by
PsychoSlashDot
·
· Score: 5, Insightful
What I've read on the Google help pages seems to indicate that they don't index punctuation or capitalization. When you search for something, your string is looked for within an existing index, and appropriate reference materials are shown. Including punctuation wouldn't result in any hits within their index, meaning no results.
Now, obviously, it is theoretically possible to do just about anything. But in this case, with the architecture they have in place, anyone ever doing what you're asking would require a full-text search through their multi-TB dataset, which I suspect is highly impractical.
My point is that as I understand it, Google has coded a number of shortcut tricks which allow reasonable search times, and full-text string-exact searching would prevent them from using those shortcuts, resulting in search times they don't seem to think is reasonable.
-- "Oh no... he found the.sig setting."
Re:More pages v.s more relevant pages
by
Erasmus+Darwin
·
· Score: 3, Interesting
"But in this case, with the architecture they have in place, anyone ever doing what you're asking would require a full-text search through their multi-TB dataset, which I suspect is highly impractical."
Actually, they could cut that down considerably. For example, say we were doing an exact search for '#windows EFNET' as in the original example. The first thing they could do is start with a traditional search on "#windows EFNET". At that point, they've cut their multi-TB dataset down to just a few megs or less of likely matches (in this case, only 10 pages matched). Then they could do a full-text check on each result, looking for an exact match and discarding all the rest.
Re:More pages v.s more relevant pages
by
PMuse
·
· Score: 2, Insightful
How about a NEAR operator? Sure, AND OR NOT are nice, but my results would be a lot more relevant if I could eliminate results where the search terms appeared a thousand words apart.
-- "We reject as false the choice between our safety and our ideals." --The American President (20.1.2009)
Do this affect how fresh their index will be?
by
Jugalator
·
· Score: 3, Insightful
I wonder if it'll take longer to index twice as many pages? Or if they, along with this change, improved their spider and/or added hardware. Otherwise I'm not sure this change is for the better, unless you like to search for really obscure topics.
-- Beware: In C++, your friends can see your privates!
What is new about this.
by
hanssprudel
·
· Score: 3, Interesting
What the article does not point out is why this something important. For just about forever google's store has been coverging on 2**32 documents. Some people have speculated that Google simply could not update their 100,000+ servers with a new system that allowed more. Apparently they have now done the necessary architecture changes to allow for identifying documents by 64 bit (or more identifiers) and back in the business of making their search for comprehensive.
Good timing to conincide with MSN attempt to start a new searchengine too!
Re:What is new about this.
by
Jugalator
·
· Score: 3, Interesting
Good timing to conincide with MSN attempt to start a new searchengine too!
Yes, they'd better fight back, as they now have a serious competitor in MSN. It's giving very accurate results.
Doesn't anyone find it strange that Google gave the same top result there a while back?
MSN must be using a very similar algorithm.
Maybe a bit too similar...?
*tinfoil hat on*
-- Beware: In C++, your friends can see your privates!
Re:What is new about this.
by
slavemowgli
·
· Score: 2, Insightful
I don't quite believe that Google would've limited themselves that way (using 32 bit identifiers for documents) - that would've been incredibly short-sighted.
-- quidquid latine dictum sit altum videtur.
Re:What is new about this.
by
Anonymous Coward
·
· Score: 4, Interesting
For just about forever google's store has been coverging on 2**32 documents. Some people have speculated that Google simply could not update their 100,000+ servers with a new system that allowed more. Apparently they have now done the necessary architecture changes to allow for identifying documents by 64 bit (or more identifiers) and back in the business of making their search for comprehensive.
As someone who routinely follows these things, I couldn't agree more with your statement. My company operates a number of sites, and over the past 6 months, we've seen an obvious trend. Sites with, say, 5000+ pages, which used to be entirely indexed in Google, gradually had pages lost from Google. A search for site:somesite.com would return 5000 results 6 months ago. 3 or 4 months ago, the same search gave maybe 1000 results. This month maybe 500 or 600. We were definitely of the opinion that Google's index was "maxxed out" and was dropping large portions of indexed sites in favor of attempting to index new sites.
Now after seeing this story, I did a search and found literally all 5000+ pages are indexed once again. This is a huge step forward for webmasters everywhere. If your site had been slowly edged out of Google's index it's most likely back in its entirety now.
Thanks G.
Re:What is new about this.
by
Jugalator
·
· Score: 2, Insightful
Wow, Microsoft must have fixed it... It now no longer shows microsoft.com as top hit.
Haha, I guess the joke reached MS headquarters.:-P
-- Beware: In C++, your friends can see your privates!
Re:What is new about this.
by
bighoov
·
· Score: 3, Interesting
Probably not short sighted, but rather an space and cpu efficiency issue. Space - If you have 64-bit doc ids, even if you index 2^48 documents you're still wasting 16 bits per stemmed word per document. CPU - dealing with 64-bit integers on 32-bit hardware usually involves multiple loads, and decreases what can fit in the hardware data caches.
no update on the images
by
bvdbos
·
· Score: 3, Informative
Google makes minor change to website - news at 11!
by
Sanity
·
· Score: 3, Insightful
Does every minor Google or Apple related thing deserve a slashdot story? Can slashdot create a "Fanboy" section for insignificant stories advocating Google (with their software patent) and Apple (with their iTunes DRM)? That way I could filter them out more easily.
Quality - not quantity
by
seanyboy
·
· Score: 3, Insightful
Google needs to stop obsessing about the number of indexed pages, and start concentrating on the quality. Since pagerank was switched off, 2 out of 5 searches now seem to be jammed with pages full of nothing but random words and adverts. It's even more galling when the adverts are Google Ads. Much as I love Google, they're becoming increasingly less effective as a tool.
-- Training monkeys for world domination since 1439
Re:Quality - not quantity
by
Ingolfke
·
· Score: 3, Funny
I agree search engines are so 1990. I rely exlusively on word of mouth to find websites. If Firefox would add a button to the toolbar that said 'Cool Sites', maybe with an icon of a pair of glasses, and have the button link to a webpage with links to the latest cool sites on the net, that would certainly be the end of Google and their 8 billion pages. Pah!
Re:Quality - not quantity
by
Onionesque
·
· Score: 2, Insightful
To paraphrase Churchill, Google is the worst system devised by the wit of man, except for all the others. Where else would you go? Yahoo? Hey, how about AltaVista?
The problems faced by Google in their battle against the scumbags who would game the system are faced by every other search engine. Google, IMHO, handles them better.
Re:Quality - not quantity
by
dabadab
·
· Score: 3, Informative
"[i]Since pagerank was switched off[/i]"
Since when is Pagerank switched off?
-- Real life is overrated.
Re:Quality - not quantity
by
seanyboy
·
· Score: 4, Interesting
My bad. I'd skimmed a few things on the web, and assumed that it had been switched off. Looks instead as though Google have changed how it works. See PageRank is dead. I need to investigate further.
-- Training monkeys for world domination since 1439
And I for one welcome...
by
mu22le
·
· Score: 2, Funny
No, wait, they are our internet search overlords since, like, 1999?
Mhm to anonymous coward or not to anonymous coward? Will moderators smack my karma below zero?
Re:This is news ?
by
PerpetualMotion
·
· Score: 3, Interesting
A bigger index does not equal better search results, however, with the press this will generate, it will equal profits.
Re:Google Schmoogle
by
seanyboy
·
· Score: 2, Funny
Makes you wonder...
by
manmanic
·
· Score: 5, Insightful
Does this mean that I've been missing a huge amount of important information until now? I'd just assumed that Google covered the entire relevant web but now it seems to cover the whole same amount again. My Google alerts also seem to have started producing a lot more results which suggest that a lot of these new pages are rated quite highly. Who knows how much more quality content on the web we're just not seeing?
Re:Makes you wonder...
by
jlar
·
· Score: 5, Interesting
"Does this mean that I've been missing a huge amount of important information until now?"
Maybe the steep increase is due to all the new file formats they are indexing now. That might be useful for some people (although I sometimes find it kind of annoying that a search returns MS-Word documents).
Re:Makes you wonder...
by
RedWizzard
·
· Score: 2, Informative
Maybe the steep increase is due to all the new file formats they are indexing now.
The steep increase is probably due to an architecture change. Google has, for a long time, been indexing around 4 billion pages. That implies that they have been giving each page a 32 bit unique identifier, and had exhausted that id space. It would be a lot of work for them to seamlessly upgrade all their software to support a larger id, and it has taken them a long time to do so. Now that they have the large jump in pages is simply due to the fact that they can index much more of the web.
Google needs your cookie badly
by
Anonymous Coward
·
· Score: 2, Informative
Until today you could save your google settingswithout loosing your privacy.
You can still save those settings but google refuses to use them when you block their cookie. In my case I
get 10 search results although I like to receive 100. Seems that they are
making many dollars on a user's cookie, and now they are a public company my
privacy is less important than "stock holders' interests".
Re:Google needs your cookie badly
by
Anonymous Coward
·
· Score: 3, Informative
You can still save those settings but google refuses to use them when you block their cookie. In my case I get 10 search results although I like to receive 100.
Give it the keyword 100, then type 100 search_term in the address bar to use it.
Re:Google makes minor change to website - news at
by
dotmike
·
· Score: 2, Funny
At the same time, can Slashdot create a "Curmudgeon" section for those who like to gripe about the less than monumental significance of some story topics?
Google domination.
by
Anonymous Coward
·
· Score: 2, Informative
Local tabloid Aftonbladet is running a poll on search engine use:
Google (81.4 %)
Yahoo (2.2 %)
MSN (3.8 %)
Other (11.4 %)
Don't know (1.2 %)
61730 votes so far.
I'm a little surprised, either the masses who use the "default" (MSN?) aren't bothering to answer, or google is simply very very dominant and those "default using masses" do not exist [in this country].
Re:Google domination.
by
Mostly+a+lurker
·
· Score: 2, Insightful
the masses who use the "default" (MSN?) aren't bothering to answer
I think it is more that many users of IE just do not twig that their failed page access resulted in an automatic query to MSN.
In reality, most users make occasional deliberate queries to Google and more frequent accidental queries to MSN.
If I kept eating so much spam...
by
dos_dude
·
· Score: 2, Funny
... my weight would probably double, too.
Re:This is news ?
by
Ford+Prefect
·
· Score: 2, Interesting
A bigger index does not equal better search results, however, with the press this will generate, it will equal profits.
It would be terribly easy to get trillions of pages indexed. For instance, a site I've been working on has a public calendar system, with results fished out of a database. There are very few actual events in it at the moment, but with the 'Previous' and 'Next' links it'll run from 1970 to 2038. A naïve web-crawler would index every single month for every single year, but Google would appear to have crawled over just a few, presumably flagging the pages as too similar to warrant further investigation.
With stuff like public web forums, Slashdot and the like, I can easily imagine comparatively small sites producing thousands of pages apiece. Is there useful information in there? Quite possibly, but it definitely needs treating in a different manner to an old-fashioned, static-pages-only site...
A lot of people have been asking what the point of the artical is, why does it matter, well possibly because Microsoft announced the launch of their search engine http://news.bbc.co.uk/1/hi/technology/4000015.stm and are claiming more pages index than google (5 billion) so google have responded by effectivly doubling their pages indexed.
Mine is bigger than yours!!!
by
ayjay29
·
· Score: 4, Informative
Now it's going to be even harder to get my name in the top spot. Why was I cursed with the surname Smith!
-- I used to have a better sig but it broke.
Searching LiveJournal.com
by
hackrobat
·
· Score: 4, Informative
Looks like they've added a gazillion LiveJournal pages to their index. I used to have a Google search box on my LJ that didn't throw up relevant results until last week or so. Now it works perfectly, just like builtin search (like what you see in MT and WordPress).
Re:Searching LiveJournal.com
by
cavemanf16
·
· Score: 2, Insightful
MSN's "msnbot" has been crawling/spidering my webserver (which runs Geeklog and is just another blog of my random crap) pretty extensively for weeks now. (Lie 5 times a day it seems) Searching on Google for my site's name now reveals more results from my site, but not a lot of those circle-jerk style search results pages that are just trying to generate some ad revenues. However, using the beta.search.msn.com site DOES yield a lot more random crap (mostly blogs and personal webservers) that somehow generated some kind of link to my site because of the title of one of my articles, someone linking to my site in one of their blog posts, etc.
I have a feeling MSN's new search site is gonna be mostly blogs and advertisements, not relevant information. I think it's good Google has indexed more pages, but I still believe their algorithm will continue to provide more USEFUL results than MSN. (BTW, the googlebot doesn't hit my site too frequently which tells me Google's bot understands that my site isn't updated too frequently, nor is it linked to from other important sites)
Doubled? Wait a minute...
by
't+is+DjiM
·
· Score: 5, Funny
From 4 to 8 billion pages... I guess they just indexed the google cache...
-- --Use ant to make.war
Competing with Microsoft's 5bn?
by
Richard+W.M.+Jones
·
· Score: 4, Informative
On the same day that
this story hits the BBC. In that
story Microsoft claim that they have
5 billion pages indexed, more than
the 4.2 billion pages indexed (at that
point) by Google. The BBC have just
updated the story with the 8bn figure.
Yes, this is probably a troll, but anyway... I take it you've never heard of the robots.txt file? You sound like you might want to read up on it. It's designed to help control the spidering of your pages for whatever reason, particularly cases like yours or situations where a spider would get confused and end up doing something stupid (recursive stuff, etc).
-ReK
-- md5sum -c reality.md5
reality: FAILED
md5sum: WARNING: 1 of 1 computed checksum did NOT match
meta-no-archive
by
Anonymous Coward
·
· Score: 3, Interesting
apparently my sites will never get a good ranking on google because I don't want the search engine to cache the site. So I'm using meta no-archive tags. That's the only thing I can figure out why the sites rank so poorly on google, when they come up in the top 10-20 hits on yahoo and other search engines. The keywords for the searches are valid, the sites are relevant to the keyword searches, yet the sites don't show in the top 100-300 on google.
I've avoided all the usual spam type of tags (auto refreshing, hidden text, cloaking, etc.) and the sites are legitimate and on the up and up, and yet the only page or two that google is spidering are the one or few that appear to be without the no-archive tags and possibly the revisit/expire tags.
Is google's policy, allows us to cache your site, or get penalized? Anyone else run into a similar problem or can shed some light on this? The only other thing I can think of is the robots text file, that keeps googlebot, and then other spiders through a *, from entering images directories. The spiders, including googlebot, aren't restricted from entering any other directories, they are given free reign.
Anyone else with problems with no-cache, no archive, tight revisit/expire times, or similar non-spam tags that result in penalties in google ranking?
I've been using google exclusively for a few years now. But the poor page ranking of sites on my server got me wondering about other sites that may be relevant to my own searches which may be exluded or penalized by google. So I've started using Yahoo search again, as much as I hate Yahoo (what they do with advertising to Yahoo groups and Yahoo mail is a shame). It appears that Yahoo is including better results because other sites show up with higher ranking that actually are relevant. So I've learned that Google isn't as perfect as I thought it was, which was disappointing in itself. It was easy using one search site. Now I have to use two to make sure I'm getting good results. Anyone know if there is a plugin for Firefox with both Google and Yahoo search boxes on the toolbar?
Re:meta-no-archive
by
justMichael
·
· Score: 2, Informative
I'm sure I've seen some way of doing a sort of backwards search on a page, that will show all the pages in Google that link to it.
It isn't about having a better search engine, so much as it is knowing how to use it. If you are looking for information on a recipe for oriental rice using asian spice, how would you search?
Bad search example:
oriental rice recipe asian spice
Good search example:
recipe+"oriental rice"+spice
See the difference? google tries its best to get rid of the spam pages, but it won't ever combat them all. Half of the work has to be done with you understanding the best way to describe to the search engine, what it is you want to do. The better you explain it, the better it can search for you.
-- "We're breaking out the ramen noodles. . . "
"Really? Is it someone's birthday?"
Re:Google thieves my bandwidth
by
Rakshasa+Taisab
·
· Score: 2, Insightful
You can rant all you want, but Google still has a fair use right to your images. They are reduced resolution images and therefor legal for non-commercial use.
Not to mention robot.txt, but that is so obvious it shouldn't need to be mention.
-- -
These characters were randomly selected.
great but where are the .txt and directories?
by
js7a
·
· Score: 2, Informative
Google won't be within reach of the pinnacle until they index.txt files, directory listings, and anonymous ftp sites.
Re:great but where are the .txt and directories?
by
geminidomino
·
· Score: 2, Informative
One out of 3 ain't a bad start. Add a few more keywords to narrow down the google-crawling.
Re:Google thieves my bandwidth
by
jvj24601
·
· Score: 5, Informative
Well, if you know that Google is indexing your site and "stealing" your bandwidth, then you must have looked at the server logs, right? You'd see the name of the search bot is googlebot. Search for it, and you'll find that the first relevant link explains how to prevent googlebot from accessing your site.
The logs would probably also show failed attempts to find the file/robots.txt. Similar info is gained from searching on that term as well.
Search terms: oriental rice recipe asian spice Search Results: Results 1 - 10 of about 254,000 for oriental rice recipe asian spice . (0.40 seconds) Search Effectiveness: REASONABLE. good list of relivent items matched.
Search terms: recipe+"oriental rice"+spice Search Results: Your search - recipe+"oriental rice"+spice - did not match any documents. Search Effectiveness: UTTER SHITE
The user wants SIMPLICITY. If google cannot give decent results for simple search criteria, then people will go elsewhere.
Its the KISS principle in effect.
-- liqbase:: faster than paper
So, to sum up...
by
kahei
·
· Score: 5, Insightful
I am feeding this troll because there are people who really _do_ think like that and I wish I could yell at them to their faces:)
You put content in a place where it is publically accessible. You explicitly and proactively made that content available to everyone, including 'the average surfer' and googlebots. You took no steps to make it available only to the select few of whom you approve.
Now you are all cross and bothered because average surfers / googlebots have read / copied your content, such as it is.
The solution is to drown yourself in a bucket. I have a bucket.
-- Whence? Hence. Whither? Thither.
Proximity search will help
by
Sai+Babu
·
· Score: 3, Insightful
This is why I've been begging google folks to implement NEAR operator!
Erm, that's only because of the bizarre plus signs the grandparent poster put in - try this. Note to grandparent: Just about any modern search engine assumes words not prefixed by anything are to be included in the Boolean search query. No need for +.
I know why it has doubled...
by
jmcmunn
·
· Score: 2, Interesting
Because every blogger in the universe has added at least 3 pages since the last index. I fail to see how it is significant to me that there are now 8 billion mostly worthless sites out there. The number of actually useful sites has not gone up considerably.
Re:Read more carefully.
by
Mant
·
· Score: 4, Insightful
Robots.txt isn't some thing that only applies to Google, it is (supposed) to be honoured by all search engines, and uses the Robots Exclusion Standard. So, when you claim these are Google's arbitary rules, you are in fact wrong. They are neither Google's nor arbitary (at least no more than any web standard).
So your clue, not so much of clue, as robots.txt doesn't fit your description.
As for why you should know about it, you are putting up a web site, it is part of running a web site. You might as well complain why you need to know about HTML, CSS or registering a domain name. Quite what coming from the UK has to do with it (something I also do), I have no idea.
"I simply do not want the average surfer to be able to visit my site, I am not interested in serving my pages to them, they simply would not appreciate or understand what it is I am showing."
Then a publicly accessable webiste is the wrong place. It is not your personal space, and it isn't private. You made it available to the world, nobody made you. To turn around and complain when (some of) the world visits it is hypocracy.
It's like putting up posters around a town, then running around complaining all these people are looking at them, won't appreciate them, and you don't want them too. It's also comes across as condescending and arrogant, which probably explains the nastiness of some of the responses.
You opted in when you put up the publicly accessable website. If all search engines had to be opt in, nobody could find anything on the web, and it would use a lot of its utility. Your assumed to want them crawling becuase the vast majority of people do, they want their site to be found. If you don't though, no problem, just use the standards for stopping searches, or password protect the site. No scandal at all, just hysterics.
Showing the low res thumbnail of your image isn't violating your copyright either. The only legitimate claim you have is the amount of time it took to remove something from the cache.
The "thieves" accusation is even more ridiculous. If you put something up on the web people can see for free, you can't complain. There are options if you want to protect it. Google doesn't claim you work as theirs (which would be 'stealing' or at least copyright violation), they help people find you public web site.
If you don't want a public website but made one, whose fault is that? If you are going to run a website and can't be bothered to find out how to do it properly, you can't blame Google.
Slashdot Mirror
Have they updated image search yet?
Personally I find that the lack of relevant pages if the biggest problem with search engines, not the lack of pages with information. It seems I always find what I'm looking for eventually, what I need improved is the time I spend looking though spam-bomb pages before I find a page with the correct information.
These spam-pages seem to be increasing; I mean those pages with just a buch of keywords or the output of some search system.
9/11: Never forget it was a false-flag operation
8 billion pages and not a single link to my blog.
/.
Can't figure of I should just shoot my self or maybe just open a subscription to
TC - My Photos..
I wonder if it'll take longer to index twice as many pages? Or if they, along with this change, improved their spider and/or added hardware. Otherwise I'm not sure this change is for the better, unless you like to search for really obscure topics.
Beware: In C++, your friends can see your privates!
What the article does not point out is why this something important. For just about forever google's store has been coverging on 2**32 documents. Some people have speculated that Google simply could not update their 100,000+ servers with a new system that allowed more. Apparently they have now done the necessary architecture changes to allow for identifying documents by 64 bit (or more identifiers) and back in the business of making their search for comprehensive.
Good timing to conincide with MSN attempt to start a new searchengine too!
Unfortunately they didn't update the image-search yet.
Does every minor Google or Apple related thing deserve a slashdot story? Can slashdot create a "Fanboy" section for insignificant stories advocating Google (with their software patent) and Apple (with their iTunes DRM)? That way I could filter them out more easily.
Google needs to stop obsessing about the number of indexed pages, and start concentrating on the quality. Since pagerank was switched off, 2 out of 5 searches now seem to be jammed with pages full of nothing but random words and adverts. It's even more galling when the adverts are Google Ads. Much as I love Google, they're becoming increasingly less effective as a tool.
Training monkeys for world domination since 1439
No, wait, they are our internet search overlords since, like, 1999?
Mhm to anonymous coward or not to anonymous coward?
Will moderators smack my karma below zero?
A bigger index does not equal better search results, however, with the press this will generate, it will equal profits.
They already have.
Training monkeys for world domination since 1439
In case of slashdotting use this mirror.
In Soviet America the banks rob you!
Yeah, but it'd be news if the sun set twice in one night or rose twice as bright.
It's more the exponential increase in the size of the index rather than the piecemeal addition.
over eight billion pages crawled
You don't just go from 4 billion to 8 billion overnight.
They are probably just crawling the same 4 billion twice.
Blearf. Blearf, I say.
Does this mean that I've been missing a huge amount of important information until now? I'd just assumed that Google covered the entire relevant web but now it seems to cover the whole same amount again. My Google alerts also seem to have started producing a lot more results which suggest that a lot of these new pages are rated quite highly. Who knows how much more quality content on the web we're just not seeing?
Until today you could save your google settings without loosing your privacy. You can still save those settings but google refuses to use them when you block their cookie. In my case I get 10 search results although I like to receive 100. Seems that they are making many dollars on a user's cookie, and now they are a public company my privacy is less important than "stock holders' interests".
At the same time, can Slashdot create a "Curmudgeon" section for those who like to gripe about the less than monumental significance of some story topics?
Local tabloid Aftonbladet is running a poll on search engine use:
61730 votes so far.
I'm a little surprised, either the masses who use the "default" (MSN?) aren't bothering to answer, or google is simply very very dominant and those "default using masses" do not exist [in this country].
... my weight would probably double, too.
A bigger index does not equal better search results, however, with the press this will generate, it will equal profits.
It would be terribly easy to get trillions of pages indexed. For instance, a site I've been working on has a public calendar system, with results fished out of a database. There are very few actual events in it at the moment, but with the 'Previous' and 'Next' links it'll run from 1970 to 2038. A naïve web-crawler would index every single month for every single year, but Google would appear to have crawled over just a few, presumably flagging the pages as too similar to warrant further investigation.
With stuff like public web forums, Slashdot and the like, I can easily imagine comparatively small sites producing thousands of pages apiece. Is there useful information in there? Quite possibly, but it definitely needs treating in a different manner to an old-fashioned, static-pages-only site...
Tedious Bloggy Stuff - hooray?
Google respects the robots.txt file. Use it.
A lot of people have been asking what the point of the artical is, why does it matter, well possibly because Microsoft announced the launch of their search engine http://news.bbc.co.uk/1/hi/technology/4000015.stm and are claiming more pages index than google (5 billion) so google have responded by effectivly doubling their pages indexed.
From BBC News here.
In a statement Microsoft said its search engine returned results from five billion web pages - more than any other search engine.
But this quickly won a response from Google which announced that its index has now grown to more than 8 billion pages.
Prior to the Microsoft announcement, Google was only indexing 4,285,199,774 web pages.
Steve Ballmer is soon to announce that his daddy is one hundrad years old, and kan kick your daddy's ass...
Offtopic, Inflammatory, Inappropriate, Illegal, or Offensive comments might be moderated up.
Now it's going to be even harder to get my name in the top spot. Why was I cursed with the surname Smith!
I used to have a better sig but it broke.
Looks like they've added a gazillion LiveJournal pages to their index. I used to have a Google search box on my LJ that didn't throw up relevant results until last week or so. Now it works perfectly, just like builtin search (like what you see in MT and WordPress).
From 4 to 8 billion pages... I guess they just indexed the google cache...
--Use ant to make
I smell competition!
Rich.
libguestfs - tools for accessing and modifying virtual machine disk images
Does this mean twice as many pages with "Search for 'printer problem linux' on Kelkoo"?
Yes, this is probably a troll, but anyway... I take it you've never heard of the robots.txt file? You sound like you might want to read up on it. It's designed to help control the spidering of your pages for whatever reason, particularly cases like yours or situations where a spider would get confused and end up doing something stupid (recursive stuff, etc).
-ReK
md5sum -c reality.md5
reality: FAILED
md5sum: WARNING: 1 of 1 computed checksum did NOT match
apparently my sites will never get a good ranking on google because I don't want the search engine to cache the site. So I'm using meta no-archive tags. That's the only thing I can figure out why the sites rank so poorly on google, when they come up in the top 10-20 hits on yahoo and other search engines. The keywords for the searches are valid, the sites are relevant to the keyword searches, yet the sites don't show in the top 100-300 on google.
I've avoided all the usual spam type of tags (auto refreshing, hidden text, cloaking, etc.) and the sites are legitimate and on the up and up, and yet the only page or two that google is spidering are the one or few that appear to be without the no-archive tags and possibly the revisit/expire tags.
Is google's policy, allows us to cache your site, or get penalized? Anyone else run into a similar problem or can shed some light on this? The only other thing I can think of is the robots text file, that keeps googlebot, and then other spiders through a *, from entering images directories. The spiders, including googlebot, aren't restricted from entering any other directories, they are given free reign.
Anyone else with problems with no-cache, no archive, tight revisit/expire times, or similar non-spam tags that result in penalties in google ranking?
I've been using google exclusively for a few years now. But the poor page ranking of sites on my server got me wondering about other sites that may be relevant to my own searches which may be exluded or penalized by google. So I've started using Yahoo search again, as much as I hate Yahoo (what they do with advertising to Yahoo groups and Yahoo mail is a shame). It appears that Yahoo is including better results because other sites show up with higher ranking that actually are relevant. So I've learned that Google isn't as perfect as I thought it was, which was disappointing in itself. It was easy using one search site. Now I have to use two to make sure I'm getting good results. Anyone know if there is a plugin for Firefox with both Google and Yahoo search boxes on the toolbar?
It isn't about having a better search engine, so much as it is knowing how to use it. If you are looking for information on a recipe for oriental rice using asian spice, how would you search?
Bad search example:
oriental rice recipe asian spice
Good search example:
recipe+"oriental rice"+spice
See the difference? google tries its best to get rid of the spam pages, but it won't ever combat them all. Half of the work has to be done with you understanding the best way to describe to the search engine, what it is you want to do. The better you explain it, the better it can search for you.
"We're breaking out the ramen noodles. . . "
"Really? Is it someone's birthday?"
You can rant all you want, but Google still has a fair use right to your images. They are reduced resolution images and therefor legal for non-commercial use.
Not to mention robot.txt, but that is so obvious it shouldn't need to be mention.
- These characters were randomly selected.
Google won't be within reach of the pinnacle until they index .txt files, directory listings, and anonymous ftp sites.
Well, if you know that Google is indexing your site and "stealing" your bandwidth, then you must have looked at the server logs, right? You'd see the name of the search bot is googlebot. Search for it, and you'll find that the first relevant link explains how to prevent googlebot from accessing your site.
/robots.txt. Similar info is gained from searching on that term as well.
The logs would probably also show failed attempts to find the file
I see the difference...
Search terms: oriental rice recipe asian spice
Search Results: Results 1 - 10 of about 254,000 for oriental rice recipe asian spice . (0.40 seconds)
Search Effectiveness: REASONABLE. good list of relivent items matched.
Search terms: recipe+"oriental rice"+spice
Search Results: Your search - recipe+"oriental rice"+spice - did not match any documents.
Search Effectiveness: UTTER SHITE
The user wants SIMPLICITY. If google cannot give decent results for simple search criteria, then people will go elsewhere.
Its the KISS principle in effect.
liqbase
I am feeding this troll because there are people who really _do_ think like that and I wish I could yell at them to their faces
You put content in a place where it is publically accessible. You explicitly and proactively made that content available to everyone, including 'the average surfer' and googlebots. You took no steps to make it available only to the select few of whom you approve.
Now you are all cross and bothered because average surfers / googlebots have read / copied your content, such as it is.
The solution is to drown yourself in a bucket. I have a bucket.
Whence? Hence. Whither? Thither.
This is why I've been begging google folks to implement NEAR operator!
Here is an example msn search: http://search.msn.com/results.aspx?FORM=SMCRT&q=f
Now I'm the grandest Tiger in the Jungle!
Erm, that's only because of the bizarre plus signs the grandparent poster put in - try this. Note to grandparent: Just about any modern search engine assumes words not prefixed by anything are to be included in the Boolean search query. No need for +.
== Jez ==
Do you miss Firefox? Try Pale Moon.
It is not clear to me how I can help them improve. Suggest they switch their servers to Linux?
Yes there is, try to search for
The Doctor
vs
+the doctor
Because every blogger in the universe has added at least 3 pages since the last index. I fail to see how it is significant to me that there are now 8 billion mostly worthless sites out there. The number of actually useful sites has not gone up considerably.
Robots.txt isn't some thing that only applies to Google, it is (supposed) to be honoured by all search engines, and uses the Robots Exclusion Standard. So, when you claim these are Google's arbitary rules, you are in fact wrong. They are neither Google's nor arbitary (at least no more than any web standard).
So your clue, not so much of clue, as robots.txt doesn't fit your description.
As for why you should know about it, you are putting up a web site, it is part of running a web site. You might as well complain why you need to know about HTML, CSS or registering a domain name. Quite what coming from the UK has to do with it (something I also do), I have no idea.
"I simply do not want the average surfer to be able to visit my site, I am not interested in serving my pages to them, they simply would not appreciate or understand what it is I am showing."
Then a publicly accessable webiste is the wrong place. It is not your personal space, and it isn't private. You made it available to the world, nobody made you. To turn around and complain when (some of) the world visits it is hypocracy.
It's like putting up posters around a town, then running around complaining all these people are looking at them, won't appreciate them, and you don't want them too. It's also comes across as condescending and arrogant, which probably explains the nastiness of some of the responses.
You opted in when you put up the publicly accessable website. If all search engines had to be opt in, nobody could find anything on the web, and it would use a lot of its utility. Your assumed to want them crawling becuase the vast majority of people do, they want their site to be found. If you don't though, no problem, just use the standards for stopping searches, or password protect the site. No scandal at all, just hysterics.
Showing the low res thumbnail of your image isn't violating your copyright either. The only legitimate claim you have is the amount of time it took to remove something from the cache.
The "thieves" accusation is even more ridiculous. If you put something up on the web people can see for free, you can't complain. There are options if you want to protect it. Google doesn't claim you work as theirs (which would be 'stealing' or at least copyright violation), they help people find you public web site.
If you don't want a public website but made one, whose fault is that? If you are going to run a website and can't be bothered to find out how to do it properly, you can't blame Google.