geekfiend writes "Today Google updated their website to indicate over eight billion pages crawled, cached and indexed. They've also added an entry to their blog explaining that they still have tons of work to do."
More pages v.s more relevant pages
by
xiando
·
· Score: 5, Insightful
Personally I find that the lack of relevant pages if the biggest problem with search engines, not the lack of pages with information. It seems I always find what I'm looking for eventually, what I need improved is the time I spend looking though spam-bomb pages before I find a page with the correct information.
These spam-pages seem to be increasing; I mean those pages with just a buch of keywords or the output of some search system.
Re:More pages v.s more relevant pages
by
Kithraya
·
· Score: 5, Insightful
I'm especially irritated by the increasing number of highly-ranked pages that are nothing more than another search engine's results. If Google could find some way to identify and remove these from my result set, Google's usefulness to me would increase 10 times over.
Re:More pages v.s more relevant pages
by
jez9999
·
· Score: 4, Interesting
One thing that would really help me sometimes would be if Google allowed you to do an 'exact match' search. No, I don't mean enclosing something in double quotes, that still ignores capitalization, whitespace, and most non-letter characters. I'd like to be able to search for pages that have the EXACT string '#windows EFNET', for example, or '/usr/bin/' or whatever. '/Usr/biN' wouldn't match, and nor would '#windows^^EFNET' (where ^ is equal to a space:-) ).
I sent an e-mail to Google about this and the guy who replied didn't seem to think it was possible... anyone know if it is?
Re:More pages v.s more relevant pages
by
PsychoSlashDot
·
· Score: 5, Insightful
What I've read on the Google help pages seems to indicate that they don't index punctuation or capitalization. When you search for something, your string is looked for within an existing index, and appropriate reference materials are shown. Including punctuation wouldn't result in any hits within their index, meaning no results.
Now, obviously, it is theoretically possible to do just about anything. But in this case, with the architecture they have in place, anyone ever doing what you're asking would require a full-text search through their multi-TB dataset, which I suspect is highly impractical.
My point is that as I understand it, Google has coded a number of shortcut tricks which allow reasonable search times, and full-text string-exact searching would prevent them from using those shortcuts, resulting in search times they don't seem to think is reasonable.
Makes you wonder...
by
manmanic
·
· Score: 5, Insightful
Does this mean that I've been missing a huge amount of important information until now? I'd just assumed that Google covered the entire relevant web but now it seems to cover the whole same amount again. My Google alerts also seem to have started producing a lot more results which suggest that a lot of these new pages are rated quite highly. Who knows how much more quality content on the web we're just not seeing?
Re:Makes you wonder...
by
jlar
·
· Score: 5, Interesting
"Does this mean that I've been missing a huge amount of important information until now?"
Maybe the steep increase is due to all the new file formats they are indexing now. That might be useful for some people (although I sometimes find it kind of annoying that a search returns MS-Word documents).
Re:What is new about this.
by
Anonymous Coward
·
· Score: 4, Interesting
For just about forever google's store has been coverging on 2**32 documents. Some people have speculated that Google simply could not update their 100,000+ servers with a new system that allowed more. Apparently they have now done the necessary architecture changes to allow for identifying documents by 64 bit (or more identifiers) and back in the business of making their search for comprehensive.
As someone who routinely follows these things, I couldn't agree more with your statement. My company operates a number of sites, and over the past 6 months, we've seen an obvious trend. Sites with, say, 5000+ pages, which used to be entirely indexed in Google, gradually had pages lost from Google. A search for site:somesite.com would return 5000 results 6 months ago. 3 or 4 months ago, the same search gave maybe 1000 results. This month maybe 500 or 600. We were definitely of the opinion that Google's index was "maxxed out" and was dropping large portions of indexed sites in favor of attempting to index new sites.
Now after seeing this story, I did a search and found literally all 5000+ pages are indexed once again. This is a huge step forward for webmasters everywhere. If your site had been slowly edged out of Google's index it's most likely back in its entirety now.
Thanks G.
Re:Google thieves my bandwidth
by
Anonymous Coward
·
· Score: 5, Informative
A lot of people have been asking what the point of the artical is, why does it matter, well possibly because Microsoft announced the launch of their search engine http://news.bbc.co.uk/1/hi/technology/4000015.stm and are claiming more pages index than google (5 billion) so google have responded by effectivly doubling their pages indexed.
Mine is bigger than yours!!!
by
ayjay29
·
· Score: 4, Informative
Now it's going to be even harder to get my name in the top spot. Why was I cursed with the surname Smith!
-- I used to have a better sig but it broke.
Searching LiveJournal.com
by
hackrobat
·
· Score: 4, Informative
Looks like they've added a gazillion LiveJournal pages to their index. I used to have a Google search box on my LJ that didn't throw up relevant results until last week or so. Now it works perfectly, just like builtin search (like what you see in MT and WordPress).
Doubled? Wait a minute...
by
't+is+DjiM
·
· Score: 5, Funny
From 4 to 8 billion pages... I guess they just indexed the google cache...
-- --Use ant to make.war
Competing with Microsoft's 5bn?
by
Richard+W.M.+Jones
·
· Score: 4, Informative
On the same day that
this story hits the BBC. In that
story Microsoft claim that they have
5 billion pages indexed, more than
the 4.2 billion pages indexed (at that
point) by Google. The BBC have just
updated the story with the 8bn figure.
Re:Quality - not quantity
by
seanyboy
·
· Score: 4, Interesting
My bad. I'd skimmed a few things on the web, and assumed that it had been switched off. Looks instead as though Google have changed how it works. See PageRank is dead. I need to investigate further.
-- Training monkeys for world domination since 1439
Re:Google thieves my bandwidth
by
jvj24601
·
· Score: 5, Informative
Well, if you know that Google is indexing your site and "stealing" your bandwidth, then you must have looked at the server logs, right? You'd see the name of the search bot is googlebot. Search for it, and you'll find that the first relevant link explains how to prevent googlebot from accessing your site.
The logs would probably also show failed attempts to find the file/robots.txt. Similar info is gained from searching on that term as well.
So, to sum up...
by
kahei
·
· Score: 5, Insightful
I am feeding this troll because there are people who really _do_ think like that and I wish I could yell at them to their faces:)
You put content in a place where it is publically accessible. You explicitly and proactively made that content available to everyone, including 'the average surfer' and googlebots. You took no steps to make it available only to the select few of whom you approve.
Now you are all cross and bothered because average surfers / googlebots have read / copied your content, such as it is.
The solution is to drown yourself in a bucket. I have a bucket.
Erm, that's only because of the bizarre plus signs the grandparent poster put in - try this. Note to grandparent: Just about any modern search engine assumes words not prefixed by anything are to be included in the Boolean search query. No need for +.
Re:Read more carefully.
by
Mant
·
· Score: 4, Insightful
Robots.txt isn't some thing that only applies to Google, it is (supposed) to be honoured by all search engines, and uses the Robots Exclusion Standard. So, when you claim these are Google's arbitary rules, you are in fact wrong. They are neither Google's nor arbitary (at least no more than any web standard).
So your clue, not so much of clue, as robots.txt doesn't fit your description.
As for why you should know about it, you are putting up a web site, it is part of running a web site. You might as well complain why you need to know about HTML, CSS or registering a domain name. Quite what coming from the UK has to do with it (something I also do), I have no idea.
"I simply do not want the average surfer to be able to visit my site, I am not interested in serving my pages to them, they simply would not appreciate or understand what it is I am showing."
Then a publicly accessable webiste is the wrong place. It is not your personal space, and it isn't private. You made it available to the world, nobody made you. To turn around and complain when (some of) the world visits it is hypocracy.
It's like putting up posters around a town, then running around complaining all these people are looking at them, won't appreciate them, and you don't want them too. It's also comes across as condescending and arrogant, which probably explains the nastiness of some of the responses.
You opted in when you put up the publicly accessable website. If all search engines had to be opt in, nobody could find anything on the web, and it would use a lot of its utility. Your assumed to want them crawling becuase the vast majority of people do, they want their site to be found. If you don't though, no problem, just use the standards for stopping searches, or password protect the site. No scandal at all, just hysterics.
Showing the low res thumbnail of your image isn't violating your copyright either. The only legitimate claim you have is the amount of time it took to remove something from the cache.
The "thieves" accusation is even more ridiculous. If you put something up on the web people can see for free, you can't complain. There are options if you want to protect it. Google doesn't claim you work as theirs (which would be 'stealing' or at least copyright violation), they help people find you public web site.
If you don't want a public website but made one, whose fault is that? If you are going to run a website and can't be bothered to find out how to do it properly, you can't blame Google.
Slashdot Mirror
Have they updated image search yet?
Personally I find that the lack of relevant pages if the biggest problem with search engines, not the lack of pages with information. It seems I always find what I'm looking for eventually, what I need improved is the time I spend looking though spam-bomb pages before I find a page with the correct information.
These spam-pages seem to be increasing; I mean those pages with just a buch of keywords or the output of some search system.
9/11: Never forget it was a false-flag operation
8 billion pages and not a single link to my blog.
/.
Can't figure of I should just shoot my self or maybe just open a subscription to
TC - My Photos..
In case of slashdotting use this mirror.
In Soviet America the banks rob you!
Does this mean that I've been missing a huge amount of important information until now? I'd just assumed that Google covered the entire relevant web but now it seems to cover the whole same amount again. My Google alerts also seem to have started producing a lot more results which suggest that a lot of these new pages are rated quite highly. Who knows how much more quality content on the web we're just not seeing?
Now after seeing this story, I did a search and found literally all 5000+ pages are indexed once again. This is a huge step forward for webmasters everywhere. If your site had been slowly edged out of Google's index it's most likely back in its entirety now.
Thanks G.
Google respects the robots.txt file. Use it.
A lot of people have been asking what the point of the artical is, why does it matter, well possibly because Microsoft announced the launch of their search engine http://news.bbc.co.uk/1/hi/technology/4000015.stm and are claiming more pages index than google (5 billion) so google have responded by effectivly doubling their pages indexed.
From BBC News here.
In a statement Microsoft said its search engine returned results from five billion web pages - more than any other search engine.
But this quickly won a response from Google which announced that its index has now grown to more than 8 billion pages.
Prior to the Microsoft announcement, Google was only indexing 4,285,199,774 web pages.
Steve Ballmer is soon to announce that his daddy is one hundrad years old, and kan kick your daddy's ass...
Offtopic, Inflammatory, Inappropriate, Illegal, or Offensive comments might be moderated up.
Now it's going to be even harder to get my name in the top spot. Why was I cursed with the surname Smith!
I used to have a better sig but it broke.
Looks like they've added a gazillion LiveJournal pages to their index. I used to have a Google search box on my LJ that didn't throw up relevant results until last week or so. Now it works perfectly, just like builtin search (like what you see in MT and WordPress).
From 4 to 8 billion pages... I guess they just indexed the google cache...
--Use ant to make
I smell competition!
Rich.
libguestfs - tools for accessing and modifying virtual machine disk images
My bad. I'd skimmed a few things on the web, and assumed that it had been switched off. Looks instead as though Google have changed how it works. See PageRank is dead. I need to investigate further.
Training monkeys for world domination since 1439
Well, if you know that Google is indexing your site and "stealing" your bandwidth, then you must have looked at the server logs, right? You'd see the name of the search bot is googlebot. Search for it, and you'll find that the first relevant link explains how to prevent googlebot from accessing your site.
/robots.txt. Similar info is gained from searching on that term as well.
The logs would probably also show failed attempts to find the file
I am feeding this troll because there are people who really _do_ think like that and I wish I could yell at them to their faces
You put content in a place where it is publically accessible. You explicitly and proactively made that content available to everyone, including 'the average surfer' and googlebots. You took no steps to make it available only to the select few of whom you approve.
Now you are all cross and bothered because average surfers / googlebots have read / copied your content, such as it is.
The solution is to drown yourself in a bucket. I have a bucket.
Whence? Hence. Whither? Thither.
Erm, that's only because of the bizarre plus signs the grandparent poster put in - try this. Note to grandparent: Just about any modern search engine assumes words not prefixed by anything are to be included in the Boolean search query. No need for +.
== Jez ==
Do you miss Firefox? Try Pale Moon.
Robots.txt isn't some thing that only applies to Google, it is (supposed) to be honoured by all search engines, and uses the Robots Exclusion Standard. So, when you claim these are Google's arbitary rules, you are in fact wrong. They are neither Google's nor arbitary (at least no more than any web standard).
So your clue, not so much of clue, as robots.txt doesn't fit your description.
As for why you should know about it, you are putting up a web site, it is part of running a web site. You might as well complain why you need to know about HTML, CSS or registering a domain name. Quite what coming from the UK has to do with it (something I also do), I have no idea.
"I simply do not want the average surfer to be able to visit my site, I am not interested in serving my pages to them, they simply would not appreciate or understand what it is I am showing."
Then a publicly accessable webiste is the wrong place. It is not your personal space, and it isn't private. You made it available to the world, nobody made you. To turn around and complain when (some of) the world visits it is hypocracy.
It's like putting up posters around a town, then running around complaining all these people are looking at them, won't appreciate them, and you don't want them too. It's also comes across as condescending and arrogant, which probably explains the nastiness of some of the responses.
You opted in when you put up the publicly accessable website. If all search engines had to be opt in, nobody could find anything on the web, and it would use a lot of its utility. Your assumed to want them crawling becuase the vast majority of people do, they want their site to be found. If you don't though, no problem, just use the standards for stopping searches, or password protect the site. No scandal at all, just hysterics.
Showing the low res thumbnail of your image isn't violating your copyright either. The only legitimate claim you have is the amount of time it took to remove something from the cache.
The "thieves" accusation is even more ridiculous. If you put something up on the web people can see for free, you can't complain. There are options if you want to protect it. Google doesn't claim you work as theirs (which would be 'stealing' or at least copyright violation), they help people find you public web site.
If you don't want a public website but made one, whose fault is that? If you are going to run a website and can't be bothered to find out how to do it properly, you can't blame Google.