Google URL Index Hits 1 Trillion

mytrip points out news that Google's index of unique URLs has reached a milestone: one trillion. Google's blog provides some more information, noting, "The first Google index in 1998 already had 26 million pages, and by 2000 the Google index reached the one billion mark. Over the last eight years, we've seen a lot of big numbers about how much content is really out there. To keep up with this volume of information, our systems have come a long way since the first set of web data Google processed to answer queries. Back then, we did everything in batches: one workstation could compute the PageRank graph on 26 million pages in a couple of hours, and that set of pages would be used as Google's index for a fixed period of time. Today, Google downloads the web continuously, collecting updated page information and re-processing the entire web-link graph several times per day."

Comments

Screenshot.

[Shaitan Apistos]

Or it didn't happen.

Re:Screenshot.

[ocularDeathRay]

you are just the best first poster ever. quick, funny, and in no way relevant. Honestly sir, you are the reason why I come to /. thanks, and keep up the good work

Re:Screenshot.

[Anonymous Coward]

Seriously, you just want to get into his/her pants...

Re:Screenshot.

[Anonymous Coward]

"We don't index every one of those trillion pages -- many of them are similar to each other, or represent auto-generated content similar to the calendar example that isn't very useful to searchers. But we're proud to have the most comprehensive index of any search engine, and our goal always has been to index all the world's data."

Thus their index doesn't hit 1 Trillion its simply that they have found "1 trillion (as in 1,000,000,000,000) unique URLs on the web at once!" But they do not index it all.

Re:Screenshot.

[Anonymous Coward]

It's 1,000,000,000,001 now, Google indexes Slashdot very quickly.

Re:Screenshot.

[Anonymous Coward]

1,000,000,000 GET!

Re:Screenshot.

[Shaitan Apistos]

That can be arranged.

Re:Screenshot.

[Anonymous Coward]

1,000,000th spoke claimed on behalf of the beautiful Mercatur

Re:Screenshot.

[halcyon1234]

An in a shocking victory for the trolls, the one trillionth page was a mirror of goatse.

Re:Screenshot.

[sipatha]

At that growth rate we will end up searching the index first before the web.

First Post

[txoof]

And about 600,000,000 of those are, "FrIST P0ST!"

Re:First Post

[Vectronic]

-1 Redundant sure...

But that's sort of along the lines I was/am thinking... take txoof's post alone (or mine, or whoever may reply) there are 3 separate URLS for each Slashdot comment

The Header:
http://search.slashdot.org/comments.pl?sid=626647&cid=24345519

The User:
http://slashdot.org/~txoof

The Score:
http://search.slashdot.org/article.pl?sid=08/07/26/0036245#

How many Slashdot comments are there? It's probably in the high millions, (rhetorical, but I'm interested to know none-the-less) There's like an average of about 250 comments per article, about 25 articles a day, thats about 2 million a year, so 6 million links, then take into consideration stuff like Facebook, which bounces URLs (http://www.facebook.com/link=###/etc) or sites that generate a random identifier every few minutes, making those "unique", gets unexciting quite quickly, Although billions is still fairly high.

Re:First Post

[repvik]

Considering your comment is #24345983, I'd say about 24.3 million comments. Also, I believe there's about 1.5 million different users.

Re:First Post

[Anonymous Coward]

Also, I believe there's about 1.5 million different users.

yeah but if you take out Twitter and all his sock-puppets you'll just be left with 500K unique users...

Re:First Post

[Anonymous Coward]

Let's just hope there's no race to "GET 25M".

Re:First Post

[Anonymous Coward]

According to TFA they strip out (precise) content duplicates. [Anon due to mod points]

Re:First Post

[Anonymous Coward]

-1 Redundant sure...

But that's sort of along the lines I was/am thinking... take txoof's post alone (or mine, or whoever may reply) there are 3 separate URLS for each Slashdot comment

The Header: http://search.slashdot.org/comments.pl?sid=626647&cid=24345519

The User: http://slashdot.org/~txoof

The Score: http://search.slashdot.org/article.pl?sid=08/07/26/0036245#

How many Slashdot comments are there? It's probably in the high millions, (rhetorical, but I'm interested to know none-the-less) There's like an average of about 250 comments per article, about 25 articles a day, thats about 2 million a year, so 6 million links, then take into consideration stuff like Facebook, which bounces URLs (http://www.facebook.com/link=###/etc) or sites that generate a random identifier every few minutes, making those "unique", gets unexciting quite quickly, Although billions is still fairly high.

I guess you didn't read TFA, "In fact, we found even more than 1 trillion individual links, but not all of them lead to unique web pages. Many pages have multiple URLs with exactly the same content or URLs that are auto-generated copies of each other. Even after removing those exact duplicates, we saw a trillion unique URLs"

How long till..

[loconet]

Once the index reaches a google (or rather a googol), the universe explodes.

Re:How long till..

[txoof]

Is that the modern equivalent of the Mayan calendar running out of days?

Re:How long till..

[Anonymous Coward]

Won't happen since the universe's max integer is significantly smaller than a googol or a 'google'.

Re:How long till..

[rho]

I'm more interested in when Google starts returning relevant results to my queries.

I can't believe that I'm the only one that finds Google's quality of service somewhat below par. I guess they're better than randomly stabbing in the dark, and there certainly isn't any alternative that's obviously better, but Google sure isn't everything they think they are.

I know--stop trying to compete with Wikipedia and cut out Experts-Exchange.com from your search results since their pages don't actually return the information you think they do.

Re:How long till..

[onedotzero]

... and cut out Experts-Exchange.com from your search results since their pages don't actually return the information you think they do.

Perhaps you should try scrolling to the bottom of the page... :)

Re:How long till..

[cdrudge]

It took me a while to realize it, but if you scroll clear to the bottom of an expert exchange post, you'll find the comments unhidden and relevant.

Re:How long till..

[Anonymous Coward]

...and cut out Experts-Exchange.com from your search results since their pages don't actually return the information you think they do.

If you block cookies from experts-exchange.com you can actually see the answers on any e-e page - after you visit the first time, it normally sets a cookie to not show results next visit, which is how they get Google to index their pages anyway. With cookies from them blocked, you can then see the answers - you just have to scroll 7/8s of the way down the page past all the fake "Please sign up to see this result" boxes.
(First AC post in years... tee hee. :)

Re:How long till..

[ModernGeek]

I don't know about you, but I really think that potato chips don't have to be a by yourself food.

I mean, imagine, you can have a bag of chips in the middle of the table during a football game get together event

Going to a party and asked to bring something? A bag of potato chips does well. It's cheap, and you don't look like a mooch by coming in empty handed.

I really fail to see how your argument in that potato chips are a food for by yourself.

Re:How long till..

[blahplusplus]

"I'm more interested in when Google starts returning relevant results to my queries.

I can't believe that I'm the only one that finds Google's quality of service somewhat below par."

You're not the only one, but for the most part it is better then most other search engines out there. The real problem is spammers and paid advertising, I think spammers have really made search frustrating for a lot of companies. And ad companies pay other people to promote their sites for them (digg, slashdot, etc). I've noticed the increase in spam-vertised websites in search results for a lot of things.

Personally I think the idea of sharding and search being more specific for what you're looking for is needed. I'd like to see a google with 'tags' and a delicious interface, things like educational institutions and universities get lumped into their own search engine space for instance, this would help narrow down what one is looking for, although it would take time and feedback to design something well for other areas. The fact is that search results get diluted as you put more and more stuff online (numbers and geometric scale).

For fun, I've noticed stumble upon and del.ico.us are not bad alternatives when looking for new and interesting sites without having to use search

Re:How long till..

[Anonymous Coward]

The Experts-Exchange site should be de-listed off of Google for the simple reason that they show different content to the crawler, as opposed to if you actually view it.
Google's terms of service prohibit sites from showing different content based on if it's a browser, or the crawler.

Re:How long till..

[Anonymous Coward]

took me a while to figure that out to, someone needs to make a good userscript to so straight to the bottom

Re:How long till..

[hardlyleet]

"I know--stop trying to compete with Wikipedia" Their algorithms are favoured towards Wikipedia...Not very competitive I must add.

Re:How long till..

[Anonymous Coward]

Uhm, no.

Re:How long till..

[Anonymous Coward]

I'm more interested in when Google starts returning relevant results to my queries.

Search isn't just about the engine, it's also how you use it. It's interactive, and a bit of a game of "what makes what I want to find unique". I can almost always find what I want, but it often takes a few tried to refine the query.

I can't believe that I'm the only one that finds Google's quality of service somewhat below par. I guess they're better than randomly stabbing in the dark, and there certainly isn't any alternative that's obviously better, but Google sure isn't everything they think they are.

So, it's the best available, but it's below the average expected performance?

I know--stop trying to compete with Wikipedia and cut out Experts-Exchange.com from your search results since their pages don't actually return the information you think they do.

If you think google is competing with wikipedia, then you don't understand either wikipedia or knol. Members of both have pointed out that they don't compete. about.com is the one that will get sandwiched.

Re:How long till..

[Eddi3]

Actually, If you go to the cached version of those pages, you can see all the answers. You can also just use the Googlebot's user agent via the User Agent Switcher.

Re:How long till..

[Mike89]

If they cut out all the stupid "Find it cheapest here!!" style domains, I'd find it much more useful.

Re:How long till..

[bliip]

Slightly less catastrophic is the result of typing "google" into google: http://uk.youtube.com/watch?v=Fet0SCt7uGg

Re:How long till..

[hairyfeet]

That is why I switched to Yahoo in FF. Google search just seems to be getting crappier of late while Yahoo seems IMHO to be getting better. At least fo me it hits what I'm looking for on the first page a lot more than Google,and I love when i type something like Bioshock I can hit the more tab and get Bioshock demo,cheats,patches,reviews,etc. Whereas you hit the more tab in Google you just get more crap like Google groups and blogger. But of course it will probably be bought out by MSFT and turn into a giant turd like Live Search,so i'm just going to enjoy it while it lasts. And as always this is my 02c,YMMV

Re:How long till..

[kipman725]

googles search is good unless your trying to find something that has its keywords stolen by something more popular and despite putting a hughe list of - terms its still shows in presedance the more popular thing. It also has an annoying tendancy to bring up patent sites so I have to do almost all electronics serches -patent.

Re:How long till..

[peragrin]

you do know you can use google to search specific websites right? So if your looking through a specific domain you can add that to your search query.

The other trick is to add search terms until you have gotten a smaller list of results.

Re:How long till..

[neonmonk]

I was thinking exactly the same thing. There's been many times when experts exchange has been the only result to directly answer my query. But if you don't know to scroll down to the bottom of the page you probably think that you need to pay to get the answers..

Re:How long till..

[MPAB]

Try searching for a given product and the word "review" or something alike. You'll get endless pages of stores with no review whatsoever that must be scrolled away till you find a real technology site that has actually reviewed the product.

Re:How long till..

[Anonymous Coward]

This is one of the reasons I really like Yahoo!'s search assist stuff. It can really be helpful to know what other people are using to narrow it down. There are many times I have one of those mental "Aha!" moments.

Re:How long till..

[Poorcku]

i would rather then type google into google.

Re:How long till..

[osobear]

I know--stop trying to compete with Wikipedia and cut out Experts-Exchange.com from your search results since their pages don't actually return the information you think they do.

Also, asking for the google cache of an Experts-Exchange page will give you the unblocked version.

Re:How long till..

[freddy_dreddy]

I know--stop trying to compete with Wikipedia and cut out Experts-Exchange.com from your search results since their pages don't actually return the information you think they do.

Since we're on the subject and people @ google are reading this: please add a chekbox that excludes Wikipedia results. The "Exclude URL Query Parameters" don't work and neither is the man on that. WP is just noise. Not pink, or brown, or white, but irritating community noise.

Re:How long till..

[WebmasterNeal]

Wow, I never thought to scroll to the bottom of the page, I guess expert-SEXCHANGE lost another battle. - http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_23596292.html

Re:How long till..

[kv9]

Since we're on the subject and people @ google are reading this: please add a chekbox that excludes Wikipedia results.

just add -wikipedia at the end of your query.

Re:How long till..

[anilg]

But if you don't know to scroll down to the bottom of the page you probably think that you need to pay to get the answers.

Yeah.. their business model is kinda built around that.

Re:How long till..

[freddy_dreddy]

Then you exclude all pages which include the word wikipedia (e.g. other sites which mention it and the likes). I want to exclude all the pages from wikipedia.

Thanks for the helping hand, though.

Re:How long till..

[maestroX]

experts-exchange is a waste of time. the internetz wants to be free. -- where's my meds.

Re:How long till..

[wmbetts]

I have an account there and I didn't pay.

Re:How long till..

[Tablizer]

Generally if I don't find much in one search engine, I try a different one. Since they all use different algorithms, they give different results. An algorithm that works well for one kind of search may not work well for another.

Re:How long till..

[Anonymous Coward]

I can't believe that I'm the only one that finds Google's quality of service somewhat below par. I guess they're better than randomly stabbing in the dark, and there certainly isn't any alternative that's obviously better, but Google sure isn't everything they think they are.

I think that the note "there certainly isn't any alternative that's obviously better" is telling. At work, I occasionally have to use Internet Explorer to access certain applications on the intranet, and sometimes (out of habit), use the search box on that browser, which is set to the default search provider for that browser.

The slip becomes apparent when the search results don't return anything useful - and the Google results suddenly seem, well, a whole lot more relevant.

Re:How long till..

[DiarmuidBourke]

I'm more interested in when Google starts returning relevant results to my queries.

I can't believe that I'm the only one that finds Google's quality of service somewhat below par. I guess they're better than randomly stabbing in the dark, and there certainly isn't any alternative that's obviously better, but Google sure isn't everything they think they are.

I find this larger index rather unsettling as I feel my search results are becomming more unrelevant. Mostly due to the following reasoning.

Finding 1 page in a billion page index is relatively easier than finding 1 page in a trillion page index.

Has the relevance of the results kept up with the growing index size? or does the growing index cause more unrelevant pages to appear higher in the search results?

In my opinion the next big advancement that is needed on the web is better auto-catagorisation of content, so your search queries can better be matched to the content in the index. Simply relying on pagerank and text matching can't 'cut it' alone anymore IMO.

Re:How long till..

[anilg]

And that disproves my statement how? Avg Joe who needs a computer problem fixed urgently will pay to see the "hidden" replies.

Re:How long till..

[Anonymous Coward]

Google's success and renown is based on the simple phenomenon that most users don't have a fucking clue what they're looking for, and don't have the patience to look for more than 15 seconds.

PageRank(tm) will probably give them something related AND +popular in the first 20 results.

God help you if you need to include punctuation or stopwords in your search. It's better than it used to be, but still, PageRank wins all.

Re:How long till..

[Anonymous Coward]

someone needs to make a good userscript to correct spelling errors and grammar issues in /. posts.

Re:How long till..

[fyoder]

If you block cookies from experts-exchange.com you can actually see the answers on any e-e page

I don't know if Americans would be allowed to do that. Sounds like a potential violation of the DMCA.

Re:How long till..

[kv9]

Then you exclude all pages which include the word wikipedia (e.g. other sites which mention it and the likes). I want to exclude all the pages from wikipedia.

-inurl:wikipedia ?

Re:How long till..

[Anonymous Coward]

Have you ever tried "QUERY_HERE site:edu"? It does a very good job of returning educational institutions and universities.

Re:How long till..

[RealGrouchy]

No, it's from Arthur C Clarke's "The Nine Trillion URLs of God"

- RG>

Re:How long till..

[timhillu03]

Google's cached version of the page will give you the same result.

Re:How long till..

[Braino420]

...things like educational institutions and universities get lumped into their own search engine space for instance, this would help narrow down what one is looking for, although it would take time and feedback to design something well for other areas.

There is google scholar. That and the source code one are the only that I've used, but there could be others.

Re:How long till..

[Anonymous Coward]

You can achieve the same by searching for 'google' in google. its a bit like eating your own mouth.

Re:How long till..

[Anonymous Coward]

Contribute to search.wikia.com?

Re:How long till..

[ichigo 2.0]

That only happens when Google is the referer. See for example this question which is the first result for this google query. When visiting the link from slashdot the answers are hidden.

Re:How long till..

[joelwyland]

I would mod you way the hell up if I had any points...

Re:How long till..

[Anonymous Coward]

cunt

Re:How long till..

[Anonymous Coward]

or -site:wikipedia.org

Re:How long till..

[amRadioHed]

Serach results with wikipedia results. Search results without wikipedia results. No check boxes required.

Re:How long till..

[cdrudge]

I did not know that. I never go to ExpertSexchange unless it's a google search result.

Wow! Five of those are mine! Mine I tells you!

[Anonymous Coward]

Wow! Five of those are mine! Mine I tells you!

I'm rich! I'm famous!

Don't you fuck with me, man! I am in Google! and I will rip you new ones!

Amazing

[SoupIsGoodFood_42]

As someone who is partially engineering/analytically minded (but not a great programmer) it amazes me how Google has manged to index so much data, yet at the same time, serve up results in a fraction of a second to so many people.

Re:Amazing

[timmarhy]

i wish they would work on weeding out the crap. anything you google now is infested with cheesy search sites that list other websites and try plaster you with ads. they contribute zero to the web.

Re:Amazing

[Freaky Spook]

I couldn't agree more.

Many of the clients I support are constantly asking me "Is there a program that does this? or Can you find me a program to do this" etc etc.

I used to be able to just use google to help me get started but these days the top level searches are all those bloody link farms peddling "free" software, even when typing in the word review you come up with link farms that offer no reviews.

Re:Amazing

[arotenbe]

Many of the clients I support are constantly asking me "Is there a program that does this? or Can you find me a program to do this" etc etc.

I used to be able to just use google to help me get started but these days the top level searches are all those bloody link farms peddling "free" software

Have you tried SourceForge? That's what it's there for, you know.

Re:Amazing

[SoupIsGoodFood_42]

Yeah, that's a problem. I'm sure they'll work it out. I don't find it to be a problem most of the time though, just on certain searches in certain places. They have a real spam problem if you search for info on pharmaceuticals in their groups search last time I checked (about a month ago). The problem wasn't the Usenet groups, but their own special groups, and the worse thing is you can't filter out their groups and just search Usenet ones.

I tried to contact them about it and discovered that they could also really do with making their site more accessible to general enquirers and feedback -- Apple make it easy to give feedback. Yeah, they probably ignore most of the stuff sent through there, but at least you can feel as if there is some possibility of them knowing about a certain issues, unlike when you can't find a way to seed feedback and just get end up with sales and PR contacts (who probably delete anything not related to their department) after 10 mins of browsing -- not good for customer relations.

Re:Amazing

[ILuvRamen]

They rub the servers with cheetah blood. Anyway, the real magic is how is their index that big when there's not that many websites? I thought I just saw an estimate like a half a year ago saying that there's about 4 billion or 10 billion or whatever domain names registered and they estimate xxx number of total web pages and it wasn't even close to 1 trillion. Did all those stupid web 2.0 pages screw with it that much?

Re:Amazing

[SoupIsGoodFood_42]

I imagine that certain sites, such as sites the size of Slashdot (in terms of dynamically generated pages), make a difference. After all, the index talks in pages, not domains. I bet there's also a lot of junk and redundancy in there, but still, it's quite an achievement to be able to deal with that much data.

Re:Amazing

[gingerTabs]

Anyway, the real magic is how is their index that big when there's not that many websites?

If you look at this page in slashdot, and find someone who's replied more than once, then you'll see more than one link to their profile - each link goes to the same page so more links than content - even in this simple example

Re:Amazing

[cammoblammo]

I imagine that certain sites, such as sites the size of Slashdot (in terms of dynamically generated pages), make a difference. After all, the index talks in pages, not domains. I bet there's also a lot of junk and redundancy in there, but still, it's quite an achievement to be able to deal with that much data.

Surely you're not saying that Slashdot's full of junk and redundancy and redundancy?

Re:Amazing

[KillerCow]

As someone who is partially engineering/analytically minded (but not a great programmer) it amazes me how Google has manged to index so much data, yet at the same time, serve up results in a fraction of a second to so many people.

See "map reduce"

Re:Amazing

[tuomoks]

Thanks - I was going to reply but you got there first. Map reduce and big table, etc - old technologies found again. A must when you have to match arbitrary queries - SQL (actually relational databases but in many minds same?) just doesn't do it, something what was know, forgotten and slowly acknowledged again.

Re:Amazing

[Anonymous Coward]

i wish they would work on weeding out the crap.

There are a *lot* of people at Google working on that problem. Please understand that it is really difficult to keep up with new attacks when your site is #1, because many people out there are aiming directly for it. No matter how many work on ranking and relevance inside the company, there will always be 10x-100x that number of people outside who are working on the shady side of SEO, spamming, etc. It's a never-ending battle, much like spam email. We're trying.

anything you google now is infested with cheesy search sites that list other websites and try plaster you with ads. they contribute zero to the web.

We're working on that both from the search side (ranking) and the ads side (not letting those sites run using Google ads).

If you want to help, you have many options:

1. Join Google. If you get in, and say you want to work improving search results or stopping spammy ads sites, you'll have no trouble joining an appropriate group.
2. If you've got a better approach, start a company. If it is a better approach, sooner or later you'll get noticed, and probably bought by a search company. Good ideas are worth a lot in a bid industry.
3. Read the available research in the area, do your own experiments, and contribute to the pool of knowledge. Could easily lead to #1 or #2.
4. If a company you recognize engages in shady practices, tell them you'll take your business elsewhere unless they clean up. If you're a blogger, remind them of that fact. If the company you work for starts to do shady things, point out that you don't think its ok.

Re:Amazing

[Anonymous Coward]

You are better off posting here, on blogoscoped, or blogging about it yourself (We're human after all). Any sort of normal feedback system doesn't work when there are individuals out there who don't *want* your feedback to make it through, and they can spend money to protect their "business" that you'd like to complain about.

Can you give some specific example queries?

Re:Amazing

[hairyfeet]

Just go to Freewareworld Team. I work in PC repair and am always needing a piece of freeware to do a specific task. Freewareworld Team has an excellent search engine. Simply type what you need the program to do and they find you a freeware program that does it. And I have never run into spyware or trialware there. So enjoy and I hope this helps.

Re:Amazing

[cheater512]

If you design your queries well enough, then you dont see any of the crap.

Re:Amazing

[neonmonk]

Yes, redundancy.

Re:Amazing

[SoupIsGoodFood_42]

Groups search for diazepam Not one legit result on the whole page, and none of them from traditional Usenet. Doing an advanced search and restricting to alt.drugs or some specific group does work better, but you need to know what groups are useful in the first place for that to work.

Re:Amazing

[freddy_dreddy]

The answer you get from google depends 100% on how you ask the question.
Bear in mind that offering this amount of data is prone to "granny proofing". They have to index every possible combination of words, in any language (including typo-correction) and provide it to people who don't know how to set the clock on their Âwave oven.

And no, I'm not implying that you're technophobic or incompetent.

Re:Amazing

[maestroX]

This is a me too posting.

I'd like a site with reviews, good articles and news specifically aimed at the youngster coming out of the basement.

Can you help me?

Re:Amazing

[Anonymous Coward]

You can filter out the biggest offenders with the CustomizeGoogle Firefox extension. It has some other nice functions as well: anonymization, no text ads, crosslinks to other engines, ...

I wonder why Google is so soft on SEO. Can't be so hard to filter out spam results that appear in too many searches (mister-wong.*!) Maybe Google profits more (ad revenue) if they don't weed out those sites, and they get away with it because of their near-monopoly.

Odd

[Tubal-Cain]

So unless there is a screenshot showing the 1,000,000,000,000 site count, Google's index didn't reach that milestone? Even if it now shows 1,000,000,000,001?

Re:Odd

[Anonymous Coward]

So unless there is a screenshot showing the 1,000,000,000,000 site count, Google's index didn't reach that milestone? Even if it now shows 1,000,000,000,001?

The 1,000,000,000,000th page had only one word on it:

"woosh"

Re:Odd

[Tubal-Cain]

I seriously debated whether I should indicate that a woosh was unnecessary.

Re:Odd

[Anonymous Coward]

*Gasp, Tubal-Cain is the freemason password. Are you a mason?

Re:Odd

[physburn]

"woosh" Funny? It went straight over my head.

Re:Odd

[Tubal-Cain]

Tubal-Cain is the freemason password

*Gasp* Am I infringing on their IP?
All the good Greek & Norse mythology names were taken, and so I moved on to biblical names. Tubal-Cain was the first one I came across that appealed to me.

Genesis 4:22:

Zillah also had a son, Tubal-Cain, who forged a kinds of tools out of bronze and iron.

The I like the thought of forging tools virtually (programming).

Wow, that's a lot of porn.

[Anonymous Coward]

Seriously, since the web is something like 42% porn. (Yes, that is the ultimate answer.) So that's on average, 60-70 pages of each person in the world naked.

Re:Wow, that's a lot of porn.

[sweet_petunias_full_]

"the web is something like 42% porn"

That probably stopped being the case after namespace speculators started buying up expired domains in large numbers just to put up a mildly useless index on *each* and *every* site to collect ad revenue or marketing statistics off of unwary visitors. I would also include typosquatters in that category, and maybe someone else can name a few other examples of utter namespace hogging uselessness.

Whatever it is, you can rest assured that it's mostly repetitive trash... no need to stand in awe of it.

Re:Wow, that's a lot of porn.

[Anonymous Coward]

And now, in the "advanced" options of Google Image Search, you can search for only pictures with faces in them.

Now, what I've always wondered, is when you do a "safe search," you get "safe" results, and when you turn off "safe search," you get all results.
But is there any way to return the "unsafe" results, without the "safe" ones mixed in?

Re:Wow, that's a lot of porn.

[brendan.hill]

What's sad is that you took the time to work that out.

What's even sadder is that I took the time to verify it.

And you're right. It's about 70 pages per person.

1 trillion url's

[jollyreaper]

How many of those are automatically generated rank-spoofers, 80%?

My favorite spoof pages were the ones that randomly substituted search terms into porno stories.

"Yes!" she screamed as he thrust his SAMSUNG CD PLAYER deep into her. "I want you balls-deep in my CHEAP HARD DRIVES!" The smell of DISCOUNT SOFTWARE filled the room.

Re:1 trillion url's

[Anonymous Coward]

"Yes!" she screamed as he thrust his SAMSUNG CD PLAYER deep into her.

Given the nature of some sites, that may not be a spoof. Besides spoofing never works.





horny, girl, girls, wet, juicy, breasts, bouncy, elf porn, porn, pr0n, sex, sexy, CD insertion, warcraft sex, CD Player insertion, double insertion, slashdot, natalie portman, adult toy black orgy girls smoking milf seeker amateur facial black shemale hairy twins gangbang black in blondes mommy got boobs orgy teen mother naked college coeds nurse bondage hentai busty asian maturebig black girls fetish sex lesbian sex taboo big black tits filthy sluts lesbian spanking taboo charming mother big brother first anal lesbian strap on teachers big girls in bondage first timers lesbian teens teen big natural tits first time anal lesbian threesome teen facial foot fetish lesbian videos teen lesbian at school fucking machines massage orgy gangbang mature latin thick black ass round ass anal mature lesbian tight teens round asses sex mature sex medical fetish water big wet butt girls gone wild messy white cotton panties next door

Re:1 trillion url's

[Anonymous Coward]

That's amazing! I've got the same combination on my luggage!

Re:1 trillion url's

[Anonymous Coward]

Thank you.

Re:1 trillion url's

[Anonymous Coward]

link?

(seriously - I can't find it)

My CAPTCHA was delight :)

Re:Wow! Five of those are mine! Mine I tells you!

[Tubal-Cain]

Wow! Five of those are mine! Mine I tells you!

Only if Google has indexed your sites.

No concern for the foreign readers?

[Anonymous Coward]

Trillion can mean 1E+12 or 1E+18 depending on which country you are in.

Re:No concern for the foreign readers?

[Shaitan Apistos]

Trillion can mean 1E+12 or 1E+18 depending on which country you are in.

I'm inclined to go with whatever Japan & China say.

Re:No concern for the foreign readers?

[kclittle]

Google is headquartered in Mountain View, CA -- I know, 'cause I googled it. Now, California is rather inclined to think of itself as it own country (some would say, universe), but it is indeed part of the United States of America (again, I checked with Google). And in the US, "trillion" == 1E12 (again, Google).

Re:No concern for the foreign readers?

[Samantha Wright]

The UK is the only English-speaking country where a significant portion of the population still uses the long scale system. I don't htink it's all that much of a concern.

Re:No concern for the foreign readers?

[Smauler]

No one in the UK uses the long scale system really. For example, traditional UK billions are _never_ used in governmental budgets, and no one points out that the "American" billion is being used. A billion is just 1E9 here, like just about everywhere else.

I guess some older people may be confused (what's new ;)), but I'll wager a large proportion of the younger UK population don't even know what a traditional English billion is. I'm 30, and I've never used 1E12 as a billion, or even been taught it could be.

Re:No concern for the foreign readers?

[freddy_dreddy]

I have English as a third language, people that deal with numbers are generally aware of these double meanings.
E.g. in Dutch a "biljoen" = 10E12 and the English "billion" is "miljard" in Dutch while "triljoen" is 10E18 ... The linguistic term for this is "False friend" in case you're interested.

And I rest in peace..

[consonant]

..knowing that the vast amounts of porn just keep getting vaster. And more searchable. Amen. *sheds a tear or two*

Hi, tech support?

[Anonymous Coward]

End-user: "I want to download the internet. Will I need a bigger hard drive?"
Google: "Yes"
End-user: "I didn't even tell you how big my hard drive is!"

Some numbers

[Reality Master 101]

Counts of words:

the: 18.3 billion pages
a: 23.9B
0: 12.7B
1: 25.4B
in: 17.1B
I: 10.2B

I know these numbers aren't exact, but you'd think one of them would be over 100B if Google is really indexing a trillion pages. What's on them? Anyone find any keywords that produce more?

Re:Some numbers

[ooburns]

The only one I've managed to find is "www", which produces 27 billion. Still a long way from a trillion...

Re:Some numbers

[Shaitan Apistos]

My hobby:

Getting the fewest possible google results above 0 with a quoted string.

"interspecies gangbang": 6
"hot topic meets disney world": 2
"died in a blogging accident": 15,300
"can boys make babies": 4
"why does it hurt when I read": 1

Re:Some numbers

[Reality Master 101]

Hmm. This probably means something statistically, but I'm not sure what... I started adding digits to a number until I hit one result. I had to get to nine digits:

123512553
215323703
684354537

I also found a few with 0 to 3 results. Interestingly, I couldn't find any eight digit numbers that scored zero hits.

Re:Some numbers

[chillax137]

The only way to save your accomplishments is to not post them on the internet

Re:Some numbers

[txoof]

I couldn't find any eight digit numbers that scored zero hits.

That's really interesting. Perhaps 8 digit numbers are common for serial numbers and dates. Today's date is 20080723. I can't even begin to think of all the logs and file names that I've generated that contain a similar string.

Anyone else have any ideas why 8 digit numbers are so common?

Re:Some numbers

[arotenbe]

Indeed. Google already has two hits for the phrase "why does it hurt when I read": the actual (and rather disturbing, but amusing) page the quote was found on, and this page. Scary, isn't it?

Re:Some numbers

[miraboo]

My hobby:

Getting the fewest possible google results above 0 with a quoted string.

"interspecies gangbang": 6
"hot topic meets disney world": 2
"died in a blogging accident": 15,300
"can boys make babies": 4
"why does it hurt when I read": 1

My Hobby

Attributing my sources: http://xkcd.com/369/

Re:Some numbers

[bigplrbear]

maybe a majority of Googles indexed websites are blank

Re:Some numbers

[SevenSpirits]

That's odd. I get:

"interspecies gangbang": 7
"hot topic meets disney world": 3
"died in a blogging accident": 15,400
"can boys make babies": 5
"why does it hurt when I read": 2

Huh.

Re:Some numbers

[Anonymous Coward]

People speak different languages in different parts of the world, and some of them have webpages? Who'd have thought!

Re:Some numbers

[antifoidulus]

Or maybe a lot of those pages are in non-Latin script. According to the Chinese government there are more Chinese online than Americans, and they don't use the latin alphabet....

Re:Some numbers

[Anonymous Coward]

"Getting the fewest possible google results above 0 with a quoted string"

Results 1 - 1 of 1 for "Getting the fewest possible google results above 0 with a quoted string". (0.09 seconds)

Re:Some numbers

[WithLove]

I just want to know who has time to Google all 1E8 eight digit numbers....

Re:Some numbers

[crenshawsgc]

Chinese people overwhelmingly use arabic numerals in everyday use, just like everywhere else in the world...

Re:Some numbers

[Anonymous Coward]

I know these numbers aren't exact, but you'd think one of them would be over 100B if Google is really indexing a trillion pages. What's on them?

Foreign languages? (I admit it doesn't completely explain the lack of 0s or 1s, though.)

Re:Some numbers

[Shaitan Apistos]

My Hobby

Attributing my sources: http://xkcd.com/369/

In , my humble opinion my usage of "My Hobby" was sufficient attribution, all by itself.

Re:Some numbers

[Anonymous Coward]

but sense that are from china, it is far less likely that they will be creating content, lacking freedom of speech and all that.

Re:Some numbers

[Aelix]

Bump all those strings up by one, google has it already lol.

Re:Some numbers

[dword]

They just increased by 1, all of them... This page is in Google's index :)

Re:Some numbers

[Ihmhi]

You mean Googlewhacking, except not nearly as hard?

Re:Some numbers

[SnowZero]

You can list all of them with less than a gigabyte: 10^8 * (8+1) ~= 858 MB
The web is pretty big, so all of them are bound to happen *somewhere*.

Plus, I just registered all8digits.net

Re:Some numbers

[Anonymous Coward]

Anyone find any keywords that produce more?

mmmh... any chinese "word" maybe?
a.

ps: no, i can't test it for you.

Re:Some numbers

[spacefrog]

Much like Fight Club, the number one rule of your particular hobby is not to talk about your results hence you taint them.

Kind of like counterfeiting currency or collecting teenybopper porn, it sucks to have a cool hobby you can't brag about.

Re:Some numbers

[Anonymous Coward]

"why does it hurt when I read": 1

please update: 2 hits, since you posted it to /.
... well, there will be 3 of them in a few seconds...

a.

Re:Some numbers

[Anonymous Coward]

+1 now that you've posted them on Slashdot...

Re:Some numbers

[bakuun]

Not all pages are in english.

Re:Some numbers

[miraboo]

I do not object at all to your use of 'My Hobby' meme; I considered this an appropriate allusion. But it is not sufficient attribution for your use of 'died in a blogging accident'. That usage was in my view, an appropriation. You intermingled the work of another with your own such that the it would be taken as your own.

It is unreasonable to expect a reader to share the encyclopaedic knowledge of xkcd that you and I possess. The reasonable reader would have taken 'died in a blogging accident' to be the product of your intellect just as they took 'why does it hurt when I read' to be the product of your intellect.

I readily admit that I was somewhat immature in the style in which I drew slashdot's attention to your appropration but your response was no less immature.

Perhaps I am wrong, I don't think I am; but if I am I would like to hear why. Please do show me the courtesy of a reasonable response.

Re:Some numbers

[Anonymous Coward]

UPDATE

"why does it hurt when I read" 2

Slashdot hurts me deeper than CHEAP HARD DRIVES.

Re:Some numbers

[MagdJTK]

pfft. So they both type strings into google? WOW HE SO MUST HAVE STOLEN IT!!!!

By that logic, xkcd stole it from Dave Gorman.

Re:Some numbers

[miraboo]

pfft. So they both type strings into google? WOW HE SO MUST HAVE STOLEN IT!!!!

By that logic, xkcd stole it from Dave Gorman.

My point was not that s/he stole the idea of typing strings into google, my point was that s/he appropriated the phrase 'died in a blogging accident'. If you had actually read the complete comic you might have realised this.

Re:Some numbers

[edmac3]

Oh boo hoo! Are you the author of the comic or something? There's something called collective knowledge. Everybody intermingles other people's work into their own. Having to attribute a phrase like that would just be tiresome and severely mitigate it's fluid usage and appeal. Do people credit Dawkins whenever they use the word 'meme'?

Re:Some numbers

[WK2]

I tried the following: the OR a OR 0 OR 1 OR in OR i

It says "Results 1 - 10 of about 0". Must be a wrap-around bug.

Re:Some numbers

[kv9]

Kind of like counterfeiting currency or collecting teenybopper porn, it sucks to have a cool hobby you can't brag about.

I brag about that all the time.

Re:Some numbers

[Anonymous Coward]

"why am i out of cereal": 0

Re:Some numbers

[freddy_dreddy]

these numbers are estimates that can be off-range a few decades. E.g. "synandrospadix vermitoxicus" yields 159 results, but if you click on the 10th page it drops to 81.

Re:Some numbers

[ion.simon.c]

My contribution?

http://www.google.com/search?aq=f&num=100&complete=1&hl=en&safe=off&q=%22sharpshooter%22+%22kill+me+now%22+%22stop+crying%22+%22stay+alive%22+%22stop+crying%22+%22stay+alive%22&btnG=Search

*grins* Don't be an amateur. Dump the URL for the search, instead of the words *in* the search.

Re:Some numbers

[Anonymous Coward]

com: 25.79B

Re:Some numbers

[Reality Master 101]

I thought maybe different languages might explain things, but I'd say English is by far the most popular languages. I might even buy English is not the majority of pages anymore (though, I'm skeptical), but only 1 in 25 pages is English?

Re:Some numbers

[Shaitan Apistos]

died in a blogging accident was there solely as a pop (nerd?) culture reference to xkcd. The post topic is getting the fewest google results above 0 with a quoted string, a string with 15k+ results is completely irrelevant to the post as anything other than as a allusion for the benefit of people who are already familiar with the comic.

The defense rests, you are sentenced to finding every instance of "the cake is a lie" on slashdot and demanding that they attribute Portal, and when you're done with that you can move on to asking people to attribute 'all your X are belong to us' to Zero Wing.

I bid you good day.

Re:Some numbers

[Anonymous Coward]

www: 25.75B

Re:Some numbers

[miraboo]

Hmm. Perhaps I was wrong. You have done your duty, sir: http://xkcd.com/386/ .

Re:Some numbers

[RealGrouchy]

TFA (the second one) says that not all pages will be indexed. That, and many pages are in other languages/scripts, as pointed out by other replies to your post.

Of course, if not all the pages are indexed, then the story kinda fizzles out.

- RG>

Re:Some numbers

[miraboo]

While I think this discussion is long past its used by date [and I have conceeded that the original usage may have been reasonable]. Common knowledge is not attributed not because it would 'severely mitigate it's fluid usage and appeal' but because the reasonable reader would be well aware that it was not the writer's idea or formulation. This then of course renders your example unhelpful - did you think I coined the word 'meme'.

The question is then was the phrase common knowledge and perhaps, in this community, it was.

And incidentally, you might want to know that the preposition 'with' usually follows 'intermingle' not 'into'.

Re:Some numbers

[edmac3]

Okay, you're right.

Re:Some numbers

[old and new again]

"interspecies gangbang" 7 results now, your post being the first result now

Re:Some numbers

[old and new again]

Results 1 - 10 of about 195 for "why does it hurt when I read". (0.12 seconds)

Re:Some numbers

[tehcyder]

"died in a blogging accident": 15,300

God, that cheered me up on a Monday morning.

What's going on with the founders' studies?

[bogaboga]

This might be off-topic but I wonder what's going on with Sergey Brin and Larry Page's [PhD] education? Just wondering...did they give up?

Re:What's going on with the founders' studies?

[Anonymous Coward]

you are fucking kidding right?

Re:What's going on with the founders' studies?

[Anonymous Coward]

You do realise that a lot of money doesn't make you innovative, and that the best way for Google to improve their index would be to not rest on their laurels, right? Indexing algorithms are such that "indexed one trillion" over "indexed one billion" is not an impressive conceptual or technical feat, and this news is mere marketing wowspeak (misleading marketing wowspeak, considering the number of fluff pages that exist purely to generate AdWord revenue - thanks for that, Google).

N.B. It doesn't matter how many PhDs you employ as glorified code monkeys underneath you, because the direction of the company is your decision, and you need to stay enlightened to make an enlightened decision. See Microsoft and the replacement of Gates with Ballmer.

Re:What's going on with the founders' studies?

[James Youngman]

I try never to comment on Google stories but this comment shines forth as being so glaringly dismissive that I can't hold back. If you think that going from 1e9 to 1e12 known URLs is not "an impressive conceptual or technical feat" you should review some course materials on asymptotic complexity and think about how you would index the entire web using only algorithms that don't simply become infeasible at 1e12. In fact, forget about the indexing complexity per se, have a think about how you would select which subset of the 1e12 URLs to actually index. That's a hard problem in itself.

Re:What's going on with the founders' studies?

[cheater512]

I really cannot see how a PhD helps you to innovate.

Re:What's going on with the founders' studies?

[Anonymous Coward]

what do you think. when you are a billionaire with world status what does a phd do for you anymore..

Re:What's going on with the founders' studies?

[Jeff DeMaagd]

If I turned billionaire before I finished schooling, I don't think I would finish unless I got bored enough to do so. I don't think it's necessary for them to finish either, because they can hire plenty of PhDs with the knowledge that they would have gotten if they finished.

Running a business doesn't require that you know everything needed to run the business, only that you know how to get people with the skills that you need.

Re:What's going on with the founders' studies?

[inKubus]

Well, you're crawling, so you're already getting a list of what people link TO; all you have to do is index that and spit out the reverse: Who links to page x? The interesting thing about Google is the way they made a self-aware machine that basically asks for more hardware and puts it to use. All they need to do is feed it webpages in the form of network connections (and a lot of them are fed to it by webmasters using the webmaster tools). Adwords is the same way. They knew that they only had to get slightly better results than everyone else to make money in advertising online. Online advertising still SUCKS but at least they have a huge index to match ads to.

Every other one of their business models is a variation on either: self-aware computers which expand themselves (a pretty simple idea, really), indexing, and ads.

I don't want to undermine them at all, I think it's a great thing they did, but the thing is, what else will they do? I mean, they hacked the web. Great! But will they turn professional or will they take the money and run the the shit hits the fan? This is all still so new (only 10 years old), there's a lot of rise and fall yet. I'm sure they have more up their sleeves, and hiring the smartest talent was a good move. Now they just need some managers...

So, how's it compared

[Korbeau]

to the increment of garbage in American landfill since 1998? ;)

Google says hi

[Anonymous Coward]

http://www.365tomorrows.com/09/12/the-nine-billion-names-of-god/

I'd be more impressed if...

[Anonymous Coward]

the first couple hundred results for every search didn't consist of useless price comparison sites, a Target ad, youtube videos of dubious quality, link farms and experts exchange crap.

No wonder I cringe every time I need to start doing research on the web.

No, it didn't.

[aiken_d]

They have identified that there are 1T pages out there, somewhere. They have indexed 40 billion pages. Read the entire Google post. It says it right there.

Bad on Google for the misleading post. Bad on the submitter for not reading the misleading post. Bad on Slashdot for further descending into mindless repetition of mindless submissions of mindless PR announcements.

Re:No, it didn't.

[Anonymous Coward]

How is the Google post misleading? It never makes the claim that its index is 1 trillion documents. One might argue that it is meaningless, but not misleading.

Re:No, it didn't.

[Johnny Mnemonic]

They have indexed 40 billion pages. Read the entire Google post. It says it right there.

I didn't see that information in either post, only that their index is somewhat less than the full 1T unique URLs. The full size of the Google index is probably confidential, proprietary info. Their competitors could use it as a benchmark for assessing how their service compares, so I'm not surprised that it's not there--or if it was there that it was removed.

Bandwidth

[Anonymous Coward]

I wonder how much bandwidth the daily/continuous Google index process takes.

Re:Bandwidth

[WeblionX]

All the search engines kept causing database errors on my forums until I blocked most of the forum via robots.txt. Then it was only Google causing problems. Of course, I finally switched hosts and it doesn't seem to be causing problems so far.

So, in short, too much.

Re:Bandwidth

[Anonymous Coward]

Too much. On some of the websites I've run, as much as 20% of all traffic was from Google. I'm seriously considering disallowing Googlebot on pages without ads.

Re:Wow! Five of those are mine! Mine I tells you!

[Anonymous Coward]

I can tell you without reservation that more than half those "unique URLs" are dead pages. Over the course of my 12+ years, which pre-dates Google, 99% of all my "uniqie URLs" are dead and buried. I've done 1000s of pages, the bulk being online manuals. I know, I know. This is an oddity among readers here, but manuals are essential parts of software products, even those that are only around for a few years. It's not a product if it doesn't have a real, honest-to-goodness manual.

And most of them are webspam

[Animats]

But how many of those trillion pages have unique, useful content? E-mail is over 95% spam, and the web is getting there.

There were about 153 million registered domains at the beginning of the year. The ones from the spam-friendly registrars are mostly junk. Tim Bernars-Lee said in 2006 that web junk was becoming a major problem, and it's become worse since then.

If you throw out all the anonymous but commercial domains (we call them "bottom-feeders"), as we do with SiteTruth, the Web looks a lot better. Search engines are getting stricter about this. You don't see that many "landing pages" in Google any more. Bad news for companies like Marchex, the publicly traded web spammer that cranks out all those junk "What you need, when you need it" sites.

"The mass trials are going well. There will be fewer Russians, but better ones." - Greta Garbo in Ninotchka.

A trillion URLs...

[Chester K]

A trillion URLs, and still no sign of clownpenis.fart in the index anywhere!

At this rate it really will be the last one to go.

Re:A trillion URLs...

[Anonymous Coward]

Don't worry, there will be there now.

Re:A trillion URLs...

[repvik]

Huh? I just *had* to google that one...
Results 1 - 10 of about 8,580 for clownpenis.fart

Try "Live" search

[symbolset]

And you'll be back faster than a Google search result. Weeding out the crap?

Just for a sample, try this one: getfirefox. If the first link on that search goes to a Mozilla mirror you will win one Internet. Try Linux. Hey, this is fun. Spoiler: the first link there is always "www.Microsoft.com/Windows : Special Offers from Windows Vista® w/ the Purchase of Select Laptops." The first time I tried this I was looking for Open Office and wound up misdirected to a members only site where you had to register to download a probably spyware infested Open Office and signing up for unlimited pharma spam. The scary part is that the text of the link misled me to believe I was headed for "OpenOffice.org". Try it and see. Let's find more horrifically inappropriate ad placements and query results, shall we? I'll bet you could come up with a really funny one.

Note: Please don't go to any of the sites linked to those search results through live.com. Bad things might happen to your Windows box and there's nothing there of interest for your powerbook.

Yeah, that's a good search result ad, don't you think? No wonder Google is becoming a verb.

Re:Try "Live" search

[Anonymous Coward]

The first result for 'getfirefox' is http://www.getfirefox.net/ which seems to be correct to me.
The first result for 'Linux' is http://www.linux.org/ not some Microsoft website.
The first result for 'Open Office' is http://www.openoffice.org/ and not some Microsoft website.

You lie?

Re:Try "Live" search

[symbolset]

The "tell" for this lie is that people will try it. Did you think of that, Mr astroturf man? And what do you think they will find besides proof? You shouldn't submit this one for credit if you want to keep your job.

Re:Try "Live" search

[cammoblammo]

You lie?

Perhaps, perhaps not. I've experienced odd things like this inthe past.

A few years ago I tried to get to the FSF site by typing the URL into the address bar in Firefox or Phoenix or whatever it was then. For some reason it took me to the Microsoft site.

It turned out I'd mistyped the URL. In that situation the browser automatically does a Google 'I feel lucky' search on the term you type, apparently hoping to get to where you hoped to go.

In my case, I mistyped the '://' after the protocol signifier. Consequently, the search term used was 'http'. For some reason, the number one search for 'http' was Microsoft. I tried it in other search engines and they took me to more obvious places, like w3c and so on. Google went to Microsoft.

This odd behaviour was only corrected recently. It did concern me, though, that many people would mistype the first part of a URL. How many, I wonder, have ended up at MS?

Re:Try "Live" search

[pagaboy]

Turns out Live.com's market share for today has tripled due to Slashdot users clicking on the above links...

Re:Try "Live" search

[Anonymous Coward]

Not that I advocate or even like Microsoft's Live Search, but you are an idiot.

Your first "sample" returns the getfirefox.net site as the very first search result. Similarly your second "sample" returns linux.org as the very first search result. Guess what your third "sample" returns as the very first result? If you guessed anything but openoffice.org, you lose.

What you are confusing with search results are actually ads, just like every other search engine has. I don't even think you really are confusing the ads with the results, this is just your thinly disguised means to MS bash.

Re:Try "Live" search

[LinuxFreakus]

Are you one of those people who can't tell the difference between sponsored links and search results?

Looks like there is nothing wrong with those searches at all. You just didn't realize those sections which are highlighted in a different background color and a very visible sponsored link disclaimer are actually not search results.

Re:Try "Live" search

[totally bogus dude]

It may be location-specific, but I don't get any links to Microsoft's site as the first results.

The results for the UK seem reasonable, although the first one wasn't what you'd be expecting (e.g. the first result for "open office" was http://www.vnunet.com/vnunet/downloads/2128963/openoffice). I changed my browser language preference to English/Australia so I'd get more appropriate results (and deleted the live.com cookies), and I got fairly different results. The first few links for "linux" were to linux.org.au, with linux.org the second last result on the first page.

Interestingly, the search for open office with my language preference set to UK English didn't give me any ads, but with it set to AU English I do get some pretty lame "sponsored links":

Open Office 2008 Download - OpenOffice.2008-Version.com
Latest Version of Open Office Suite. Fast Download - 100% Guaranteed.

OPEN OFFICE 2008 WORD - computers.oztion.com.au
Computer software at bargain prices.

Microsoft Open Office - www.gumtree.com.au
Find amazing bargains for computer software. Browse 100's of ads.

Okay... so what about the US? Interestingly, just setting my language preference to EN-US doesn't work; it decides I'd prefer Australian results. Maybe EN-US is so common Microsoft ignore it and do geo detection instead?

Fortunately I have a server at a US colo facility, so using that as my proxy did the trick. If you're from the US, then search.live.com returns one ad for "getfirefox":

Internet Browser - www.Microsoft.com/Windows
Everyday Web Tasks Are Easier w/ Windows Vista. Get Special Offers.

and one ad for "linux":

Linux - www.Microsoft.com/Windows
Special Offers from Windows Vista® w/ the Purchase of Select Laptops.

The ads are pretty clearly marked as being ads (sorry, "sponsored links") so the GP seems to be trying to grind an axe. However, that kind of advertising is usually reserved for 2-bit companies with no real products trying to dupe naive users, so it's a bit surprising to see a company like Microsoft resorting to such cheap tactics.

Interesting that the sponsored link for "firefox" is actually mozilla.com; only "getfirefox" returns Microsoft's ad. A search for "getfirefox.com" has a sponsored link for another site which appears to be trying to make a few bucks by sending people to Google's download for Firefox.

Re:Try "Live" search

[wmbetts]

At first I thought well he's wrong, but maybe they changed the results. Then I looked again. What he's talking about is the sponsored links section at the top of the search results. Either he's confused or just likes to twist the truth.

Re:Try "Live" search

[symbolset]

I know this is slashdot, but if you carefully reread the post you responded to you'll find the last line mentions that these maliciously deceptive links are ads. I'm not trying to mislead anybody. Are you?

Re:Try "Live" search

[wmbetts]

I stand corrected!

Re:Try "Live" search

[RzUpAnmsCwrds]

I don't know what you're trying to prove. For me, the first link for "getfirfox" is the MoFo/Google getfirefox.net. The first link for "Linux is linux.com. And the first link for "Open Office" is OpenOffice.org.

Now if you want to argue that the "sponsored links" are crap, I can't help but agree. But Google has the exact same problem.

Re:Try "Live" search

[symbolset]

The ads are pretty clearly marked as being ads (sorry, "sponsored links") so the GP seems to be trying to grind an axe. However, that kind of advertising is usually reserved for 2-bit companies with no real products trying to dupe naive users, so it's a bit surprising to see a company like Microsoft resorting to such cheap tactics.

I don't know why so many people think that I missed that they're ads. I mentioned it twice. Is all of slashdot caffeine deficient this morning?

-- "ad placements and query results" -- "a good search result ad"

And yeah, taking misleading ad placements on your own trademark to help the advertisers dupe naive users is low even for Microsoft. Doing it with somebody else's trademark is worse, but all search providers have problems with this. My point is that Google fights this nonsense pretty actively. Live.com doesn't even appear to try. The thought of using it to try to find a driver update, repair a broken .dll and select a good antivirus out from amongst the attempts to dupe you into being exploited is like sifting manure for pearls in a minefield even to me. I can't imagine what the prospect must be like to someone who's not in the biz. And there's so much of this spam garbage in their search results that once you move away from the top 1000 terms it takes all day to sort through the irrelevant data to find what you're looking for - which is what the search engine is supposed to be doing for you. That is entirely the point of the search engine after all.

Anyway, that's why when most people need to know something they just Google it.

Re:Try "Live" search

[Anonymous Coward]

live has fairly unbiased results :)

search for IE6 on live, check out the 4th result :
http://search.live.com/results.aspx?q=ie6

Re:Try "Live" search

[totally bogus dude]

Fair enough, I think I didn't notice because your mention of "ads" was so far away (in the text) from where you mentioning the "first link".

I think most of slashdot ignored it because we ignore ads by default. Half the people that tried the search probably didn't even know there were ads, because they automatically ignore the "sponsored links" box. I pretty much ignore all ads unless I'm actually looking to spend money, because it's rare for someone to pay for an ad if they don't expect to sell something by doing so. All three examples - Firefox, Linux and Open Office - were free software, so they definitely fell into the "not expecting to pay anything" category for me.

The rest of the people were commenting on the quality of the actual search results, which you didn't actually make mention of except to imply that they were "horrifically inappropriate". I didn't look any further than the first page for each, but they did seem pretty reasonable.

Re:Try "Live" search

[Taxman415a]

Now I remember why I had never bothered to use their search. I expected tricks like that and there they are. It's funny to note they didn't buy/reserve that misleading "Linux" ad that points to Windows for the two word search term ubuntu linux but they did for the one word ubuntu which has more meanings.

It's also funny to note how much more the interface looks like google's than anything MS has done before. It's so obvious they are admitting they don't know how to innovate anything themselves.

Re:Try "Live" search

[KGIII]

I'm almost embarassed but I was so tired of the results from Google that I switched to the http://search.live.com/ as my homepage. (Yes, I open a new tab for searching and seldom use the brower's search box.) I find that the results that I get there are quite appropriate for my search terms usually though I'm thinking of finding a new search engine because over the past two months their quality has done the opposite of what it had been doing and that is not good. The results *are* getting worse than what they were just a couple of months ago. I'll give them a while longer to get back on track but, for now, I'll actually be looking for a new search engine/home page.

Re:Try "Live" search

[Anonymous Coward]

Maybe you should learn to differentiate between search results and "sponsored sites" that are clearly labeled as such. Are you really surprised that Microsoft is purchasing ad links on their own search engine for keywords leading to competing technologies? If they purchased the same keywords in adsense, do you really think Google's results would be magically different?

Re:Try "Live" search

[symbolset]

C'mon guys. I mentioned that they were ads in my post twice. Try and follow along.

Re:Try "Live" search

[Lotunggim Ginsawat]

You have been proved to be wrong. The proof is that Live.com search results is not the one you have described. For my POV, there is nothing wrong with Live.com results. If you want to lie, at least make sure other people can repeat your result.

Re:Try "Live" search

[Anonymous Coward]

You acknowledged that they were ads, but also said they were misleading search results. They can't be both. Since most ad models are "purchase X keyword for $Y and it will show up as a sponsored result", the search algorithm has little-to-nothing to do with the ad placement. Microsoft has purchased ad keywords for Firefox-related terms as well as Linux-related terms. Are you surprised? I'm guessing they might even do the same with AdSense but there is more competition on AdSense so they

As for the placement of the ads... Google also places ads at the top of search results, but they do it in a yellow box instead of a blue box. Granted, it's not every search (some still show up in the 'classic' right-pane position), but it does happen. I feel the way Google does it is almost more deceptive because users can't condition themselves to automatically discard the links directly at the top of search results since they're not always ads.

How many sites are REAL websites?

[Doug52392]

And not just those stupid "parked by GoDaddy" or "domain farms"? Like those ones where when you Google something, and you get a promising result, so you click on it, and it turns out to be nothing more than a page full of ads.

Re:How many sites are REAL websites?

[lintux]

Possibly that page really *was* useful, but the domain name expired. Lots of spammers on the nets like to snatch expired domain names and "monetize" (sic) them.

Googlewhacking

[symbolset]

It's harder to do now. If you can find two words unquoted that result in one result it's a googlewhack like this one was before Google found "eltiguan parainaugurarme" on this page and made it the second result.

BTW, my first google search was "war" and it returned something equivalent to "Your search term is too common to return a meanignful result. Narrow your search." Today it returns about 974 million results. It was long ago...

Try "Live" search

[symbolset]

You'll be back faster than a Google search result.

Re:Bandwidth... too much

[Doug52392]

On my home Web server, I accidentally left a copy of the PHP manual in a browsable folder, which was linked to the homepage. So when Google indexed my homepage, guess what it also checked for? Every single page the homepage linked to! Including that manual... and damn the PHP manual has a LOT of pages.

So when I got back on the server and pulled up the logs (it was running strangely slow) I found Googlebot accessing page after page after page of the PHP manual. Thousands of pages. Lagging the server and Internet to hell.

What I want to know is in Google somewhere...

[symbolset]

Specifically which page was the trillionth?

Re:What I want to know is in Google somewhere...

[cheater512]

More than likely a spammer's page. :)

Give up? No.

[Anonymous Coward]

They grew up.

SAT reading comprehension for slashdot.

[Anonymous Coward]

According to the googleblog post, which of the following is not true?

A. The initial google had 26 million pages.
B. Google has seen 1 trillion unique urls.
C. Google indexes every one of those trillion pages.
D. The web is infinite.

We all know what they are too

[Anonymous Coward]

Now what isn't said here is that 999,999,998,000 of those pages are porn.

google's search becoming steadily useless.

[blind biker]

I think google.com's search engine achieved its peak usefuleness about 5 years ago. Now, for the most part when I google for a certain electronic component I get some crappy webstore front (and by crappy I mean I can't actually order the component but must "contact by phone" first) or if I search for an electronic device, be it pro or just home electronics, I get those "Read reviews and compare prices"-sites. Which I hate with a passion. WTF google, you have the world's most talented programmers, can't you weed out this crap from your search? At least so it doesn't come up as top hits?

Re:google's search becoming steadily useless.

[Anonymous Coward]

i use the customizegoogle firefix plugin to dump "expertsexchange.com", "about.com" and "gamespot.com". it removes any matches from search results

Re:google's search becoming steadily useless.

[PatrickThomson]

It's almost like google stopped bothering about returning relevant commercial sites in the main search, and farmed it off to some subsidy. I mean, talk about saving money, that's a very froogle thing to do.

Please then allow me to bludgeon you with the point, which is google's most excellent product search at www.froogle.com . If it's buyable on the internet, it's there.

Re:google's search becoming steadily useless.

[blind biker]

Thanks, but what I was trying to say (and I'll admit to bad wording), is that not only does google.com search return webstore fronts when I am actually looking for technical information about electronic components (this is the point I did not get across well - I am not looking for shops, but for info), but it returns the worst kind of webshops. The kind that isn't really a webshop at all, as in, you can't actually buy anything from them using the web.

As for froogle: I just tried searching for "NAD 701" (without the quotes). The results I got were quite amusing. Not really related to the fine receiver/amp from NAD.

Re:google's search becoming steadily useless.

[PatrickThomson]

Ah yes, technical information about something that gives you 5 pages of crap, even with "technical manual" specified in the search, is a problem. quite right. Of course, if the manufacturer hasn't released it in the first place, it might just not be on the internet.

Re:google's search becoming steadily useless.

[ben2umbc]

google to me is more relevant than ever. There are so many different search operators that you can use in the search bar. Math Operations and conversions, define: (word) quote: (ticker symbol) maps, businesses.... the list goes on. Try improving your query and see the results.

Re:google's search becoming steadily useless.

[blind biker]

Of course, if the manufacturer hasn't released it in the first place, it might just not be on the internet.

Well, of course. But I am comparing the results I had some 5 years ago and now, for the same items (or same kinds of items), so I am being fair and comparing apples to apples. For example, I have an old Adcom amplifier, for which I found the manual online back in those days. I put the model name in the searchbox, and the manual came up - and I think it was definitely among the top 10 results. Now, (because I couldn't find where the heck I saved the .pdf file) when I search for exactly the same thing, I have to wade through pages and pages of useless stuff, before I found the manual.

I am grateful Wikipedia entries come up as high in the ranks as they do, as those are usually very relevant and well written. That's also why I am an advocate of Wikipedia being the repository of all sorts of knowledge - Wikipedia works as a (partial) fix for Google's now broken search algorithms.

Re:google's search becoming steadily useless.

[Tarqwak]

Google Custom Search Engine is the answer: http://search.ratbert.info/ - 1224 hand picked shopping/spam sites excluded, 179 brand sites boosted. Geared towards IT gadgets.

But how useful is it all?

[Colin Smith]

I mean, really. 90% of it is junk.

 

Dynamic pages pollute count

[Coolhand2120]

There are so many dynamic pages on the net now that one web site, like slashdot as an earlier poster commented, can contain literally millions of pages. People use programs like modrewrite, isapirewrite and linkfreeze to manipulate spiders into crawling pages that are near identical. For more than one customer I've made meta, title and content randomization, serialization and or URL rewriting schemes to make damn sure spiders index every possible dynamic page, and it works. I have a single dynamic page that must have been indexed hundreds, maybe thousands of times with slightly different content, and they are all in the index.

Google tries to detect a dynamic page by looking for ampersands and equal signs, as well as looking at the content of the page, it is really quite easy to fool.

e.g.: http://somesite.com/itemlist.php?listmode=1&category=beds&orderby=7
when 'rewritten' shows up as
http://somesite.com/items/1/beds/7.html

So 1 billion web pages could be, and I know a few thousand pages like this, just a few hundred thousand dynamic pages. Not that the pages don't have relevant information, some of the stuff can be redundant though. For instance, when the spider crawls across "Records per page = 10" > "Records per page = 20" > "Records per page = 30" etc.. or when lazy programmers don't use cookies and databases to store information but try and concatenate the URL with the user's selections. Thank god for that GET limit. People need to use POST!

If someone knows how to stop this message board from creating links out of false URLs please, let me know.

Re:Dynamic pages pollute count

[cheater512]

Google seems to filter duplicates rather well.

I doubt its a problem.

Re:Dynamic pages pollute count

[Anonymous Coward]

> If someone knows how to stop this message board from creating links out of false URLs please, let me know.

1. Use example.com: http://en.wikipedia.org/wiki/Example.com
2. It seems the "Extrans" and "Code" post formats won't turn the url into a link:

http://example.com/itemlist.php?listmode=1&category=beds&orderby=7

Google proved rich = evil

[Anonymous Coward]

Google has proved, to become rich one has to do evil.

Adsense is the cause 90% of poor quality MFA web pages.

Mod parent down - completely false

[ricotest]

All three of his examples go directly to the most official site for Firefox, Linux and OpenOffice respectively. Nice try though.

That said, Google's results are still generally better.

Who cares?

[nfk]

"Why is McDonald's still counting? How insecure is this company? Forty million eighty jillion killion trillion....is anyone really impressed anymore? Oh eighty-nine billion sold! All right I'll have one."

-- Jerry Seinfeld

Meh

[Anonymous Coward]

Call me when the index is bigger than the pr0n index on my PC

hurray for google

[Danzigism]

regardless of the amount of legitimate websites that are a part of that 1 trillion, Google always deserves positive recognition in my opinion. i really wish to see them grow even bigger than they are because of all the incredibly helpful things they have provided for humanity and mostly free of charge. i really hope to see more businesses especially, start using Google's online services a lot more. such as Google Docs, Google Calendar and Gmail. So many businesses in small towns are tricked in to wasting their money on Microsoft licensing because they simply aren't aware of any other solutions. Between the amount of money that is wasted on MS Exchange and MS Office, think of the money that could be saved. when you think of how big Google is, you still don't think about the fact that the majority of people have no idea what other services Google provides other than their search engine, or if they are lucky they may have seen Google Earth on one of their friend's computers. there are many uneducated people out there that don't know even 3/4ths of the services that Google provides. it sounds stupid, but unfortunately it's true and I can only imagine what some business owners would think if there was an absolutely free version of the software they use on a regular basis.

Motivitation for the founders' studies?

[AlpineR]

Sometimes it's nice to be called Dr. Brin or Dr. Page. Especially when dealing with somebody who doesn't know you or talking to a roomful of PhD's (such as their employees).

Re:Motivitation for the founders' studies?

[Anonymous Coward]

i highly doubt it man. i know what you mean but when you are the founders of google, everyone will be listening..

Cluebat

[symbolset]

Now go to google and type "live search", "Microsoft" and "Microsoft Office" (without the quotes). If you can't explain why the ads and search results that Google puts on those pages are qualitatively better then you're not qualified to judge my comment. On a lark I went back to Live and did those searches too. If you search for "Microsoft" on Live today it shows three ads, each of which is likely to be more harm than help. This is why Microsoft is third in search and fading despite wasting billions on it. They just don't get it and they never will.

BigInt

[revlayle]

google finally has a use for a bigint auto-increment primary key now!!!!

Re:BigInt

[Tablizer]

google finally has a use for a bigint auto-increment primary key now!!!!

With today's variable-record technology, integers should be auto-expanding. Why pre-limit the bytes? A small key will take up very little space and a big key will take up lots. Perhaps its because fixed-sized byting makes indexes more efficient? But an RDBMS can still offer an auto-sizing integer as an option.
     

Re:Wow, that's a lot of porn (musins

[Anonymous Coward]

And yet, the phrase "hot teen ass" accounts for less than a million of that trillion....

As for the phrase "star trek ass", it's barely even worth mentioning...

What about error rates ?

[freddy_dreddy]

Isn't 10E12 in the range of soft errors in computing ? The statistical change of a bit changing somewhere ... how do they deal with that ? And we're just talking about the amount of URLs (basically pointers), not the amount of data.

THAT'S OVER...

[gparent]

nine hundred and ninety nine million nine hundred and ninety nine thousand nine hundred ninety nine!!!!!!!

But do we know??

[Schmyz]

The saturation level of the "net"???

if you feel lonely on your crappy home page

[mapkinase]

If your guestbook is empty, if the only person who checks your page is your mom, know that you are not alone: a friendly Google bot (think Wall-e) is visiting your page regularly, thoroughly parses through your lols and other pimples, indexes them and assigns shiny sunny ranks.

Kinda warms your heart, ah?

what now?

[nx6310]

Maybe the www will finally have multiple encoded links enabled to get google a kicking in Chinese and Arabic.

Obligatory Futurama Quote

[kalidasa]

"My God, It's full of ADS!"

Googol

[po134]

Hope I will still be alive when they reach Googol indexed pages

Thirty posts in before I get the reader

[symbolset]

Now go round up the understander because if the team doesn't get what I'm saying here their mission will fail no matter how often you dismiss informed criticism.

Not that I care -- making fun of them is one of my favorite things. They should buy AOL and Yahoo. It would complete the triumvirate of Internet Suck and implode. That would be fun to watch.

well

[banners56]

good for google