Slashdot Mirror
Google URL Index Hits 1 Trillion
mytrip points out news that Google's index of unique URLs has reached a milestone: one trillion. Google's blog provides some more information, noting,
"The first Google index in 1998 already had 26 million pages, and by 2000 the Google index reached the one billion mark. Over the last eight years, we've seen a lot of big numbers about how much content is really out there. To keep up with this volume of information, our systems have come a long way since the first set of web data Google processed to answer queries. Back then, we did everything in batches: one workstation could compute the PageRank graph on 26 million pages in a couple of hours, and that set of pages would be used as Google's index for a fixed period of time. Today, Google downloads the web continuously, collecting updated page information and re-processing the entire web-link graph several times per day."
Or it didn't happen.
Once the index reaches a google (or rather a googol), the universe explodes.
[alk]
Seriously, since the web is something like 42% porn. (Yes, that is the ultimate answer.) So that's on average, 60-70 pages of each person in the world naked.
How many of those are automatically generated rank-spoofers, 80%?
My favorite spoof pages were the ones that randomly substituted search terms into porno stories.
"Yes!" she screamed as he thrust his SAMSUNG CD PLAYER deep into her. "I want you balls-deep in my CHEAP HARD DRIVES!" The smell of DISCOUNT SOFTWARE filled the room.
Kwisatz Haderach
Sell the spice to CHOAM
This Mahdi took Shaddam's Throne
i wish they would work on weeding out the crap. anything you google now is infested with cheesy search sites that list other websites and try plaster you with ads. they contribute zero to the web.
If you mod me down, I will become more powerful than you can imagine....
..knowing that the vast amounts of porn just keep getting vaster. And more searchable. Amen. *sheds a tear or two*
[Slashdot Comments We Liked]
So unless there is a screenshot showing the 1,000,000,000,000 site count, Google's index didn't reach that milestone? Even if it now shows 1,000,000,000,001?
The 1,000,000,000,000th page had only one word on it:
"woosh"
Counts of words:
the: 18.3 billion pages
a: 23.9B
0: 12.7B
1: 25.4B
in: 17.1B
I: 10.2B
I know these numbers aren't exact, but you'd think one of them would be over 100B if Google is really indexing a trillion pages. What's on them? Anyone find any keywords that produce more?
Sometimes it's best to just let stupid people be stupid.
This might be off-topic but I wonder what's going on with Sergey Brin and Larry Page's [PhD] education? Just wondering...did they give up?
I couldn't agree more.
Many of the clients I support are constantly asking me "Is there a program that does this? or Can you find me a program to do this" etc etc.
I used to be able to just use google to help me get started but these days the top level searches are all those bloody link farms peddling "free" software, even when typing in the word review you come up with link farms that offer no reviews.
Google is headquartered in Mountain View, CA -- I know, 'cause I googled it. Now, California is rather inclined to think of itself as it own country (some would say, universe), but it is indeed part of the United States of America (again, I checked with Google). And in the US, "trillion" == 1E12 (again, Google).
Generally, bash is superior to python in those environments where python is not installed.
They have identified that there are 1T pages out there, somewhere. They have indexed 40 billion pages. Read the entire Google post. It says it right there.
Bad on Google for the misleading post. Bad on the submitter for not reading the misleading post. Bad on Slashdot for further descending into mindless repetition of mindless submissions of mindless PR announcements.
If I wanted a sig I would have filled in that stupid box.
But how many of those trillion pages have unique, useful content? E-mail is over 95% spam, and the web is getting there.
There were about 153 million registered domains at the beginning of the year. The ones from the spam-friendly registrars are mostly junk. Tim Bernars-Lee said in 2006 that web junk was becoming a major problem, and it's become worse since then.
If you throw out all the anonymous but commercial domains (we call them "bottom-feeders"), as we do with SiteTruth, the Web looks a lot better. Search engines are getting stricter about this. You don't see that many "landing pages" in Google any more. Bad news for companies like Marchex, the publicly traded web spammer that cranks out all those junk "What you need, when you need it" sites.
"The mass trials are going well. There will be fewer Russians, but better ones." - Greta Garbo in Ninotchka.
Many of the clients I support are constantly asking me "Is there a program that does this? or Can you find me a program to do this" etc etc.
I used to be able to just use google to help me get started but these days the top level searches are all those bloody link farms peddling "free" software
Have you tried SourceForge? That's what it's there for, you know.
Tomato wedge sperm darts that are Republican.
And you'll be back faster than a Google search result. Weeding out the crap?
Just for a sample, try this one: getfirefox. If the first link on that search goes to a Mozilla mirror you will win one Internet. Try Linux. Hey, this is fun. Spoiler: the first link there is always "www.Microsoft.com/Windows : Special Offers from Windows Vista® w/ the Purchase of Select Laptops." The first time I tried this I was looking for Open Office and wound up misdirected to a members only site where you had to register to download a probably spyware infested Open Office and signing up for unlimited pharma spam. The scary part is that the text of the link misled me to believe I was headed for "OpenOffice.org". Try it and see. Let's find more horrifically inappropriate ad placements and query results, shall we? I'll bet you could come up with a really funny one.
Note: Please don't go to any of the sites linked to those search results through live.com. Bad things might happen to your Windows box and there's nothing there of interest for your powerbook.
Yeah, that's a good search result ad, don't you think? No wonder Google is becoming a verb.
Help stamp out iliturcy.
-1 Redundant sure...
But that's sort of along the lines I was/am thinking... take txoof's post alone (or mine, or whoever may reply) there are 3 separate URLS for each Slashdot comment
The Header:
http://search.slashdot.org/comments.pl?sid=626647&cid=24345519
The User:
http://slashdot.org/~txoof
The Score:
http://search.slashdot.org/article.pl?sid=08/07/26/0036245#
How many Slashdot comments are there? It's probably in the high millions, (rhetorical, but I'm interested to know none-the-less) There's like an average of about 250 comments per article, about 25 articles a day, thats about 2 million a year, so 6 million links, then take into consideration stuff like Facebook, which bounces URLs (http://www.facebook.com/link=###/etc) or sites that generate a random identifier every few minutes, making those "unique", gets unexciting quite quickly, Although billions is still fairly high.
I imagine that certain sites, such as sites the size of Slashdot (in terms of dynamically generated pages), make a difference. After all, the index talks in pages, not domains. I bet there's also a lot of junk and redundancy in there, but still, it's quite an achievement to be able to deal with that much data.
Surely you're not saying that Slashdot's full of junk and redundancy and redundancy?
Cogito, ergo sig.
Considering your comment is #24345983, I'd say about 24.3 million comments. Also, I believe there's about 1.5 million different users.
I think google.com's search engine achieved its peak usefuleness about 5 years ago. Now, for the most part when I google for a certain electronic component I get some crappy webstore front (and by crappy I mean I can't actually order the component but must "contact by phone" first) or if I search for an electronic device, be it pro or just home electronics, I get those "Read reviews and compare prices"-sites. Which I hate with a passion. WTF google, you have the world's most talented programmers, can't you weed out this crap from your search? At least so it doesn't come up as top hits?
"The agriculture ministry is not in charge of Gundam" - Japanese ministry official.
i wish they would work on weeding out the crap.
There are a *lot* of people at Google working on that problem. Please understand that it is really difficult to keep up with new attacks when your site is #1, because many people out there are aiming directly for it. No matter how many work on ranking and relevance inside the company, there will always be 10x-100x that number of people outside who are working on the shady side of SEO, spamming, etc. It's a never-ending battle, much like spam email. We're trying.
anything you google now is infested with cheesy search sites that list other websites and try plaster you with ads. they contribute zero to the web.
We're working on that both from the search side (ranking) and the ads side (not letting those sites run using Google ads).
If you want to help, you have many options:
Also, I believe there's about 1.5 million different users.
yeah but if you take out Twitter and all his sock-puppets you'll just be left with 500K unique users...
There are so many dynamic pages on the net now that one web site, like slashdot as an earlier poster commented, can contain literally millions of pages. People use programs like modrewrite, isapirewrite and linkfreeze to manipulate spiders into crawling pages that are near identical. For more than one customer I've made meta, title and content randomization, serialization and or URL rewriting schemes to make damn sure spiders index every possible dynamic page, and it works. I have a single dynamic page that must have been indexed hundreds, maybe thousands of times with slightly different content, and they are all in the index.
Google tries to detect a dynamic page by looking for ampersands and equal signs, as well as looking at the content of the page, it is really quite easy to fool.
e.g.: http://somesite.com/itemlist.php?listmode=1&category=beds&orderby=7
when 'rewritten' shows up as
http://somesite.com/items/1/beds/7.html
So 1 billion web pages could be, and I know a few thousand pages like this, just a few hundred thousand dynamic pages. Not that the pages don't have relevant information, some of the stuff can be redundant though. For instance, when the spider crawls across "Records per page = 10" > "Records per page = 20" > "Records per page = 30" etc.. or when lazy programmers don't use cookies and databases to store information but try and concatenate the URL with the user's selections. Thank god for that GET limit. People need to use POST!
If someone knows how to stop this message board from creating links out of false URLs please, let me know.