Slashdot Mirror
Google's Technology Explored
RobotWisdom writes "Internetnews offers a moderately detailed peek at Google's technology. For example, they use stripped-down Red Hat on a massively redundant network, and they're starting to have success with automatic clustering of concepts, so that pages can match even if none of the words in your query actually appear on the page." Additional analysis on InformationWeek and C|Net. From the article: "As a search query comes into the system, it hits a Web server, then is split into chunks of service. One set of index servers contains the index; one set of machines contains one full index. To actually answer a query, Google has to use one complete set of servers. Since that set is replicated as a fail-safe, it also increases throughput, because if one set is busy, a new query can be routed to the next set, which drives down search time per box."
It really is amazing to think of the amount of information and data that we can access so quickly these days. When I stop and think about what my little search query goes through to bring me an almost instant response, it almost seems impossible. Of course the search engine side of this is only one example, but it's a nifty insight into how powerfull our infrastructure is these days. Bravo, mankind.
Google's redundancy theory works on a meta level, as well, according to Hoelzle. One literal meltdown -- a fire at a datacenter in an undisclosed location -- brought out six fire trucks but didn't crash the system.
/.ing could do this. On the other hand, they have a level of redundancy and up time many businesses would kill for.
Gee.. I wish our
Fly me to the moon Let me sing among those stars Let me see what spring is like On jupiter and mars
It's also amazing how much of the general knowledge of the world we *can't* access, because it's unconnected or unpublished.
Just think about how vast and extensive Google's search is, and then think about how little of the World's knowledge and creative achievement it actually can access.
The quantity and breadth of human knowledge is breathtaking, no?
It's been tried. From TFA:
One literal meltdown -- a fire at a datacenter in an undisclosed location -- brought out six fire trucks but didn't crash the system.
I'm not good in groups. It's difficult to work in a group when you're omnipotent. - Q
From the summary:
they're starting to have success with automatic clustering of concepts, so that pages can match even if none of the words in your query actually appear on the page.
From the help guide:
By default, Google only returns pages that include all of your search terms.
Which of these is correct? If it's the summary, is there any way to turn this behaviour off? I find it immensely annoying.
Guy asked me for a quarter for a cup of coffee. So I bit him.
Here it is, from one of the Google guys:
Google: A Behind-the-Scenes Look.
Simpy
Do they share these patches with everyone else?
" pages can match even if none of the words in your query actually appear on the page"
The main flaw I've found in Google's results has been when it returns pages without one of my query words, which doesn't respond to the sense of my query. Sometimes it's changed page content at the same URL, so I go back and get the "cached" page, if it exists. The cached pages reveal in their headings whether the page matched only because the query word was found only in another page linking to the returned page. I'd like their immediate results to show that distinction, and to have links in the results to click around those pages related by my complete query. The current click/back/"cache" combinations are frustratingly disconnected, conflicting with Google's otherwise smooth immediacy.
--
make install -not war
Question -- and this may be a dumb one, but I'm going to ask it anyway:
How much of what Google is doing -- the clustering, the redundancy, the sub-categorization -- how much of this (if any) could be described -- could fit under the mantle of "Peer-to-Peer"? Is anything that Google is doing here remotely considered P2P? (Even if the P2P is what's going on on their own, in-house servers?)
Obviously, I ask this because of the upcoming supreme court case. And I ask because it struck me as I read the article that what Google is doing *seems* to be breaking down complex tasks and simplifying them so that work across the network -- their network, your network -- and I wonder if this is (in theory?) what Peer-to-Peer is doing?
(I'm thinking, too, of the Google concept of "shards" and how their data is distributed.)
Uh? Google cache is runned by bbernal.com not Google. This is a little better: http://64.233.161.104/search?q=cache:64.233.161.10 4 but still not surprising if you think about it for a while.
http://www-db.stanford.edu/~backrub/google.html
Try doing a search for a Macintosh software product. Even though "Mac OS X" was not one of your search terms, Google boldfaces it as though it were!
I can't reproduce this with another term. I wonder whether this was a manual fix by Google programmers.
For more information, click here.
Heh, well they could NEVER do that :)
Here's another great idea you inspired that they could also never do (being a commercial company themselves and all).
When I am searching I virtually always want to do one of two distinct things:
1) Sarch only commercial sites for a product to purchase.
2) Search everything but commercial sites for information.
There really should be a "$" flag that you could add (or at least a "!$" flag) to control wheather you see commercial or non-commercial sites in the results list.
Contrary to popular belief, coding is not all free blow-jobs and beer. Those things cost MONEY!
What's interesting is that the notice "Google is not affiliated with the authors of this page nor responsible for its content." goes away when you look at the cache of Google.com! That's a change from the last time I looked at Google's cache of Google a couple of years or so ago.
Google really slaps together a pile of junk.
Parts fail left and right, and nobody bothers
to fix them. The software hides all this from
the users.
Google even checksums the data, on the assumption
that it is frequently getting corrupted by all the
junk hardware they buy.
On their own servers, then they're obeying the rules.
The question is: Do they use these patches on the search appliances they sell, and does that count as "distribution"? I honestly don't know the answer to that question, and I'd like to think Google has sharp legal advisors to go with their sharp technical people.
The company also is applying machine learning to its system to give better results. Theoretically, he said, if someone searches for "Bay Area cooking class," the system should know that "Berkeley courses: vegetarian cuisine" is a good match even though it contains none of the query words.
FYI.
Not A Sig
Anyhow, the article mentioned that in these early datacentres they experienced something like a 25% hardware failure rate, but that it didn't matter because the software worked around it and the hardware was cheap.
Here's a link to the page where I read all this neat stuff. It's probably mostly about the same stuff as the article we've all just slashdotted, but I won't be albe to tell for a while....
Never eat more than you can lift -- Miss Piggy
Why not enhance the robots.txt format to include a max crawl rate variable? Let the webmaster specify how often a robot is allowed to crawl a page.
http://brandonbloom.name
Hmm. It must have been corrected; I did a direct copy/paste for my quote.
What I say does not represent the views of my employers, my friends, my cats, or myself.
I've done a more studying in that area than most. There has been a lot of over-reacting to paradoxes such as this. Godel's Incompleteness theorem is only narrowly interesting: as soon as you start talking about physical things, these paradoxes are much less imporant.
A set which contains all sets which do not contain themselves may be a conundrum, but a catalog that lists all catalogs that do not list themselves is merely impossible (trivially impossible, in fact). There are plenty of things that can be described in English that aren't possible things, and most of them aren't very interesting.
The important consequence of Godel's Theorem to physical things was that mathematics is not a completely accurate model of physical objects. One physical object plus one physical object equals two physical objects, but not every equation describes the physically possible (OK, it was already known that this was the case, but Godel showed it was the case more often than expected).
Socialism: a lie told by totalitarians and believed by fools.
"they're starting to have success with automatic clustering of concepts, so that pages can match even if none of the words in your query actually appear on the page."
I have yet to see a "hit" served up by google where it didn't have any words I searched for and it still be relevant. It's especially annoying when I search for exact phrases (such as an error message) and I get something completely different. It's a waste of time so far.