Slashdot Mirror
What's It Like to be Google's Boss Techie?
We'd like to welcome Google Director of
Technology Craig Silverstein as our next Slashdot
interview victim... err... guest. You think you
run a big Linux server farm? Craig's is bigger.
Think your Web site gets a lot of traffic and
creates a lot of headaches? Just think what Craig
must face! Post whatever you'd like to ask Craig
below, one question per post. About 24 hours after
this runs we'll email Craig 10 of the
highest-moderated questions, and we'll post his
answers shortly after he gets them back to us.
Google always seem to be early-to-market with some really highly developed software solutions, and also always seems to have the backbone to support them.
I'm curious -- what drives the innovation? Is it the hardware team advancing architecture to permit the software team more room to play, or is it the software team saying, "Hey, look what we got!" and the hardware team dropping the iron to implement it?
I understand there must be some level of synergy, but is it completely seamless or is one side of the equation effectively driving the other?
Leem
Can I have a job?
Good judgment comes from experience.
Experience comes from bad judgment.
What type of machines/setup does Google use?
(I've heard thousands of PC's with everything in RAM, but I'd love to hear it from the horses mouth)
What are the weirdest queries?
A relatively simple, non-intellectual question, but I've always wondered -- just how many hits/how much bandwidth do you consume, and how many servers do you have to handle the load.
________________________________________________
suwain_2
Does google's policy of "ranking" the sites that have hits favor the "big guys" over more specific smaller traffic websites? That is, would a story on a site like CNN get a higher ranking in google on a keyword "Gulf War" than say a site (gulfwarveterans.com) that deals 100% with the Gulf War? Do you think you are leading to the commercialization of the web (i.e. the big power players) over smaller sites?
is your monthly bandwidth bill for google.com ?
but I noticed a few months ago that Cisco now uses the Google engine to search the CCO. Congrats on that one. I've also noticed this new search box that Google is starting to produce. And it looks *very* cool. So my question is basically which is more important to your job the website or selling the service and the engine to people who need it?
Cypherpunks: Civil Liberty Through Complex Mathematics. Those who live by the sword die by the arrow.
Has there been any progress on the Pigeon Computing initiative?
I am the evil aardvark!
A little old but interesting.
g oo gle.
The Technology Behind Google 2000-10-19 (1hr 13min) By Jim Reese, Chief Operations Engineer, Google. How to build an internet search engine that indexes 1-2 terabytes of data 200 million web pages- and serves it up at a rate of 1000 requests/second. (Hint: Start with a farm of 10,000+ Linux servers). The technology behind Google: company overview, search parameters and results, hardware and query load balancing, Linux cluster topology, scalability, fault tolerance, and more. [420]
http://technetcast.ddj.com/tnc_search.html?key=
I am wondering why they chose Linux. Specifically, I wonder how they made the choice between all major OS-es (Linux, *BSD, Solaris and possibly Windows), as well as the software they use to power the site.
The Internet is always described as a distributed system with no single point of failure. Google, however, has quickly become by far the most popular method of locating information. "Surfing" has been killed with modern search technology, it's so much easier to look through Google than the Web itself. If Google was down, I'm sure the Internet would be far less useful.
Do you think Google has become an Internet point of failure? With the competition for larger and larger indexes, is the Internet becoming centralized? Do you think this is a bad thing?
What are you doing to prevent the new generation of more sophisticated search engine spammers- spammers that use advanced software such as WebPosition Pro, spammers that feed fake pages to the Google crawler, spammers that make bogus link pages to their own sites? Doesn't this new level of sophistication on their part mean that in large part Google must emphasize human website reviewers, such as those provided by the Open Directory Project, to a greater degree?
How much bandwidth is running into his network? How many locations does he have, Are there pictures or diagrams of the network infrastructure side of his setup, and apart from Programmers how many network design type people work there?
-- -- Warning. Do not stare directly at the sun.
As a new network configuration guy, I am often stumped by a problem. I usually turn to google first, and my supervisor second. What has been the biggest problem that you have dealt with that will stand out in your mind years from now? As the "Head Techie", where did you turn, and what was the eventual resolution?
I'd rather you do it wrong, than for me to have to do it at all.
---- El diablo esta en mis pantalones! Mire, mire!
Has the Linux platform made inroads into the staff arena at Google? What platform do you develope on?
Does google plan on releasing more products like the Google Search Appliance in the near future - specifically those that are geared more towards the consumer level rather than business market? I would, personally, love to have some sort of google search engine on my machine to rummage through all the stuff I have. Does google plan on expanding into this market or will you remain focused on the web?
I know, I know, Only one question but - it begs to be asked - how well is your technology going to be able to scale? Considering the near-expotential growth of the internet will PageRank be able to keep up?
I understand that Google was using large numbers of IDE drives in lieu of more expensive but individually faster SCSI devices. What prompted the decision, and how have the concerns of reliability and performance been mitigated. What special technology, if any, was used to implement such a system
...as to what exactly Google does with the concepts it receives through the various Google-tech contests held. Have these ideas been made good use of? Do we see any of this in the Google we use every day? What about the ones that didn't win, do we see any of them?
Whats the google language of choice for web page building. I'd assume speed is the most important, so what language makes google so fast?
Good quote, too many chars. Seriously, the slashdot 120 char limit sucks!
Is there anything new that Google is working on that is not currently displayed in your labs section? If so could you explain it to us?
If you could sum it up in a nutshell, maybe you should be writing O'Reily books. --- Domasi 2001
Google's success has been well documented. Quick, relevant results are it's trademarks. Do you see any backlash against Google and what are you doing about people that use Google's success for their own purposes (ie Google bombing)?
I have a shitty sig!
Hi Craig. Google is my favorite search engine, mostly because it's so simple, fast, and has a very professional feel.
I wonder, when you're in charge of something as huge as Google, are you on call 24/7 in case something goes wrong? Have you ever been called during, say, a nice dinner, or worse, in the middle of the night? Thanks.
as Google got more popular and eventually reached the status it holds today, did you feel any pressure (either internally or from outside the organization) to switch from a Linux based cluster to a proprietary solution (Windows comes to mind, but there are others). Where you (or others at Google) affected by any of the FUD that is put out, and did it affect your perception of Linux as a viable solution?
I was thinking this exact same thing. I'd love to hear more on this. :)
... what linux distrubution does the world's largest server farm use?
It seems that Google's great successs is partly due to research coming out of the academic world. How many google employees have advanced degrees, and can they publish non-proprietry research after they join Google? How do you see the interplay between high-tech and Academia?
HOWTO get better dates on slashdot
Is Google traffic growing enough to require frequent upgrades--or is traffic leveling out?
What type of Database backend do you use and what led google to choose it?
Since sites like slashdot don't like to give out their statistics, I'd like to ask, what percent of users use what web browser? Also, what percent of users use what OS?
Good quote, too many chars. Seriously, the slashdot 120 char limit sucks!
Does Google use any natural language processing (when dealing with web pages, queries, etc.)? Are you planning on doing more with NLP in the future?
She sat at the window watching the evening invade the avenue.
Hypothetical, and hopefully fun, question.
Now assuming you had an infinite IT budget, generally which configuration of hardware/os (e.g. PC/Linux, PC/Win (boo hiss), IBM/AIX, VAX/VMS, Cray/Unicos, etc) would you adopt and why. More specifically, if pure performance were the only consideration, which would it be. Alternatively, if uptime were the primary consideration, which would it be.
Be honest and don't worry about the bias's of your audience.
How have these affected you and your job, and what are you feelings on this subject?
Xaotik Designs
What kind of bandwidth/pipes/networking setup do you use -- and how does your "macro" capacity diffuse down to each clustered server?
Basically, what's your setup and how does it work?
I have but one question... Who is the mastermind behind all the "special" logo changes that Google experiences throughout the year?
My hats off to that team!
-= Xafloc =-
alinuxbox.com
N
When will google be able to index porn? That would be a great feature that most of us geeks would like to see ;)
ps.: Keep up the good work!
Has google considered offloading traffic to other architectures besides server farms? Specifically, have you considered a distributed client/server model like peer-to-peer?
How has Google remained competitive?
Why in this day and age does google continue to penalize sites that are virtual hosted? With ip addresses becoming harder to get/justify every day why does google discount the relevance of links that don't come from a unique ip address. Please don't just deny it, I think the Internet community deserves an explanation.
Google recently ran it's "first annual programming contest," with a winner receiving $10,000. Many slashdotters suspect this was simply a way to recruit new talent. So, was finding new people one of the initial goals for this project, and have you hired any new programmers as a direct result of it? What were the other goals (PR, generation of new ideas, etc) where there?
Engineers arn't boring people, we just get excited about boring things.
That the site I run is on the top of the list when you search for"ask dick"
Cool eh?
Who draws all of your wonderful and witty Google Logo artwork?
It's well known that you use Linux in your mega clusters. I was wondering if you have ever been approached by Microsoft, Sun, or HP in an effort to switch to their proprietary OSes.
I can't imagine that you haven't. It must have been a huge decision to invest in one technology, so are you satisfied with what you have?
Moderation: Put your hand inside the puppet head!
What version of what OS do you use and why?
Also, what are your plans if any to change this
in the future?
And what do you use to benchmark/stress test your setup/servers?
Thanks!
Recently, the english division of our company [black and decker] hired 'HyperMedia Trafficing' or some other similar named company to get them 'more exposure' in the search engines.
.. or why no one bothered to ask me what to do.]
.. How does Google plan to make sure they are :
.. well .. pretty much anything.
[forget the ethical debate about that
What I want to know, is - going fowards - as more and more of these companies start up, and discover more and more unscrupulious ways of 'loading' the search engines with bogus hits/visits/data/etc.
1) Not loosing ad $$ to these folks
and
2) prefenting every search from returning something like www.hotgrannysex.com or www.top50.com as the 1st (or first 15) results for a search on
--Ne auderis delere orbem rigidum meum, non erravi pernicose!
What did you guys do with all those harding working Pigeons?!
Seriously though... can they actually do mork work than than these penguins?
No offense to Mr. Silverstein, but I'm much more interested in Cindy! Beautiful, highly successful nerds are terribly rare!
Just so I'm not off-topic:
Mr. Silverstein, how does Cindy look in tight sweaters?
Drool...
Talisman
"Study your math, kids. Key to the universe." -The Archangel Gabriel
One of the most impressive things about Google to me is how easily you seem to have embraced an open model. I realize the outward view of a company can be quite different from the internal view. How easy is it actually to make decisions such as opening API's. If it's easy can you give some advice on how one might convince their boss.
Thanks,
-Dave
"as plurdled gabbleblotchits on a lurgid bee" - Prostetnic Vogon Jeltz. (One man's humorous is another mans flamebait)
Hi Craig!
I think Google absolutely rocks. It has by far the most intelligent/helpful search engine results. Thanks for the great service.
Now onto the questions- what is the Google vision / strategy for the future? Where can Google go? From a search engine perspective, what are some of the challenges that you have and improvements that can be made (perhaps speeding up crawling to make the latest content available, for example)? How are you going about solving these challenges, and when can we expect them to be implemented?
On a similar note, I've noticed that recently Google announced a "google box" that allows for corporate to take advantage of the google search algorithms and indexing. Any more products like this being planned?
This shoulda been asked already, but I don't see it. What's next? Google has consistently added more and better "convenience" features than any other search engine. What's the next big thing? Are there any fundamental technological changes coming?
"He who would learn astronomy, and other recondite arts, let him go elsewhere. " -- John Calvin, commenting on Genesis 1
Since you have such a large farm, how do you keep track of the various computers (state/performance etc.) Do you have a network management software built in house or use 3rd party software if at all.
Is there any way we can find out what kind of suggestions Google receives from the public? It would be quite interesting to look at them all, and maybe some of them could now be implemented using the Google API.
-- Ed Avis ed@membled.com
I have a number of web servers, some Unix some Windows, and the number of attempted attacks each day from different IPs must run in to about one hundred. It is mostly people trying to execute commands or using malformed URLs trying to exploit some known past security hole. My question is, how many attempted attacks each day do the Google servers get?
Martin Piper
Owner - ReplicaNet and RNLobby
Google is a great free public resource. My concern is that it has to be expensive running a resource like that. I know Google's strategy is somewhat to use the free resource as a loss leader to promote your search technology, but the key word in "loss leader" is "loss". It's a great theory as long as you are able find people who want and need your search technology.
So my bottom line question is this: Does the web site pay for itself via the advertising? Is there a possibility that someday Google may decide the web site costs too much money to run if you get to a point where your reputation no longer needs the loss leader?
Sometimes it's best to just let stupid people be stupid.
It would be great if you did a documentary feature with TechTv or someone, because its one thing to read about your facility, but it would be another to see it.
Thanks for all of the help I've gotten from Google.com, I don't think I'd still be in schol without it.
Paradesign
PS, even just a photo feature on the site would be nice.
I want 2D games back.
Anyone who has ever needed a piece of information that was on a broken page will agree that the Google page cache is perhaps one of the most underrated and useful parts of your search engine.
There's one problem that everyone has with the cache, however - you don't deep-nest the caching, so that following any links on a cached page will lead to the original (probably broken) site, instead of to another cached page. Is there a technical or legal reason for why it works this way? Any chance we'll see deep caching at some point?
google seemed to be one of the few (only??) sites that managed to deal with the deluge of extra traffic. can you tell us a little more about google on that awful day?
I expect that the California heat and thousands of boxen require special measures to prevent overheating. What kind of measures do you take for keeping your server farm operating normally at a cool temperature?
StroudCo Heavy Industries is going to buy Google. Just you wait.
What was your favourite search engine before Google was launched? Which other search methods do you use other than your own site? Do you remember Yahoo! when it looked like this?
Conversion Rate Optimisation French / English consultant
What is the air velocity of an un-laden Eatern-European Swallow?
I am the nightmare of nightmares.
How do you avoid business pressures to make short-sighted solutions, and consistently make good, common sense ideas work instead of adopting ones from marketing sources? Not only does Google have the best search engine technology, but you consistently do the "right" thing. Clean, quick homepage, text only well-identified ads, interesting research projects, etc...This is the way many search engines start, but they all went the way of the "dark" side instead of adopting the "right" solution. In my jobs, it's been very difficult to execute and justify good engineering (or just common sense) under pressure from the people who control the money. Any advice for driving through well-thought-out decisions instead of adopting the "management fad of the month"?
Not to be too "X-File'ish", but does there come a point where too much knowledge is captured in Google? A point where anything that doesn't exist in Google doesn't exist, period? Wouldn't that represent a very tempting target for a bin Laden or a John Ashcroft, to try to control how the modern world thinks?
Kind of far out there, I know, but do you guys worry about this kind of thing?
sPh
How do you roll out patches, new versions of software, etc? How do you make sure that stuff is running?
Many sites, when referenced by Slashdot, crumble under the load. Can you folks see any difference, either to your "main" servers (www.google.com) or your cache servers?
Stupid job ads, weird spam, occasional insight at
What has been the reaction to the Google API
that was announced a while back? Have you been
able to monitor a specific rise in traffic
from sites using the API?
Can you give a rundown of what your average work-day is like...and what about your hectic days?
-- From my Best Friend (Written to me over ICQ): "i was gonna go to a party...but i had to reinstall windows"
just a compliment.
great search engine!
With all of the talk about the federal government of United States of Amerika wanting all ISPs to log their traffice:
Do you think President George W. Bush should be
impeached?
Thank you
Important News for Linux Enthusiasts
COURTESY OF THE NEW YORK TIMES
Gay Linux Stores Feel the Pinch of Customers' Liberation
By MARTIN ARNOLD
For the owners of most Gay Linux and lesbian bookstores, there seems to be little to be celebratory about this Gay Linux and Lesbian Pride Month. The owner of the Oscar Wilde Bookshop in Manhattan, generally acknowledged as the world's first Gay Linux and lesbian bookstore, said, "It's about to go out of business." Little more than a year ago, A Different Light, one of the country's largest Gay Linux bookstores, closed its New York branch.
Clearly the outlook for such stores is grim. Until recently, much the same could have been said for independent bookstores generally, their territory having been largely gobbled up by the march of chain stores across the land and the buying of books on the Internet. But there now seems to be a bit of a resurgence of the independents, publishing executives say. Still, the Gay Linux bookstore is seriously endangered.
Part of the problem is assimilation, the very success of the Gay Linux movement. Gay Linux and lesbian issues are now so openly discussed in the mainstream media that it's almost as if Gay Linux literature were no longer niche publishing. And that being true, writers and their publishers want their books to be displayed on the chain stores' A-to-Z shelves, not just in Gay Linux sections, and certainly not in Gay Linux bookstores only.
Surprisingly, the slackening in tourism affects Gay Linux bookstores. In large cities a stop on the itinerary for Gay Linux visitors is the Gay Linux bookstore, where they can often find reading not easily available back home.
"The loss of the tourists certainly affected us in Washington," said Deacon Maccubbin, owner of Lambda Rising, a Gay Linux bookstore there with two branches in Maryland and one in Virginia. "It's always been a significant part of our business. Tourists come in to get the free local Gay Linux newspapers to find out what's going on in the area for Gay Linuxs, and they buy books."
Gay Linux bookstores also have generational problems.
Larry Lingle, owner of the Oscar Wilde Bookshop in Greenwich Village and the Lobo Bookshop and Cafe in Houston, said, "Fewer people read now, and that's just as true of Gay Linux readers as it is for others." He added that most of his regular customers "are at least 50 and over."
"You don't find younger ones reading much," he said. "But if they do, they are addicted to buying on the Internet."
African-Americans "support their authors and stores, even a book signing by Gay Linux black writers," he said. "But younger Gay Linuxs don't. I had a lesbian writer in the store for a signing. She signed books but said she buys the books she reads on the Internet."
For younger Gay Linuxs and lesbians, societal acceptance is a matter of course. Kim Brinster, manager of Oscar Wilde, said: "When I was coming out, it was drilled into us the importance of supporting Gay Linux restaurants, Gay Linux bars, Gay Linux bookstores. But now Gay Linuxs take this all for granted, a byproduct of assimilation."
So in the general malaise of book publishing, Gay Linux and lesbian publishing appears to be in a particularly quiescent period. Think of this: New York is the only city in the country with more than one Gay Linux and lesbian bookstore, every store owner interviewed for this column said. Jenie Carlen, a spokeswoman for Borders, the book chain said, "The Gay Linux and lesbian category peaked about seven years ago, and since then has been flat and declining as it's moved into the mainstream."
Borders and Barnes & Noble have Gay Linux and lesbian sections, but with a limited number of titles compared with Gay Linux shops. Moreover, many of their Gay Linux books are scattered throughout other sections, particularly Gay Linux fiction, which is gaining a larger crossover readership.
But even with the far greater variety of titles, the Gay Linux bookstore is struggling. "We were a real destination for Gay Linux tourists, and that's starting to come back," Ms. Brinster of Oscar Wilde said. "Our store had slight increases until this year, but now sales are down drastically." Mr. Lingle, the owner, said that he might close the shop "because we can't get any traffic." A book signing, traditionally an attraction for potential customers in any bookstore, would draw a "pathetic" attendance, he said.
"Even the Gay Linux press pays little attention to Gay Linux books, less to bookstores," he said. "Gay Linux bars, Gay Linux parties -- those who spend on ads get the press coverage." He bought the store six years ago, he said, because of "a certain reverence for its history, and unfortunately after six years never made a dime," even though he included hard-to-get out-of-print Gay Linux classics in his stock.
There are only three Gay Linux bookstores in New York, which has the largest Gay Linux and lesbian population in the country, the other two stores being Creative Visions in the West Village and the new Bluestockings Women's Bookstore on the Lower East Side. That's more or less like having a dozen movie houses for the whole city, but at least they'd be full.
Vincent Migliore, owner of Creative Visions, said that the obvious advantages of Gay Linuxs buying books in Gay Linux stores was not only the greater Gay Linux inventory than the chains have, but also that "the customers can talk to people who have actually read the stuff and led the life."
True enough, but that doesn't seem to matter too much in New York or elsewhere. In Denver, for instance, James Dovali, owner of that city's only Gay Linux bookstore, Category 6 Books, said that after 21 years, "I'm almost ready to close." He added: "Yeah, the Internet is going to kill us all. I might survive, if I can pay my bills. Right now I'm just making it, hoping to hang on."
The mainstreaming of Gay Linux fiction -- a paradoxical problem for Gay Linux stores -- can be seen in independent stores like the Corner Bookstore in Manhattan. Christopher Lenahan, its buyer for adult books, said, "As a whole, the sales of Gay Linux fiction have gone up a bit for us because a lot of heterosexuals are now reading them as well." His store, on the Upper East Side, serves a population that is highly educated and well-to-do and "a bit older," he said.
"We are selling more titles that are Gay Linux related," he added. "Completely in fiction. Gay Linux nonfiction doesn't translate."
Trying to figure what's going on in Gay Linux and lesbian publishing and the stores, like much of the book world, is rather like struggling to bottle the wind. But one thing seems clear: unless younger Gay Linuxs bring some of their pride to the literature relevant to them and are willing to spend a bit more to buy books in Gay Linux stores, such stores will soon be extinct, and that will be another unfortunate chip in our culture
reality timed out @ 11:11
Just curious when mod_google is going to be released for the apache webserver. It would be nice to have the power of Google indexing available to those of us without significant IT budgets (i.e. wife won't let me "buy another #$*@! computer").
How do you balance load among the www.google.com servers? Do you guide users to local servers (such as www.google.co.uk)?
Stupid job ads, weird spam, occasional insight at
This question has probably been asked 1000 times but what Linux Distro do you guys use and why did you pick it over the many other great distros?
Did you develop or modify it in any way to meet your special needs?
Do you have a cluster or load sharing enviroment setup on that distro?
Thanks alot Craig We love google!
Vote early. Vote often. Vote CowboyNeal.
What's the worst thing ever to happen to the google server farm? (Besides the pidgeons knawing on cables)
Tim Dorr
Owner/Manger
A Small Orange
How hard has it been and will it be to keep google a simple interface rather than turning into a yahoo or infoseek that takes longer to load the page than it does to execute the search? How many people are pushing for that type of commercialization/sales in your business?
Where did the name Google come from?
I would like to know (Other that what distro they use) what Google uses to efficiently spread apache of mysql or whatever over alot of Linux Servers, is it automated? (NIS, NFS???)
How does google deal with denial of service attacks, particularly distributed ones?
The rest of us just suck it up with fat network pipes, but a high-profile target like google would be the holy grail of Internet vandals.
Has anyone ever poisoned your DNSes, effectively taking Google down even though the server are up? Successfully inserted bogus WAN routing info into the Internet, again effectively bringing down Google even though the servers are fine?
What's your worst cracker/net vandal story?
Craig, are you jewish?
Why don't you use FreeBSD like all the other huge sites?
Common sense is not so common.
Hi Craig,
I'm curious on the future of the Google Search Appliance. It seems like a good solution to help a company index millions of documents to be easily searchable, however have you considered tayloring it to be a drop in solution for database indexing and searching. Many companys have products that deal with data mining through millions of records on relational databases could benefit from Google's technology.
Keep up the good work!
"Be kind, for everyone you meet is fighting a hard battle." -- Plato (427?-347? BC)
To what do you credit the popularity of google? Do you consider google a "success," or are you holding out for thousands of employees and billions in cash flow?
Any thoughts?
Ive lost my keys.
Do you know where I left them?
That's for May. Of course, it's all at the Zeitgeist, as linked in other responses. I don't blame you for not knowing about it, though; I've tried to find it from Google's web page, but couldn't until I searched for it (using Google, of course).
Personally I'm usually pretty drained after a fun day staring at the screen and typing like a monkey, and sometimes completely avoid the PC when I get home, prefering to chill with a decent book (currently Cradle to Cradle), zone-out in front of the TV, or go cycling in the beautiful Isle of Man (watch "Waking Ned Devine" for an idea of the scenery - jealous?<grin/>).
So I guess my completely-non-tech question is:
What do you do in "loafing" time (ie. loaf - To pass time at leisure; idle.), when you've left the office, "lost" the pager/Blackberry/PDA/mobile etc., and got away from it all?
Cheers,not that i hate the parent, but it doesnt need to be asked now.
here's a cache (hahaha, a google-cache of the previous logos, including the dilbert comic ones mentioned in the other reply
|---------------|
practically an AC
Is the order of sites you return solely based on the number of occurences of the search term, or do you also take popularity (eg how many other sites link to it) of those sites into consideration ?
Do you expect widespread usage of RDF/DAML/OWL/TopicMaps for explicit meta-data annotation of web resources, or will it be used only in small circles of specialized content providers like academia, or maybe not at all?
How will Google react? Do you plan to use meta-data provided by web resources if found, and how will you decide if it isn't just made up to get people on some bogus pr0n site (like with those <meta>-Tags today)? Will it someday render the brute-force approach of full-text-indexing obsolete?
Programming can be fun again. Film at 11.
Have you ever considered setting up a distributed search engine client to expand your server farm through your users systems?
Donte Alistair Anderson Roberts - hi son!
Karma: Chameleon
If you had to choose one and only one thing that has made Google the most popular search engine, what would it be?
Google has become such an important part of the Internet for millions of average users. With this in mind, my friends and I often joke about what would happen if (knock on wood) Google were to go out of business. I suggest that ICANN should do something useful for a change, and fund Google as an official, non-profit project for searching the net.
Although I have heard that Google turns a good profit, what exactly is preventing Google from becoming a not-for-profit organization? Couldn't Google take the extra income from licensing its search to create better search technologies and pay the employees, rather than make some shareholders rich? Wouldn't this perhaps make Google a more sustainable organization?
At least once a week now, I read someone who proclaims that "I no longer even use bookmarks or try typing in URLs. I just always go to Google for my information." Has anyone approached you (or have you considered yourself) producing a Web Browser which has no URL line, but instead has a Google line to automatically send anything typed there to Google as a gateway to the Internet? Seems like it would "sell" to the Google-holics.
I remember reading an article a few months back that google was being used to discover exploitable webservers by using a malformed url as the search string. Has Google taken any steps to prevent this kind of malicious usage? Does Google even have a policy on this kind of thing or does it regard this as the responsibility of webmasters to keep their servers patched?
"Watch your cornhole, bud."
You've got some incredibly cool peopleon your Technical Advisory Council. How often do you interact with them?
Stupid job ads, weird spam, occasional insight at
dumbass.
If I had mod points right now... Well, it's at 5 already.
It breaks my pluginses, my precious!
How do you account for the plateau that occurs when you have a pigeon cluster of 7, 8 or 9 pigeons yet results remain in the 500 range regardless of the additional pigeons?
They have a nice graph, but no scale. I suppose you could do some careful pixel analysis of the graph to generate percentages, but it's a shame they don't list them.
Interestingly, I see "Other" has been steadily rising since it bottomed out in January, and has now surpassed Netscape 4. I would love to be able to click on that chart and see a detailed list of the percentages, and what "other" is composed of. Hopefully we'll see Mozilla get its own line on the graph soon.
It would also be nice to see a breakdown on a per-OS basis. I wonder how many people are running Internet Explorer on Linux? (Seriously, that would indicate what portion of non-IE users hack the browser tag to make web sites happy.)
Non-Linux Penguins ?
Google is an incredibly popular and effective website. I'm curious about the amount of pressure you have to expand in order to "stay competitive" or "aptly serve consumer's needs". Is there any kind of a push to go the way of yahoo or amazon and try and include EVERYTHING on that simple page? As things evolve, do you really see Google staying the top engine in 3 to 5 years?
indeed..
Having read about the amount of servers you have running, I have been wondering how you do load balancing of databases and on the web servers?
Fx. using layer 4-7 hardware switches?
my sig
Drawing on your experience and what you consider the best advice you have received, how would you go about getting where you are today if you could do it over again? What motivates you to do the things you do? What would you do differently? What advice would you have for someone with a Bachelors degree in a non-technical field but an associates in computer programming and six years of business experience? In short, what to you credit as the secret of your success and how would you advise others?
~~ What's stopping you?
I've made some really stupid posts to the newgroups in the past and I used my real name. Can you delete them for me?
How do you monitor and track the overall status of the database?
:)
Is there a custom application written to tell you queries/sec, size growth/day, time to return query, overall cpu usage etc of the entire database?
How do you know when its time to add hardware or time to clean the database?
How does google handle dead links and/or dynamic links?
Can you track search trends and correlate with world events?
Does google track and graph usage by country?
How many years of development do you think it will be before search engines are able to intelligently make decisions based on queries and return valid results without having those exact words?
And, finally, just what is your bandwidth usage?
..this is but a fantasy..
Are there any plans to augment Google's server farm with distributed computing, along the lines of SETI@Home? Would this be a benefit or a boondoggle?
Mmmmmm... Bold, yet refreshing!
My question: What's the monthly electric bill?
10,000+ PCs must burn up a lot of power, and generate a lot of heat which requires a lot of A/C.
Evil ZEN Scientist
How do you guys manage thousands of servers spread throughout multiple datacenters?
How do you handle user accounts? Event notification?
Do you guys use "enterprise" software like Tivoli or Openview, or did you roll your own solution?
Conformity is the jailer of freedom and enemy of growth. -JFK
will you answer it?
With technologies like searching google through a program using SOAP/XML, what else (developer related) do you plan on doing to keep on top of the search market?
- tristan
It seems most search engines sell out at some point and become a part of AOL or Yahoo-like. How long do we have to enjoy Google until greed erodes the precious page we know and love?
Why do you have a neurosurgeons on staff?
========================
To: Mr. Google
From: Me
Re: Google's Competition
========================
Most people are visually stimulated and are able to better grasp concepts and relationships when complex queries are organized in visual representaions......
How does the Google tech department feel about up and coming sites like Kartoo (France) that utilze a new visual (Flash) layout that shows correlation between the returned web sites? and does google have any plans to implement such a scheme into the current searching method?
What would it take to Slashdot Google? What do you do to avoid this? Have you been Slashdotted before, either from Slashdot itself or from some other link?
Carousel is a lie!
Where is the GRAMMAR NAZI and why hasn't he plagued this post for the crime of ending in a period (an inpropper declarative sentance) and not a question mark (an interrogatory sentance)?
The people of slashdot would benefit from the GRAMMAR NAZI. I kindly ask where art thou?
(oh and good question by the way)
(I could imagine that google sucks up more bandwidth from pornography queries than from anything else)
(they probably have a 20 node cluster by itself to shape the traffic and another 200 node cluster to flush the staff toilettes)
What kinds of changes have you gotten into the linux kernel to support your special needs, and what kinds of changes are you working on getting added? Are you able to use the standard kernel or require special patches?
How can you possibly test bugfixes/changes that need to get deployed to thousands of machines? Furthermore, how in the heck do you deploy the changes once they're tested. I understand you probably can't describe the exact process, but perhaps you can enlighten us on some principals learned on the subject of CM on such a massive scale.
What the hell was all that Dilbert crap about?
After the introduction of the Google API, some people, especially from the REST camp, criticized the the use of SOAP, claiming it just adds superflous bloat and is generally "unwebby". What do you think about this?
Programming can be fun again. Film at 11.
Although useful, some regard google's web caching service as a copyright infringement. What is your opinion on this?
How does google plan to be competitive against engines AllTheWeb while keeping the quality of the returns as high as they seem to be?
(a/k/a Quality or Quantity?)
forget it.
Google was one of the first major sites to start using the Accept-Language header to dynamically serve content in the user's preferred language header. My question is: How did google decide to do this? What has the response been?
This sig is not the Zahir. Lucky for you.
Can I have a job?
In college, really poor, need a flatscreen.
As adverstising revenue dollars are quickly declining many sites have had to close shop because it is no long fesible to rely on advertising to supplement the cost of having a good website with staff and services that people bother to visit.
Now that the internet free for all is slowing down, what do you see happening in the next 15 years as far as google's business model and the internet in general? Will big business be the only real players? Will google be the TV Guide Channel of the internet? Will google be able to survive on only advertising revenue, or will it need to supplement it's revenue with selling technlogy (the google lan appliance comes to mind) or will it *gasp* need to charge users?
Your mammas flamebait.
When are you planning your IPO?
that comes to mind when I think of a huge server farm like Google's: can you give a rough order of magnitude (# of zeros maybe) on what your electric bill is?
Thanks very much for Google. The more I use it the more I appreciate it.
Everyone will ask about bandwidth, incoming lines, etc. (All the network capacity and capability stuff). Here's something a little more off the beaten track:
What technologies help to support the Google server farm? What kind of automated monitoring and trouble reporting tools are in use? Are they home brew, open-source, or COTS with some customization (scripts, etc)? And if you had to point to one area of network management and say "we could use some improvement or some better tools", what would that area be?
BTW - Google Rocks! I never use anything else anymore!
-- Mal: "Well they tell you: never hit a man with a closed fist. But it is, on occasion, hilarious."
Well, does he? No offense Larry, I am just wondering.
Lasers Controlled Games!
They have already answered that one...
;)
Like everyone else here, kudos for a truly useful and fun utility.
Google seems to be a classic case of fast growth. What have you been doing to try to maintain Google's unique culture as you grow? In particular, as you add more services to Google and the interface becomes more complex, so too will your internal organization. Will a big Google become a Dilbert-like Google-plex?
Are you guys making enough money?
:)
I wish you'd give us some banner ads or something, I feel guilty. I don't want Google to go away.
Seriously, why don't you serve banner ads?
-Dan
There has been much debate about what the practical purpose for Google Voice search might be, could you fill us in? Is it really for use in cars?
In fact, there is an opposite concern. Whether through a network of links or through coordinated googlebombing [googlebombing.com], weblogs frequently show up near the top due to the nature of reciprocal linking between the blogs. Not saying that's good or bad (sometimes a sole voice is a better expert on a topic than CNN), but it is what it is. Ranking "links" seems valid enough, but then you ask if that includes machine-generated links by someone's aggregator and the issue becomes a little more cloudy.
Ahoy love the google, it's the only engine I trust these days. Nevertheless....
For a site where speed and information delivery are of the utmost importance, and archaic table-based design seems rather strange. Is there any reason you have yet switched to a more forwards-compatible xhtml/css design? (Note that by "design" I mean more the html and css than the visual appearance of it)
For my own amusement, I've been looking at recoding the google design in CSS, and it's really not that hard.
Thanks!
Karma: T-rexcellent.
Why can't I find a cache'd version of this page anywhere? if the live cache'd version goes down, and there's no cache'd cache'd version, whatever will we do? :)
dmarien
will Google be able to compete with real associative search engines like LuMriX ?
Question (1)
What do you think are the solid foundations in the management of Google comparing it to other companies.
Question (2)
Is thinking Google to spin-off the Software Engineering and Scientific know-how to other market sectors different than the "Search business"? (like IDEO did with the creativity business)
Thank You
swain
I heard somebody say (scroll up if you missed it) that if google goes down, the internet (the www at least) is crippled. I know for sure that google is on the top of my hit charts On the other hand, there's a lot of talking going on about the size of google's serverfarm, the hits they get, the bandwidth blabla, and that got me wondering about security.
:)
What is google doing to keep it's pigeons save, do the have skyscraper high packet filtering beasts at their border, do they have a server farm just for that? What about software, what are they running? Are they using a flock of pc's with linux and iptables, are they using anything commercial like pix, raptor, fw1?? How about nids, is it even possible to sniff that kind of traffic?
I imagine you don't like things to get in the way of performance, so what are you using, something custom?
Or maybe you're betting on security through obscurity and therefore not answering
Where is Osma Bin Laden, or is that more of an AskJeeves kinda question?
Your mammas flamebait.
AOL recently switch to google for their srvice. How much pressure does that put on you to perform better?
Hello Mr. Silverstein,
First off, I have always been a huge fan and user of Google. I come from a SysAdmin background, and as part of a team at one of my old employers, our job was to maintain the 99.9% uptime of a large Linux clustered environment (1200+ servers). These machines were spanned across multiple datacenters in various geographic locations in an effort to increase efficiency, performance, and redundancy. Using network monitoring applications, we had the ability to receive pager notifications whenever an issue occurred (which at times would be often and at awkward times... when it rains, it pours!).
Considering that this is a very high level description, how do Google's methods differ? Also, could you please briefly explain the motives behind the recent inception of the Google Search Appliance? I think it is an incredible system (it took my old employer a long time, a lot of effort, and tons of man hours to create a decent search capability into its custom built intranet site!).
Hopefully this message will be selected as one of the ten to be forwarded on to you, and so I thank you in advance for your time!
jz
Domains for only $8.75/year! Transfer your domain for on
======= To: Mr. Google From: Me Re: Google's Competition ======= Most people are visually stimulated and are able to better grasp concepts and relationships when complex queries are organized in visual representaions... How does the Google tech department feel about up and coming sites like Kartoo (France) that utilze a new visual (Flash) layout that shows correlation between the returned web sites? and does google have any plans to implement such a scheme into the current searching method?
Top 3 favourite *NIX/*BSD operating systems ?
Top 3 least bad M$ OSes ?
and top 3 OSes that don't fit in the first category ?
--
Why haven't you implemented yet the toolbar for open source browsers? Are there technical difficulties or rather lack of interest from Google?
At Google, I get the impression you don't buy solutions, but rather you are forced to (always?) create your own solutions. While this seems like more fun than most of us get to have at work, I'm curious... In your time at Google, which single technology or innovation that you implemented failed the worst? Tell us a story.
What type of file system do you use, and how large (in bytes), is the index, it must be quite large with the cached pages and image search.
Your mammas flamebait.
Curious as to the power and air-con requirements for your server farm ?. Any chance of a real-time sensors for the farm such as a plot of farm size and page requests verses network bandwidth verses power and inside/outside temperature and humidity ? Could be handy reference site for some large scale modeling for others. The Google farm must draw a fair amount of juice and take a few tons of cooling to stop it from glowing !.
Can we get some basic stats with the interview? I mean, we all know that Google gets a lot of traffic but how many hits per day/hour/minute? How big is the server farm? How much bandwidth are they eating? How about some other interesting stats? (I'm sure they have plenty!)
dbc
Can you tell us anything about how you are working
with the various intelligence agencies to provide
them information about seraches that are of interest to them?
Are you thinking of providing SSL access to your
web pages so that these agencies will have to work
with you instead just monitoring your network
traffic?
I'd rather pay a fee to Google and know that it will be there in the future, than see it go the way of Infoseek et al. Is Google's immediate future secure and do you see a subscription model on the horizon, either through necessity or profit ?
Never, ever lose a file again. Ever.
What are the specs of your home PC(s), and what games, (if any) do you play regularly? What OS, and general programs to you prefer to run?
To know how decisions are made at google? How many people are involved in the development process and how many are involved in the hardware end of things. I look around at our current infrastructure and think that I need about five more hands. Our product development team seems to think that a project slipping six months just means they get more vacation time. How do you guys keep things on track and running?
Frustration is the name of the game
How/why in the world was a Google search in Klingon developed!?!
How does Google benchmark software? Eg how do you benchmark Apache, SQL, your CGIs etc...
Luck favors the prepared, darling.
Are you ever tempted to drop repellant web sites from your indexes? If so, why? If not, why not?
-aiabx
Just this guy, you know?
There have been some questions lately over the legal /copyright issues surrounding internet archives. In a discussion of the topic here on slashdot, others pointed out the implications of your past, possibly as a child/minor, coming back to haunt you in the form of your newsgroup archives. What, if any, are your thoughts on these topics; and have you faced any interesting legal/moral dilemnas in those areas?
put the what in the where?
From one interview...
Jason: What led to Google's decision to use Linux? When did that start?
Sergey: Well, Larry Page and I were in the Stanford PhD program in Computer Science. And we developed Google there. The way the computer science program worked is there was a hodgepodge of computer equipment lying around, and we would grab whatever scraps we could. We had all kinds of computers: HPs, Suns, Alphas and Intel's running Linux. So, we gained a lot of experience with all of those platforms.
When we started Google, we had to make the decision of what we wanted to use. Of course we chose Linux, because it is the most cost effective solution.
PCs are not only much cheaper these days, but we can also get them very quickly, because they're such a commodity item. That's an incredible benefit. We just installed another 1,000 computers and we got that done in a few weeks. That's really hard to do with any other kind of workstation. I think that's an advantage that people don't entirely realize.
Jason: Did you view it as being better, or was cost the main reason?
Sergey: It was better in some ways. Certainly for our purposes, we felt the support was better. For example, the actual kernel authors will respond to problems pretty quickly. They are especially responsive to Google nowadays, since we're so widely used. We can have a 15 minute turnaround. You can't really beat that for support.
That was an important factor, but frankly, the cost was a bigger issue. PCs are so cheap, which is very important. Sun's Solaris is probably more stable than Linux on PCs. It's hard to determine the blame, whether it's the hardware or the operating system. But, it's a minor difference.
Jason: Then, does all of your support come from newsgroups or do you actually pay for it through Red Hat?
Sergey: We have an operations team of about ten people, which helps a lot. And other than that we check newsgroups and e-mail the authors of the code. Usually, if it's a problem we can't figure out, we go straight to the authors.
Jason: Is Linux used on desktops at Google?
Sergey: It depends. Engineering mostly runs Linux. Business development/marketing runs Windows. Actually, I use Linux with VMWare running Windows. Some people have two computers, particularly some people in engineering who do UI development and need to test things out on Windows platforms. I find it better to just use VmWare and have one computer.
Jason: In a technical sense, what does Linux lack? What does it not provide?
Sergey: The 64-bit file system, which I know they are working on. It's slowly coming around. I think there are still occasionally some stability issues. I'm not saying Linux is unique in that respect, but you definitely want to have reliability. There are some issues dealing with higher memory systems. If you get to 2GB, and you try to push it past that, we encounter various problems. I know we've had some trouble with the network stack when we really push it hard. In terms of having lost most connections from lots of different machines.
And from another...
How is Linux used at the Google Projects? Why was Linux choose to improve Google search engine?
Sergey Brin: Actually, we currently run over 6,000 RedHat servers.
Linux is used everywhere...on the 6,000+ servers themselves, as well as desktop machines for all of our technical employees. We chose Linux because if offers us the price for performance ratio. It's so nice to be able to customize any part of the operating system that we like, at anytime. We have a large degree of in-house Linux expertise, too.
Most of our administrative tools were developed in-house, as well.
I don't remember what HTTPd they're running but it sure as hell isn't apache. Someone said that they get 1k hits per SECOND; what do you use to shape that insane amount of traffic? What is the '/search' page coded in? What databases are used to index a terabyte of data? How do those 10,000 nodes find the data they need to quickly? what sort of interlinks are used?
;)
How to you build a cluster like a war machine, in other words?
With Google being a key tool to the usefullness of the internet as other people posted it could be viewed as a weak link. How prepared is Google for events such as massive DOS attacks, or other hack attempts?. If Google were to go down, for any reason, the usefullness of the internet as an information locating tool would be vastly decreased.
From what I've read about Google, it seems like the same server farm nodes spend time on both searching (crawling, indexing, and storage) and on queries (web searches from customers). Is this really the case? If so, is ad delivery another part of this system being carried out by all nodes? Put another way, how homogeneous are your server nodes: do they all do an equal share of searching, responding to web requests, and participating in the ad system? If server farm nodes are not as homogeneous as I'm thinking, then how are the different functional aspects of Google's service broken down -- crawling, indexing, storage, queries, ads, and any administrative services you need internally -- and how much of your resources are being thrown particularly at the ad serving aspects of your site? Do you have some machines focusing on ads, or is that folded in with search queries, and that in turn is folded in with the actual business of searching? No matter how it's broken down, I can imagine that it must be fiendishly complicated, and I'm continually impressed at how smoothly you manage to make it work.
From a business standpoint, how happy is Google with the ad strategy being used? Is it producing a significant portion of your revenues, or are you getting much more from the search services & hardware solutions you're providing to paying clients? How flexible is the current ad delivery system? I.e. if you're selling keyword matching to ad customers to a system distributed across thousands of servers, and promising those customers that they'll get, say, 100,000 page views, how is this work synchronized across the servers doing search queries? It seems like this could all quickly get in the way of the search services Google is really trying to offer, but it's hard to imagine if it would help more to do it all "inline" with the rest of the site (but possibly slowing everything down fractionally) or breaking it off into a separate system (but adding more internal network traffic, potentially making it harder to do up-to-date reporting, etc). More broadly, what tools are available to you internally for monitoring your overall quality of service? Do these systems co-exist with the rest of the site, or are they also broken away -- and again, if they are, how to you keep reporting information current enough to be useful?
Basically, I'm curious about the infrastructure, both from a technical and a business perspective. There have been a number of papers & articles over the past couple of years documenting how Google maintains it's server farm for delivering search services to users; I'd like to know more about what's going on at the back to keep that forward-facing system running so well.
Anyone think it's funny that the new google applicance banners coincide with this 10 question thing????????
Are there plans to index audio files (and the audio tracks of video files) so that these could be searched as well? I would guess that existing speech recognition packages could be reused for this purpose so that development would not be too complicated.
Recognizing text in images and videos and indexing that would be a similar task. I know that Google Catalog Search must be doing some OCR already, but I have no idea if this would take too many CPU cycles if applied to all images, or if there are other problems (the images themselves already get downloaded for the image search, so bandwidth should not be the problem).
Oh great. Now you've gone and slashdotted Google. Way to go.
I have google as my home page. I have the IE toolbar on my windows machine and the Galeon eqiv on my Linux box. In short - Google is normaly my first port of call.
Despite this I've always made a consious effort to keep a backup search engine that I try if I ever find myself using more than a "few" of the results from Google. I find that if there are, say, 10 good pages from google there will be at least 3 that AllTheWeb (for example) will find that Google didn't have in the top 50.
I've decided that its very dangerous to use one SE to the exclusion of all else because there will always be holes, index bias, algorithm oddities etc that hide some of the info you wanted. As long as your other SE isn't simply playing follow-the-leader you'll benefit from the little effort of using a second source.
The idea is old of course, any researcher will tell you that having one source is poor methodology.
Does Google recognise this idea and is there anything you plan to do about it?
A Super-advanced search that allowed you to alter the weightings of link importance, document age, the domain it sits on etc etc? That'd surely be too much work for the average Joe.
I don't have an answer or I'd be sat coding it up but I do find searches that simply work better elsewhere. Maybe if you had 3 algs with a 70%, 20%, 10% mix in the results we'd all be a little richer.
Discuss...
0daymeme.com: Great stuff.
What do measurements do you use to measure the success of Google? According to the May 2002 Media Metrics Report, Google is #3 in search behind MSN and Yahoo. (Although the difference between the three is probably within the margin of error). And going forward do you expect to gain more ground through technical solutions, or though more traditional marketing solutions?
Google's PageRank technology works very well on the web with lots of pages pointing to lots of other pages.
The Google Search Appliance, however, is targeted at an office environment. Most of the documents (especially the non-html ones) in the typical office stand alone and do not have links to each other.
How has Google modified or complimented (if at all) the PageRank algorithm to make it more suited to an office environment?
I am currently pushing management at my site to purchase a Google Search Appliance, so I need an answer to this to help justify the change from our existing search application. i.e. without a good PageRank score, how does the Search Appliance order the result set in a useful way?
Dealing in spamware is illegal in several U.S. states and European nations. By and large, spamware programs have no lawful use -- they are built to abuse open relays and proxies, fraudulently alter mail headers, and obfuscate spammed messages to make it harder for victims to track down the spammer. Spamware is not merely a "burglar's tool" useful for lawless action -- it is like a locksmithing kit specifically tailored to be excellent for burglary and no good for legitimate locksmithing, or a gun somehow built to be perfect for murder but nonfunctional for self defense.
Nevertheless, Google accepts ads for spamware -- as well as ads for other spamming services. Google today carries advertisements and thereby accepts sponsorship from dealers in network abuse. Given the real and present danger that spamming poses to the usefulness of the email facility, and the amount of time and money that today's Internet-using businesses and people spend defending themselves from this form of theft -- how can Google justify accepting this sponsorship?
First of all, thank you so much for providing the most useful site on the net.
I understand part of the success of Google has to do with the efficient use of open source/free software. How about in-house software development?. Do you folks develop open source software as a way of giving back to the community ?. What are your thoughts on free software ?
How come *you* still have to answer them.
Cheers,
Roshan
I'm interested in the stat that you have in your .sig. Where did that stat come from, how was is calculated (area or $ value) and why did you find it relevant to put in your sig? It seems like a very interesting topic for discussion.
I don't want free as in beer. I just want free beer.
A big part of Google's strength is in the supported search syntax, most notably that you can search for phrases instead of just keywords, that you can filter OUT certain phrases or keywords, and that you can search for content on specific sites, or NOT on specific sites. The next step for me and probably a lot of other Unix/Perl types is regular expression support.
For example, let's say I'm looking for 80's brat pack member Anthony Michael Hall (not that I would do such a think), but I can't remember his middle name. Looking for "Anthony Hall" will do me little if any good, but looking for "Anthony \w+ Hall" could do the trick nicely.
Another example is that the user can provide their own limited fuzzy searching, by searching for optional prefixes and suffixes along with the root, instead of having to get the word or phrase exactly as it's indexed.
Thanks,
John
I am interested in the beta news searching function that is currently available on Google. My assumption is that you are exploring this because it takes several days to crawl the entire web so the normal indices can be out of date. And likely many people are looking for a particular story while it is still "news." I'm wondering if you see any long term, more general purpose ways of keeping your indices up to date. For example, do you see any tricks with looking at modification dates of pages or diffing the contents against your cache? Maybe you are already doing this to some extent?
Never underestimate the power of fiber.
What is Google doing to keep itself on top? Do you think there is a lot of room for improvement? How do you think web searching can get better?
So what exactly happens or have to happen from when I get my website online, to where i can see it on google? DO i need to submit it to google to have it appear or will it find its way there eventually and HOW??
So I'm truly surprised no one has asked this one yet, as it's the first thing that popped into my head...
/. story gets posted and people go stampeding to google to find out more? Or is that happening right now? (I'd hate to think of myself as part of a huge herd of individually acting DDOS'ers, but unfortunately, that's about what it ends up being...)
The masses of Slashdotters have slashed and dotted many an unlucky website over the years...Pushing webservers to their limit and often breaking them outright...
With Google's Massive resources, Is there any noticeable difference when a
Sig currently under construction. Mind the gap....
sPh
How do you operate a network of the size that Google uses? What operating tools and software do you utilize to manage your servers. Do you use any special techniques for gathering network usage (e.g MRTG vs. other data warehouse)? Lastly, what's the biggest problem you've faced from an operational perspective?
Thanks,
a cowardly systems and network administrator
How do you deal with hackers? I would ask if anyone has ever hacked you, but the answer is pretty obvious. Do you sick the feds on them? Do you deal with them in your own "internet"-style revenge? Do you care at all?
Since Cyc has done a stint with Lycos, what about with the Google engine? Especially since Google maintains one of the largest and most relevant databases, a single question asked may result in huge amounts of additional, relevant information flowing into Cyc?
everyone asking about hardware and to be honest its not what makes google good
after all thewayback machine does kind of the same thing
its software
so this is my question
what browser do you use ?
regards
john '1.1alpha' jones
I applied for to Google Japan and did'nt receive a reply. I then posted a followup and still did'nt receive a reply. Surely, it's courteous to reject candidates as opposed to ignoring their applications.
Is Google really a great place to work in, or is there a lot of hype going on about what a great place it is to work in.
Please correct this misconception I have as from what i've read, it seems a fantastic environment.
Click here or here.
All The Web just announced that their index has surpassed yours in number of pages indexed. As a webmaster, I have also observed that not only does Googlebot fail to fully index my site, but its index of sites (mine as well as others) grows increasingly outdated. While I still worship Google, reality dictates that Teoma and All The Web are eroding the hegemony of Google. What technology upgrades to your basic search service does your team envision in order to keep Google on top?
Every major operating system has it's example of a major corporation which is perhaps the flagship company associated with the OS. e.g. Yahoo and FreeBSD, Earthlink/Ebay and Solaris, Adobe and OpenBSD, and Google and Linux.
So, having had to deal with Linux on a large scale, would you say that Linux was the right choice for what you are doing? With the benefit of heindsight, would you rather have gone with another operating system, such as FreeBSD, OpenBSD, Solaris, AIX, etc?
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
With all of these linking issues, think NPR controversy, has Google ever been sued for trawling a website for which the owners did'nt want indexing, assuming they don't know of disabling the robot.
I think I speak for everyone when I say I'd like to see this Linux cluster Google is running. Just a Matrix-esque shot of the wall(s) of rackmount servers would be enough to make me happy.
--
The Bailiwick - DESIGNHUB2005
Are there any plans to implement the generic searching of images/frames within movies? In general terms, it'd be nice to be able to look for images using something else other than keyword-matching the page it came from. Specifically, I'm thinking of the ability to say "find me more images similar to 'this' one" where 'this' is the url of an existing image.
Though LINUX is almost free, people must make a living.
What will happen when more and more application running on Linux will turn to people asking for money???
All of the technical questions are pretty much moot if google has no long-term future. Are you guys profitable yet? If not, does it look like your current business model will lead you to profitability before the wolves find their way to your door?
When will a "mozilla compatible" Google Toolbar be made available and what can we do to help with this project.
When will a "Konqueror compatible"
So is the Dept. of Justice(DOJ) or Home Land Security(HSL) talk to you? In partictular, are you(google.com) being asked to track certian 'security' queries(bombs, antrax, Command Taco, ect.) of your system?
You say things that offend me and I can deal with it. Can you?
Who pays for the bandwidth requited to server this many requests and how is the delivery organised? Does it come generally from one major server farm plugged into the backbone or is it distributed?
vi or emacs?
Petru
Not directly related to the server farm but nonetheless...
Google currently restricts searches to:
* 10 words or less
* No wildcards
* No stemming
* No boolean expressions to speak of
* No proximity operators.
While I can use AltaVista (or some other competing search engine that supports more complex queries) for Web search, there is no alternative for Usenet search.
It does not impress me that Google presents its list of relutls in less than a second if I have to spend several minutes composing different queries in an attempt to overcome the abovementioned limitations.
Why won't Google offer a more advanced query language as an option?
I use google all the time! But I was wondering if the
serach engine would be free forever. Does google have any plans to charge users to use their search engine in the near or far future? I hope it does not.
Craig, as everyone will attest, Google has become indespensible. As such a tool, virtually required for useful Internet work, I see Google as similar to a public utility. What do you see as the pros and cons of being viewed in this light?
I know many people have requested alt.tv.alias be carried by google groups, this has been going on for months, and no reply from google...?
for the inevitable straight answer to this question. I'll say it: you can remove your old posts yourself.
You head google's technology and people who make the technology possible. What's your philosophy of leading such a team?
Natural language processing has been advancing rapidly in the last 10 years due to new conceptualizations( statistical and connectionist approaches ). Do you see these advances making thier way into the mainstream IT market in the near future? Does Google have any plans to implement systems using these new technologies?
.nospam please )
-jjzeidner@yahoo.com.nospam ( take off the
Since you have a personal interest in efficiently clustering large data sets, I was wondering if Google did data mining research on submitted queries. For instance, looking at how many searches are submitted by one person in a short period of time might indicate whether or not they found what they were looking for. An interesting piece of data might be, on average, how many unsuccessful searches are submitted before a successful one is returned. Combining this with being able to categorize searches by topic might lend itself to making suggestions on search techniques for different types of searches.
What was the deal with the Dilbert logos? Were you just having fun, or were you actually looking for new logo submissions?
Has Google beeing contacted by Gov Agencies like FBI, CIA to search certain information? and has google found it?
With all the information google has, is there a plan for the future to process all that information to get some kind of results, or an intelligent robot that learn?
I read recently that you cache many of the more popular pages every 15 minutes, which was a surprise. Exactly how many pages are counted in this "popular" set, how do you decide when to move a page from the normal every 28 day rotation to this one, and what's the process for getting one of these pages (say from my server) cached on yours, indexed, page ranked and available across your whole server farm for searching.
-- Chatbear - http://www.chatbear.com -- Free messageboards, Highly Customisable
What's your uptime? How often do you crash? How often do you have major problems and/or what are your biggest problems?
While doing some scientific research I discovered that the the Google Seitgeist is a very interesting source of information for research in the area's of social and communication sciences (marketing, lifestyle, ...).
However, the available information and the explanation of used methodology is too limitid to make this information usefull scientifically.
This is a shame because the Seitgeist is just the tip of the iceberg. There must be an enormous amount of information available.
I know for sure that a few professors I know would have a field day if they were to be able to analyse all this data.
My question is: would it be possible to open all the available data to scientists for statistical analysis?
It doesn't even have to be free I think. Universities and research organisations pay a lot of money for survey's that result in datasets that are relativly small to the dataset available at Google.
I have heard that Google uses Python extensively to manage its data, grab new data, etc.
As an avid fan of the Python language, I am interested in exactly how Google puts it to use. Can you clue us in?
P.S. - Keep up the good work!
Your question netted 53,496 possible answers. I've filtered out similar answers, so let me just give you the first 1,000...
Says the RIAA: When you EQ, you're stealing bass!
Which distribution do you use on your server? How do you keep them updated?
We've all had servers crashing on us just before a deadline. We've all had to go to the office in the middle of the night to prevent a disaster. (we've all been hacked by a scipt-kid, once)
Do you have any stories of disasters or difficult moments in the datacenters that kept you all up for a few nights in a row, but went by unnoticed by the public?
if google is so linux heavy, where is the google toolbar for a linux browser?
Does Google have any plans to become a public company, so all of us devoted fans can start throwing our hard-earned money into your IPO?
What's the root password?
:)
Kickstart
You have young upstarts headed your way. But fostering competition can be good. What kind of advice would you give to a company like Teoma or Wisenut in order to keep up? What do you think gives you the edge, technologically, over these guys? What do you think they do well that Google does not?
Examples:
Pages can be manually removed from google for doing dubious things (e.g. cloaking). Someone must have actively decided that the benefits of this type of censorship exceeded the costs.
Someone decided to pull the anti-scientology pages, and then someone decided to put some of them back. Who?
Someone decided against making a list of removed sites so that we can all see what is being supressed by google.
Matt
Well...does he?
I've had root on ~13000 Akamai machines.
is that i use IE, but i must, for the google bar rocks my bitch ass. will it be available soon for any of the other browsers?
What's your biggest bottlekneck for speed?
Craig Silverstein's picture is
here and pictures of a number of other employees are here.
My mom always said, "Jim, you're 1 in a million." Given the current population, there are 7000 of me. God help us all!
They couldn't get it to work because the Beowulfs (beowulves?) had quite an appetite for squab!
What role does the programming language Python play within Google?
The Google image search is wonderful, but shouldn't that have been called Ogle?
Is there anything on the internet that you personally couldn't find with google and if so what was it?
p.s.
Thanks for all your help with my school research
How much is your data worth? Back it up now.
The Google toolbar is one of the coolest things about IE (maybe the only one <grin/>). However, you need a Windows system with IE in order to install and use it. Are there plans to have to toolbar available for Mozilla, and non-Microsoft systems in general?
She sat at the window watching the evening invade the avenue.
Can you talk a bit about how those weights have changed over time? Have there been any surprising shifts?
--- Jason Olshefsky
Karma: Poser (mostly affected by adding this line long after everyone else did)
Do you report searches that may be "against us" in the words of our idiotic commander-less-in-chief.
Examples might include:
plutonium bomb assembly dirty radioactive fake
staged Cheney Bush Rumsfeld Carlysle
Thanks in advance,
Prepared for the second fake war: The war on fat
Google rules. What I'd like to know is how it is *always* up, even in the face of huge demand and complex servers backing it up. I've used Google for a few years, and I've never seen it go down, even though I usually visit the site several times a day.
What are some of the measures that allow Google to achieve this incredible availability?
What do you worry most about about your job?
Do you Yahoo?
;-)
Hindsight is 20/20. What would you do different, technology wise, company wise, etc, etc... If you were to start again?
Craig Silverstein's picture is here and a number of other google employees are pictured here.
My mom always said, "Jim, you're 1 in a million." Given the current population, there are 7000 of me. God help us all!
As a network security analyst, I have to deal daily with intrusion attempts and DDoS, and such. Our company, being a large domain registrar, has to pu t forth a large number of resources dealing with these issues. What kind of intrusion attempt traffic does Google see on a daily basis? How much of Googles resources have to be used to deal with these problems?
-===- "Those who would sacrifice freedom for security deserver neither" -===-
to find those "against us" in the words of the idiot resident of The White House?
An example of a strange search is:
> dirty radioactive bomb staged fake Cheney Rumsfeld Bush Carlysle
Thank you
Why did you first decide to start building an internet search engine, and what have you learned in the process of becoming one of the biggest and best search engines out there?
...interesting if true.
has there been a noticeable increase in hit rate since google whacking became popular?
have you had to take counter measures?
also, you already have an elmer fudd language, how about duke nukem or yoda?
That man tried to kill mah Daddy
Considering the extremely low .com success rate and the current terrible advertising market, what do you think has set Google so apart from the many failing, similar businesses? Do you see your business model as having endured what may have been the most difficult economic environment, or do you see larger challenges in the future?
And, of course, I'd like to say I love google dearly and would like to pursue a physical relationship in the future.
Mod me down, and I will become more powerful than you can possibly imagine!
According to the Google site, in the career section one of the perks to working at Google is having a high powered linux workstation. My question is: How well does it work using linux as the Desktop/Workstation OS. What kind of compatibility issues have you run into when working with partners who use Windows. Additionally what kind of custom software soutions have been needed to make it work.
-ryan
I've been waiting a long time for an exact phrase search that really is what it claims to be.
If I want to search for the character string "/proc/net/dev" or "C++", then currently I'm totally screwed.
Any chances of us ever getting this?
Does this chain of thought keep you up at night?
Miko O'Sullivan
I noticed that at google has free gourmet lunches for all its employees couresy of Chef Charlie. My question is how good is the food and has Charlie told you any interesting stories from his days with the Grateful Dead?
I have been pondering this question for quite some time, and I think I finally found the one person who might give me the answer:
;-)
Dear Mr. Silverstein. If you could have everything, where would you put it?
This is a good question.
With all the effort that you must put into improving other people's experience of the web, what do you do to relax when you're not working?
As an addendum to this, what is it about the corporate culture at Google that makes it work so well while other "hip" dot coms went down the toilet? What's the magic ingredient that made Google turn out differently?
Got Rhinos?
There's something that I fear may happen someday, so I'll just ask it now -
When will we see the Google IRC archive?
Speaking of bookmarks, do you have any plans to offer a bookmarking service?
Miko O'Sullivan
What I really would like to know is what it is like working for Google in the post-dot.com era? A lot of tech companies these days don't treat their employees very well (Makes me I'm glad I'm not a software developer). Have you had to lay off people yet? Are you still hiring? How are people rewarded at Google? I am asking is because you can tell a lot about a company by the way they treet their engineers.About your competition Alltheweb, a friend of mine considering seeking employment there was warned by someone who already works there that he could not in good conscience recommend it since he described the culture as "The same as KGB during Stalin: hero one day, a bullet in the back of the head the next". Is this normal for the industry or is Alltheweb different?
RICK
...which would you be, a bactrian or a dromedary camel, and why?
What is google's business model?
In your opinion, if Google is overtaken as the premiere search engine, what would have been the cause ?
How exactly does the image search feature work? Apart from the filename, what makes an image more likely to be returned in the search results? Have you considered using OCR software to read text in i mages and an image as text option on the search page that will extract the text from an image?
perl -e 'print $i=pack(c5, (41*2), sqrt(7056), (unpack(c,H)-2), oct(115), 10);'
. . . and we'll post his answers shortly after he gets them back to us.
I want to see the answers shortly before he gets them back to you!
c'mon... we all want to know what percentage of searches are actually looking for porn.
I had heard from a friend that you guys are using some sort of dynamic VLAN's and that one day a server can be doing one job, and the next day will be dynamically assigned to another group of servers doing another job. Is this true? If so can you elaborate on this?
How has Google managed to deal so well with ethical issues in the current economic environment? For instance, how has Google managed to avoid going to pop up ads, unfair treatment of search returns based on payoffs, and other challenges? Most search engines and news sites have already caved in to these methods citing budget shortages. Google, on the other hand, seems to be expanding existing services, and acquiring or developing new services. How has Google managed to avoid some of the other pitfalls like clueless corporate officers who push the company into adopting bad technologies or technologies that don't fit the company? How much input and control does the non-management, but highly technical types in the company exert over the corporate vision of technology?
Just what impact has Jakob Nielsen had on Google's interface?
icqqm [ICQ:11952102]
While I think we all understand that you wont make major service announcements here, but what role do you think Location Based Services, (GPS) Telematics, and Wireless technology will play in Google's future? What major technological hurdles do you forsee? What privacy issues need to be resolved technologically vs politically?
People who think they know everything really piss off those of us that actually do.
From various articles and references on the net, it's clear that Google uses a mix of languages in developing and deploying it's services. Languages I've seen cited are Python, Java, and C++. I assume this is not a complete list.
What programming languages do Google developers use, for which tasks are they used, and why?
The real Webmaven is user ID 27463. I don't rate an imposter, because my ID is such a lame-ass high number.
It's common knowledge that people like to break systems run by companies that are universally disliked.. no need to name names on slashdot, but I don't know anyone that dislikes what google does - they seem to be held in pretty high regard all round.
We have people portscanning the place I work (education) every day, and code red continues to try and infect us (but we run mainly linux webservers), so I regard that sort of thing as background noise.
I wonder - does this make you relatively immune from these sort of things ? Do you find people trying to break or DOS google ?
Also, you are bold, but are you also...daring? One last question: Have you ever been in a Turkish prison?
do you have any plans to support wildcard searches? it's tedious to have to type (recipe OR recipes) all the time. how about regex pattern matching?
How do you guys handdle i18n? at my job we are currently working on that, and we just find it hard to keep the pages the same across different languages without replicating data or code that doesn't need to be replicated. We also think we will have issues with later releases as translations might not be done on time, do you guys have such a thing as different versions for different languages?
In particular:
1. Do you use any Topic Detection & Tracking techniques.
2. How do you cluster news stories? Do you use a Scatter/Gather approach.
3. Is the news site going to be available through an API?
Marcos
Your search results are undeniably the best available commercially on the web, but what thinking has been given to graphical information visualization? Some new search engines are trying out presenting results topologically. While these may not be very useful, there may be potential. What has google done technologically in this area if anything? Are there any plans to explore this avenue?
People who think they know everything really piss off those of us that actually do.
Hi Craig,
Great props towards one of my favourite sites, keep up the good work.
My question:
With the introduction of this service (and later searchable answers e.t.c), how long can we (and any aliens planning roadworks) expect to be able to access googles version of the HHGTG?
I realise the net is this to some extent already, but the Answers KB is pure signal.
Do you have any other plans for the sum of this collected knowledge than the web (mobile devices, schools e.t.c) and how well is it going?
Thanks,
Sy
I used to kill hours watching the search requests scroll by on Metacrawler's Metaspy page back when people still used Metacrawler. Any chance we could have something like that on Google? I would *almost* even pay to subscribe to a site where I could watch uncensored Google search requests go by.
I hope that this is not a redundate post - will mozilla ever get the famous google toolbar. (For the poor no galeon in reach win32 mozilla people...)
And what is he thinking about AllTheWeb?
Unfortunately, the people who post under anonymous coward appear to think that the only "hot" chicks out there appear in Maxim and that a woman who doesn't frequent raves and is older than 18 can't be beautiful.
Your comments would carry more weight if you would put your name behind them.
People in the know say that it is mostly down to
two search engines now: AllTheWeb and Google. Do you agree? I've also heard that while Google is better on searching the web for english content Alltheweb is built on radically more flexible and scalable technology that works better in other contexts as well.
According to what I've heard they have an index about the same size of yours, but they only use a fraction of the hardware you use. How many servers do you and AllTheWeb use?
Do you come under pressure from any government organisations to monitor users' queries?
With the success and popularity of Google, I find myself using URL's for places less and less and just entering names into Google to find places (they are almost always on the first page...) Do you think that you have almost replaced the URL?
When will it be available for Mozilla?
These two new features are very cool. What I wonder: How do you index images? And how do you get all those catalogs on the web? Your site says they are catalogs that aren't available elsewhere on the web, so did you enter them by hand? Or did you have some cool solution? Thanks! Hlynna
The one in the corner looking clueless at most everything, but enjoying it, nonetheless.
In a recent interview, Eric Schmidt mentions that, "At Google, we found it costs less money and it is more efficient to use DRAM as storage as opposed to hard disks". [http://www.pcworld.com/news/article/0,aid,81685,0 0.asp].There was a lot of interest about this remark. Could you elaborate a little more on this idea.
A lot of search engines truncate/prune low-scoring terms. For instance, do you really need to hold the scores of the word "the" for all the 2.5 billion documents that have it? Would it really make a big difference if you only held it for the top-scoring, say, 100 million documents?
I've noticed recently that hyphenated keyworded urls are placing highly in Google. Could you confirm that Google is looking strongly to the url for content information and to determine relevance?
Google does a good job of indexing html and has added new types of content over the last few months.
My question is what other contents types do you want/plan to index?
For example I searched Google for about an hour today looking for a solution to my PHP compilation problem with no luck. I turned to IRC and got an answer in a couple of minutes. If Google archived that conversation then the next person to search for the error message would get an exact hit.
/b
[Please type your sig here.]
How does Google solve the global load balancing problem? Foundry Networks claims that you use their ServerIron products. Is it true?
Do you do anything special to map different parts of the network to server farms? How do you collect performance metrics while avoiding active probe compaints?
how long is an old version of a page that no longer refers to a page kept in the cache?
How was that decision made?
Having access to gigabytes of data is always valuable for testing new algorithms and ideas. Would Google considering licensing access to their data at a level beyond the search capabilities available today?
Some of us ACs do have taste. Cindy's beautiful. A major babe.
I'm not very familiar with the Google toolbar, but IMHO, Google access from Mozilla 1.0 couldn't be much easier... just type your query in the address bar, then press up-arrow to select the 'Search Google for "fubar"' from the bottom of the drop-down menu, and hit Enter... presto! Google search in 2 keystrokes. Add Ctrl to that and you even get it in a new tab. Mozilla rocks! I just wish it had smooth auto-scroll, more customizable toolbars (such as small icons, optional text, Home on the nav bar instead the personal bar), and native support for Back/Forward buttons on mice, like IE and Opera 6 do.
"Mind, as manifested by the capacity to make choices, is to some extent present in every electron." -Freeman Dyson
Can you describe your backup solutions? And in that vein, has there ever been a time when a critical hard drive or system crashed, calling for an all-nighter? (What has been your worst nightmare?)
Craig,
When will images.google.com include PNG images in its search base? Why were the image types limited to GIF and JPEG, when most browsers could also display PNG? Now, virtually all non-text browsers support Portable Network Graphics.
Questions done. I'll take this opportunity to thank Google for groups.google.com, the searchable usenet archive. In my opinion, 15% of the total value of the internet is contained therein. Excellent!
- - -
"The sixth sick shiek's sixth sheep's sick."
What would be the one very cool feature you would love to expand Google with, that is simply too difficult to program (for now)?
Tom
Can I have a prize for slashdotting google?
Sometimes I wish that Google had additional advanced search capabilities such as regular expressions or an Altavista-like NEAR boolean (preferably one that lets me specify my own value for the 'nearness'). In fact I'd be prepared to pay for "advanced googler" status, even just to have those two available (say, $10/year at least).
Do you have any thoughts about the technical feasability of these ideas? And what about the business case; doesn't Google need just such a revenue stream from the people who actually use its service?
I was gunna mod... but I ahve a re-quest-ion.
Can you guys put up bandwidth graphs for the public to see. Like mrtg graphs page showing daily google request traffic flow. so we can see what type of overall trends in searching happens during the day.
I would love to be able to see just how massive you traffic is and what it looks like.
and let us know what tools you use to monitor all your stuff.
Are those pigeons being humanely treated?
I've been worrying about this for months now. It just doesn't seem right, making those pigeons do all the work.
I work for a large ISP and am interested in how large systems are managed. How do you manage the Google? Do you use open source, commercial or roll-your-own monitoring. Do you use a SNMP agent and, if so to what extent. How fast can you detect a problem, troubleshoot it and fix it.
- Things are the way they are because they're coded that way -
...it's how you use it! =)
That's a good question and not "redundant". Let me expand on his/her question:
Is Google actively hiring? If so, what kind of job titles are most important for Google to fill these days?
[I'm not a headhunter]
Chris
Hey Google seems to have all the keys to success. So that would be like looking for a needle in a
When you were selecting the OS to run Google, why did you choose Linux? I'm partial to FreeBSD but I'm pretty sure that you evaluated it and found something a) that you didn't like or b) something about Linux that you liked better. If so, what?
Second part of this question: Do you continue to evaluate alternative operating systems?
Chris
S.B. California, Oh, yeah!
Is it possible for Google (or the people at Google) to calculate a finite number of possible "Google Wacking" searches that result in
only one web site?
In other words, given the content of all the webpages that Google can search, is it possible to determine how many search requests will yield only one site as a result?
What does it take to even be considered for a position at google? ie. What sets one resume above the 10000/week submitted?
"Failure of Windows operating systems is extremely rare. If it happens, it is usually due to operating system file c
Cutting the BS out of the way... What makes google different from the many razor scooter riding, Latte drinking, "hi we all have PHDs" and look at us do something on the 'net' companies? Rob
try { println( SigString ); } catch( Exception e ) { println( 'Who cares?' ); }
Actually, economic inequity, and not lack of food, is the prevalent cause of hunger. Thus, in a way, making more people rich does create something wrong. Unless you consider it is ok that one sixth of the world's population faces chronic hunger. Like energy, there is only so much capital in the world...
I am saying ICANN (or someone like them) - not Venture Capitalists - should fund projects like Google.
In the staff section you say you have a neurosurgeon on payroll. I can't possibly imagine what job he/she has, except for operating on linux-machines with hart problems (yes, linux has a hart!).
just because i told you i didn't want a job that involved 24/7 oncall at times. sheesh. :)
ha ha ha lamenesss filter crapping...
hoo hooo hoo..... he he he
All bow to his Noodliness!! His Noodle Appendage has touched me!
Why is this page currently PR0?
Basically, what I'd like to know is how you see the implications of Privacy, personal and corporate, in terms of technological choices you need to make?
For example, do you see us moving like the rest of the civilized world towards a more personal-privacy oriented legal structure, and if this occurs, how would you and other search engines deal with this? Do you see any areas that Europe and Canada have already legislated that would prove very difficult for you (e.g. retention of server logs, query logs (comparitive to HIPAA regs), user logs) - or easier?
Do you think we'll move (not counting the current paranoid few months) towards a more totalitarian regime - and what implications does that have for your tech side? Would you need to mirror search queries to help hunt down Americans posting things disparaging the White House or Congress - for example.
--- Will in Seattle - What are you doing to fight the War?
The ultra lean user interface is a key factor for Googles success. Now that Google grows and more and more people are working there, how do you deal with the feature creep of "can we add this here and that there" ?
Some new features are great, but how do you draw the line to keep that lean interface ?
You don't understand how Google works at all, and you've created a completely useless hypothetical situation. Submitting that question to Google is a waste of time.
Google doesn't rank "sites", it ranks "pages" (URIs), and it doesn't rank them according to "hits". Google ranks a page according to the number (and type) of links to the page, combined with analysis of the page content.
In fact, topic-based pages often beat "news" pages in Google results because they're more stable. Pages that stick around longer generally acquire more incomig links.
Proud to be / Smiley-free / Since Nineteen / Ninety-Three
I was amazed to find out that the new site for Elizabeth Smart was crawled by Google (and ranked first) within only a few days from the kidnapping (and the site being registered). CNN pages are also routinely showing up in Google within hours of being published.
My question is: how do you estimate the rate of change for each site and how often do you crawl frequently updated sites (and update the index)? What is the range of re-crawl (few hours to one month?).
thanks a lot,
alex
PS: Congrats for the Webbys.
Hello Craig,
Google might provide better results if the web is more structured right? Is google involved in developing any standards for authoring web pages (e.g., RDF, WebOnt etc.)? How about a GUI for assembling a complex query (" I need a ticket from Boston to Atlanta for 100$ etc.". Kind of stuff which semantic web claims to do?
And also is google willing to give the data that have been collected from the past years for academic research purposes?
thanks
rajesh
A little more than a year ago, Dr. Dobb's Technet cast ran a presentation given by a PR manager at google. He talked about how google had developed in-house driver modules for linux to pump up disk speed. What other modifications or extensions have the google programmers made to the linux kernel? Any chance of releasing this under open source?
It's also a one-click install option when you install Galeon on Debian. I suspect it's a pretty universal Galeon thing.
Behind google has to stand an enormous mass of personnel and machines and bandwidth et al
Yet the major business model is the one of a few sponsored links (I think I haven't seen one in the last 1k searches or so. So I must google for pretty useless stuff, eh?).
How much money do you make per search, overall? And how does that fit with the averaged cost/search?
I know that some portion of your income flows in from appliances and stuff, but I guess the sheer volume of searches makes all the difference for you.
cheers.
I've always wondered what Linux distribution the Google machines use.
Is it a completely custom one build from the ground up by Google staff? Is it a customization of one of the well known distros (in that case which one?)? or is it just an off-the-shelves standard Linux distro that just runs the Google specific apps (again, which one)?
Now that Google is a BIG player with high visibility, what kind of pressures are being applied from Sun to M$ on buying thier tech support/ebusiness solution?
My vote goes to keep independant, use you own tech genius, and stand tall (you don't need big help to run big business).
what makes you think that lots of links to a site qualifies that site to be in the upper ranks of your query results? Is there any scientific work about this issue, or is it a design by intuition/common sense? who can check out the quality website ranking 123,457 with your current design?
Any plans to release a Google Toolbar for Mozilla 1.0?
How much of the info on the Internet is copied or repeated from other web pages vs. how much is original? A link to info does not count as copy or repetition.
How much of what appears on the Internet is crap (using the movie "Battlefield Earth" as the "10" on a scale of 1 to 10). I already have an estimate of this, and things ain't lookin' good.
The Russians have won. They have made the world a cesspool of distrust, greed, fear and hate.
When is Google going to go public?
Can I marry google? :)
Are you hiring?
Your sig sucks. Why not just use latitue and longitude? Retard.
I have noticed a high incidence of hyphenated urls in the top positions on Google. Could you confirm that you are weighting the url heavily in determining content and relevance? Additionally, and I know we're only supposed to ask one question, but this is important: Your Adwords programme seems to need some finetuning. Some terms show much higher searches on Overture, while other common queries show no results at all. Are you aware of this? And if so, do you have plans to finetune this tool in the future?
you're lucky she works for Google, she'll need it to find your ****
Google also tends to succeed where others have failed--take for instance Google Groups, formerly DejaNews. What motivated that purchase? Google also seemed very interested in augmenting the USENET archive with missing data, by hunting down CDs and other media that were published years before DejaNews started its archive--that seems like a genuine desire to preserve USENET for the ages, so what inspired that? Lastly as a corollary, Google Groups is missing one feature that DejaNews used to implement, but eliminated a year or so before it went down: Deja had been keeping archives of the text posts in alt.binaries.* groups, which can be valuable since many groups have active text discussion; will Google ever re-introduce the ability to search the text messages in the alt.binaries.* hierarchy that Deja used to offer, even if it's limited to the old archive Deja had?
Chasing Amy
(We all chase Amy...)
"The more corrupt the state, the more numerous the laws"-Tacitus
How many physical sites do you use to host your systems? And is this due to network redundancy issues, disaster management issues, or simply realestate issues? If they're all in one site, is it because you feel things are easier to manage that way, or is it a limitation some crazy developer didn't think of?
What I'm getting at is I'd love to work for google, and actually like some of the current job postings, but I don't want to move to California. (don't get me started on the reasons) If Gogle had sites in other locations, wouldn't it make sense to hire local admins to go deal with situations there? And thus the concept of the Google branch office is born...
"We are not tolerant people. We prefer drastically effective solutions"
Google, along with other search engines, filters content for explicit content. Is there any other type of content that Google has considered, or currently filters as a matter of practice, and what led to this decision?
What is the airspeed velocity of a a PigeonRank pigeon?
What is the airspeed velocity of a PigeonRank pigeon?
You must use that bandwidth at work for "personal" use at least once (a day..heh). What kind of ping do you get in Counter-Strike? Come on...we know you play it!
Hard work usually pays off over time, but procrastination pays off now.
...in their servers? And if so, what has been their failure rate?
Did you know the mail software you use is open to denial of service attacks(hint ident, no ident)?
Why hasn't this been fixed?
What precautions do you take in reguards to security of your databases? What processes do you use to avoid users of your service being 0wnz0r3d(eg preventing js attacks appearing in search results)?
I saw the keyboard mode google at google labs. Will there ever be a version with vi/emacs style key bindings?
boxers or briefs?
Comment removed based on user account deletion
I'm disseminating just THINKING about her....
Could you tell us a little about the back end of your search eangine. It's extremely fast. What database software do you use? What optimisations have you implemented at the Operating System level (cluster sizes, Raw IO...)? What type of hardware do you use (Disk drives, Raid, CPU(s)...)? How do you handle load balancing/redundancy?
Comment removed based on user account deletion
So, my question is: Would google look to splitting its services in this manner? That is, the first tier method where you get either
1: Free services (basic services)
2: Subscription services (advanced paid for services)
or perhaps the 'request for donations' line where the site is free, but donations are welcome?
You have a sick, twisted mind. Please subscribe me to your newsletter.
A/S/L?
Why do you tell everyone that google is running on linux, when it is really running on Win2k?
Don't you think that it's time to put your childish bias behind you and admit to the truth.
It's quite easy to get around the "argument list too long" error
this linux journal article covers it quite nicely.
I'm interested to know whether Google has ever been sucessfully cracked? (errrm, perhaps this is sensitive info)
How often do you detect cracking attempts?
If this occurs what do you do about it?
Thanks
* * Always question "the National Interest" - 9 times out of 10 it is a cover for evil
You forgot the most important question.
Are there any plans to make a profit? Yeh I know paying divendends are unpopular with tax accounts, but VCs eventually do eventually say no. So I assume one day Google will have to make enough money to at least pay for its self.
Afterall logging onto Google doesn't seem stimulate half a dozen porn 'n casino pages popping up, so that arn't getting their fractions of a cent that way.
How does Google do so much so smoothly?
It would seem that google's advertising syndication efforts have been focused on only a few large web properties. This is ironic considering its all of the little small sites who spread word or mouth that grew google to where they are today. Overture and others use thousands of smaller affiliates while google just uses a few large sites for ad syndication. Are there any major technical or practical considerations that prevent cobranded solutions such as Earthlink uses being offered to loads of smaller sites? I am sure there are thousands of small sites that would link to google instead of PPC search sponsors if only header and footers were allowed for our ads. Add to that a revenue share and the best search results in the world and google could grow even faster than it has in terms of reach and ad revenue. So why no revenue programs for smaller sites at the present is there a technical reason? Is this being looked into technically at the present?
I HAVENT seen Google not working till now - how do you ensure this kind of redundancy?
1. What is your favourite colour?
2. Do you ride horses?
3. Do you sleep in the nude?
How about giving a server?/Rackmount thing to the Wayback machine, so we can use the power of Google to navagate into the past! (their search sucks) Come to think of it, so does /.'s!
There has been an open posting for a Director of Technology at Google for a few months now. Are you returning to the world of research or is HR confused?
Why not support regular expressions in a web query?
The staff of google seems to have a nice lightheared stance on things, and example being the recent April Fools "technology release", as well as repeated alterations to the google banner. Do you feel actions such as these help to lighten the spirts of a team of people who much be under great pressure to deliver.
maybe this isn't the right guy to ask this question, but i've always wondered, was something like AdWords planned from the start or did you guys have to throw them up in a hurry because you found yourself low on cash?
also, are there more plans for parts of Google like this that are strictly to make money or is most of the new coding focused on projects like what can be found on the labs page?
is there ever any tension deciding what to focus on, things that make money or things that are useful or cool? do the engineers have any input or are these all management decisions?
I wonder how cool ideas at Google fare against the need to get into some serious code optimization, scaling problems, etc. By "cool idea" I mean something like the page rank algorithm. Does Google spend a lot of time coming up with more cool ideas, or do you already have plenty of ideas, and you need to invest more in developing the existing ideas? How many developers are there for every ideas person -- 1 to 1, 10 to 1, 100 to 1 ??
And yes, IE6 is WinXP, IE5 is Win98 SE, IE4 is Win98. I think Win2k shipped with IE5 and WinME shipped with 5.5 but I could be mistaken.
Bleh!
Thank you for your time. Thank you for providing a great search engine?
Democracy Now! - your daily, uncensored, corporate-free
Do you have / plan to have any way of using canonical URLs to get more accurate page hits/listings? If that sounds obscure, let me give an example from my own experience, which I'm sure is not the only one. /usa/-0400/ if you're using EDT. So people can link to any given page using up to 10x28=280 URLs, but in most cases the page doesn't vary whatever URL they choose, and there are never more than 4 actual variant pages coming out of all those URLs. Weird, but it works, and I'm sure that quite a few other sites adopt the same sort of strategy.
To avoid the technical support problems that come from using cookies, we use URLs for customisation: (in our case, 10 regions and 28 timezones) - thus you might use
Trouble is, Googlebot sees 280 possible URLs and sedulously visits them all: this wastes the robot's time, distorts page rankings, and clutters up Google's search results.
I don't know of any way to tell Googlebot that "all these linked URLs are variants of one canonical URL, so use that instead" - I can imagine that simple rewrite rules could be put in robots.txt, or we could use "303 See Other" or something... Is there a way? Is it documented? Ought to be a way? (that can't be subverted by evil webmasters trying to subvert the page ranking system).
Do you have any pigeons working on this?
man, she looks like david spade.. that aint beauty..
Just curious, but which part of the toolbar do you use that isn't already supported in Moz?
:)
I only ever used the google toolbar for searching - ie. never any of the other fluff, whatever it was
Moz lets you search google (or other sites or your choice) directly from the url bar so I no longer have the need for the google toolbar.
Cheers,
Roger
Do you have any better hostages?
What and When are you planning to to do about the frauds being perpetuated on Google Answers (http://answers.google.com) with questioners paying their own alias instead of the expert poster who deserves to be paid. They are destroying the cedibility of what has the potential to be an absolutely brilliant service.
I think you need to be a much more strigent about suspending and banning these abusers if you wish this service to take off.
It's called Netscape 4.7.
... more so.
I think the real question is whether Google could singlehandedly eliminate Netscape 4.7 by switching. They'd be the savoir of webmasters... er
PageRank was the innovation that made Google's results better than the rest. This, along with the quick loading, uncluttered interface was why people switched to Google.
When is Google going to move from an 'importance' method to a 'theme' method; boosting pages that are widely cited from pages relevant to the query, not just widely cited pages?
(That's one question, of course, only it has four question marks :-)
Google provide data both directly and indirectly to the searching public through the Google sites and through partnerships with other sites. The recent launch of the API beta program suggests a move towards encouraging results to be delivered by others. Syndication of Adwords lessens reliance on the Google site for income from that stream.
By ceasing to provide a search function directly Google would benefit from removing the onerous responsibility of providing customer service to millions of search users, savings in hardware and bandwidth and remove the situation where you are in competition with paying customers for providing search results.
How core to Google's business is the provision of public access to the Google database through the www.google.tld sites?
There is the Google Answers Service (answers.google.com) only problem is it riddled with frauds.
There has been discussions about the legality of caching material that is copyrighted, do you have any thoughts of this?
Have you considered any technical solutions to this problem?
eg: a cache.txt file that tells robots if their allowed to cache a site as opposed to robots.txt that tells if they're allowed to index the site. Or perhaps the other way around that people can have a cache.txt that tells robots that they are allowed to cache a site. This would let people opt-in instead of opt-out and would also eliminate a lot of unnessesary pages. (I wouldn't mind if you cached my homepage but I won't expect anyone to be interested in the cached copy)
- We are the slashdot. Resistance is futile. Prepare to be moderated -
Particularlly the scalability issues mostly have been resolved -if not in the 2.4.18 kernel then in the 2.5 dev series.
Anthony
At the rate the web has been growing, how long do you think it will be before Google is actually indexing 1.0*10100 pages at a time? How long would it take, under the current setup, to perform a string search of that much web space? How do you plan on growing the tech so that such hefty tasks won't daunt the pigeon flock (for that matter, how many pigeons will it take to search through an index containing 1.0*10100 items?)
This begs the question... how do they cope with hardware failures? Even using the wildly exaggerated MTBF figures published by manufacturers, that's a significant number of failures *every* day. Does Google have dedicated hardware techs running round replacing broken drives, fried memory and faulty power supplies?
"The invisible and the non-existent look very much alike." -- Delos B. McKown
How many servers would you need to have if you only had one user? (That is, how many of the servers are there for faster searches, and how many are there for handling all the users?)
http://science.slashdot.org/comments.pl?sid=34509& cid=3742297
What percentage of them are CS Ph.D.s who focused on something that relates directly to what Google does? Do they get paid significantly more than the average senior technical Google employee? Are they treated differently (as they often are in biotech), or are they mixed in with the rest of the folks? Do you feel that having so many Ph.D.s it is a competitive advantage so you sought them out, or is it just something that happened?
(It sounds like I'm asking because I'm considering getting a Ph.D., but I'm not. I'm just curious.)
When was the last time Google was taken offline (for upgrades, to fix a problem, etc.)?
This deserves more than a 3. PNG really is the lossless image format of the future, and needs adoption on as many fronts as possible.
Google's image archive is fantastic, but it seems to only archive PNG and JPG files. Given Google's general trailblazing attitude on algorithms and technology, it would seem appropriate that more media would make it into the archive than just these legacy formats, like PNG files.
Google is crawling Word docs and PDF files nowadays...what other media types are in store for the future?
While I admit I am a native english speaker (which means I get to trash the language without appearing completely stupid and/or uneducated) I read several European (live and dead) languages passably.
Are you going to be expanding your non-latin alphabet coverage? How do you reconcile the various codepages and character sets to return a comprehensive (language independent) results set? As a codicil, do you plan to incorporate other translation features to make "other" language pages available in other languages besides english? (e.g. translate a Chinese page directly into Greek instead of Chinese to English to Greek as it curenly seems)
Your complaints about being offended offend me.
I know Google relies heavily on links to weight its search results, but it also uses meta keywords. When will Google reward those of us who slog through the process of creating Dublin Core metadata? When will Google use that metadata to index pages? While some people load up meta keywords with extraneous entries, Dublin Core users are likely not to do that, producing more reliable self-indexing information.
Just a thought.
Have you thought about giving this a shot? Once you get as big as you, this has gotta make sense!
I've wasted a lot of money in my life, the rest I spent on motorcycles and women.
Pardon me for the shameless plug, but as you see it has some relevance to the comment. I am the author of Freecell Solver, which is a library and a stand-alone command line program that solves games of Freecell and similar solitaire variants. Now, since its first public version, I posted announcements for its subsequent releases on Freshmeat, each time announcing new features.
As a side-effect of this publicity, the Google search for "freecell solver" yields almost exclusively hits that has something to do with it. However, the query itself is generic enough that a user would just want to find a solver for Freecell, not necessarily my own.
I call this phenomenon the "Freshmeat Effect", albeit it is by no means restricted to Freshmeat. Is there anything Googlers plan to do to restrict such clogging of searches by constant publicity of a package with a mis-chosen title?
(Refer here for a slightly earlier record of this effect, and a call for developers to use original names to avoid it.)
We have two eyes and ten fingers so we will type five times as much as we read. http://www.shlomifish.org/
i was wondering if you had ever cuckoo for Cocoa Puffs?
U TF 8&q=cocoa+puffs
http://www.google.com/search?hl=en&ie=UTF-8&oe=
Original link had a gratuitous space courtesy of slashcode. Correct link: http://science.slashdot.org/comments.pl?sid=34509& cid=3742297
Body:
Google Answers (http://answers.google.com)
by Martin Spamer on 05:27 AM -- Friday June 21 2002 (Score:2) (#3742297)
(User #244245 Info) http://www.kitv.co.uk/ [ Neutral ]
What and When are you planning to to do about the frauds being perpetuated on Google Answers (http://answers.google.com) with questioners paying their own alias instead of the expert poster who deserves to be paid. They are destroying the cedibility of what has the potential to be an absolutely brilliant service.
I think you need to be a much more strigent about suspending and banning these abusers if you wish this service to take off.
I didn't know you had to be white to be intelligent and successful.
Let me ask you this: What's it like to be a total dick?
Drummond was first introduced to Google in 1998 as a partner in the corporate transactions group at Wilson Sonsini Goodrich and Rosati, one of the nation's leading law firms representing technology businesses. Drummond served as Google's first outside counsel, and worked with Larry Page and Sergey Brin to incorporate the company and secure its initial rounds of financing. During his tenure at Wilson, Sonsini, he worked with a wide variety of technology companies, advising them on all aspects of their business and financial activities and helping them manage complex transactions such as mergers, acquisitions and initial public offerings.
Curious if there are any plans to extend the Google API for web searches to include searching the Google Groups archives?
Putting aside the techie style questions; Who at google owns the coolest car ? Romero has his Hummers and Ferraris. Carmack has his twin turbo Ferraris some making upwards of 1000 horsepower at the wheels. Gates has some Porsche 959's that arent legal to drive on the roads in America. Do you drive anything out of the ordinary ? what about your coworkers ?
I was wondering how you find sites...
A friend of mine are running linux at home and use some kind of webbased quing system for his mp3 files (yes yes all legal of course *erhm*), he mentioned that every once in a while the google bot drops by and ques some of the files, so the question is, how did it ever find his site given the fact that probably no one links to his mp3 que list? (and what is the prefered music of the bot?? (seems to go for the dance/techno genre))
I will keep this brief! 0 Pageranks and site penalties are used on a site by Google if they link to "bad neighbourhoods" - but what happens if the Webmaster does this without knowing the full dangers and then, once aware of dangers, removes the dodgy links? Currently there is no response from Google to repeated requests on restoring a sites rank to "normal". Despite many, many e-mails people never get a reply. Personally I've had 2 sites restored but 5 of my sites are still affected despite the fact they are so clean they squeak! As it's over 6 months since this happened, does Google plan on lifting such penalties completely soon? Or at least giving some time to such questions via E-mail to people?
Well yeah, she's probably not too much of a nerd.
:)
But do you need a degree to be a nerd?
She's pretty for the older woman category, which is probably what a lot of the other posters are protesting. Nice eyes. But unfortunately no pictures on Google other than that one
And given her position, you won't be seeing her outside of respectable wear, as it'd be a bad career move for her.
Too bad.