Slashdot Mirror
What's It Like to be Google's Boss Techie?
We'd like to welcome Google Director of
Technology Craig Silverstein as our next Slashdot
interview victim... err... guest. You think you
run a big Linux server farm? Craig's is bigger.
Think your Web site gets a lot of traffic and
creates a lot of headaches? Just think what Craig
must face! Post whatever you'd like to ask Craig
below, one question per post. About 24 hours after
this runs we'll email Craig 10 of the
highest-moderated questions, and we'll post his
answers shortly after he gets them back to us.
Google always seem to be early-to-market with some really highly developed software solutions, and also always seems to have the backbone to support them.
I'm curious -- what drives the innovation? Is it the hardware team advancing architecture to permit the software team more room to play, or is it the software team saying, "Hey, look what we got!" and the hardware team dropping the iron to implement it?
I understand there must be some level of synergy, but is it completely seamless or is one side of the equation effectively driving the other?
Leem
What type of machines/setup does Google use?
(I've heard thousands of PC's with everything in RAM, but I'd love to hear it from the horses mouth)
Does google's policy of "ranking" the sites that have hits favor the "big guys" over more specific smaller traffic websites? That is, would a story on a site like CNN get a higher ranking in google on a keyword "Gulf War" than say a site (gulfwarveterans.com) that deals 100% with the Gulf War? Do you think you are leading to the commercialization of the web (i.e. the big power players) over smaller sites?
but I noticed a few months ago that Cisco now uses the Google engine to search the CCO. Congrats on that one. I've also noticed this new search box that Google is starting to produce. And it looks *very* cool. So my question is basically which is more important to your job the website or selling the service and the engine to people who need it?
Cypherpunks: Civil Liberty Through Complex Mathematics. Those who live by the sword die by the arrow.
Has there been any progress on the Pigeon Computing initiative?
I am the evil aardvark!
I am wondering why they chose Linux. Specifically, I wonder how they made the choice between all major OS-es (Linux, *BSD, Solaris and possibly Windows), as well as the software they use to power the site.
The Internet is always described as a distributed system with no single point of failure. Google, however, has quickly become by far the most popular method of locating information. "Surfing" has been killed with modern search technology, it's so much easier to look through Google than the Web itself. If Google was down, I'm sure the Internet would be far less useful.
Do you think Google has become an Internet point of failure? With the competition for larger and larger indexes, is the Internet becoming centralized? Do you think this is a bad thing?
What are you doing to prevent the new generation of more sophisticated search engine spammers- spammers that use advanced software such as WebPosition Pro, spammers that feed fake pages to the Google crawler, spammers that make bogus link pages to their own sites? Doesn't this new level of sophistication on their part mean that in large part Google must emphasize human website reviewers, such as those provided by the Open Directory Project, to a greater degree?
As a new network configuration guy, I am often stumped by a problem. I usually turn to google first, and my supervisor second. What has been the biggest problem that you have dealt with that will stand out in your mind years from now? As the "Head Techie", where did you turn, and what was the eventual resolution?
I'd rather you do it wrong, than for me to have to do it at all.
Does google plan on releasing more products like the Google Search Appliance in the near future - specifically those that are geared more towards the consumer level rather than business market? I would, personally, love to have some sort of google search engine on my machine to rummage through all the stuff I have. Does google plan on expanding into this market or will you remain focused on the web?
I know, I know, Only one question but - it begs to be asked - how well is your technology going to be able to scale? Considering the near-expotential growth of the internet will PageRank be able to keep up?
I understand that Google was using large numbers of IDE drives in lieu of more expensive but individually faster SCSI devices. What prompted the decision, and how have the concerns of reliability and performance been mitigated. What special technology, if any, was used to implement such a system
...as to what exactly Google does with the concepts it receives through the various Google-tech contests held. Have these ideas been made good use of? Do we see any of this in the Google we use every day? What about the ones that didn't win, do we see any of them?
Whats the google language of choice for web page building. I'd assume speed is the most important, so what language makes google so fast?
Good quote, too many chars. Seriously, the slashdot 120 char limit sucks!
Is there anything new that Google is working on that is not currently displayed in your labs section? If so could you explain it to us?
If you could sum it up in a nutshell, maybe you should be writing O'Reily books. --- Domasi 2001
as Google got more popular and eventually reached the status it holds today, did you feel any pressure (either internally or from outside the organization) to switch from a Linux based cluster to a proprietary solution (Windows comes to mind, but there are others). Where you (or others at Google) affected by any of the FUD that is put out, and did it affect your perception of Linux as a viable solution?
It seems that Google's great successs is partly due to research coming out of the academic world. How many google employees have advanced degrees, and can they publish non-proprietry research after they join Google? How do you see the interplay between high-tech and Academia?
HOWTO get better dates on slashdot
Does Google use any natural language processing (when dealing with web pages, queries, etc.)? Are you planning on doing more with NLP in the future?
She sat at the window watching the evening invade the avenue.
How have these affected you and your job, and what are you feelings on this subject?
Xaotik Designs
Why in this day and age does google continue to penalize sites that are virtual hosted? With ip addresses becoming harder to get/justify every day why does google discount the relevance of links that don't come from a unique ip address. Please don't just deny it, I think the Internet community deserves an explanation.
Google recently ran it's "first annual programming contest," with a winner receiving $10,000. Many slashdotters suspect this was simply a way to recruit new talent. So, was finding new people one of the initial goals for this project, and have you hired any new programmers as a direct result of it? What were the other goals (PR, generation of new ideas, etc) where there?
Engineers arn't boring people, we just get excited about boring things.
It's well known that you use Linux in your mega clusters. I was wondering if you have ever been approached by Microsoft, Sun, or HP in an effort to switch to their proprietary OSes.
I can't imagine that you haven't. It must have been a huge decision to invest in one technology, so are you satisfied with what you have?
Moderation: Put your hand inside the puppet head!
Recently, the english division of our company [black and decker] hired 'HyperMedia Trafficing' or some other similar named company to get them 'more exposure' in the search engines.
.. or why no one bothered to ask me what to do.]
.. How does Google plan to make sure they are :
.. well .. pretty much anything.
[forget the ethical debate about that
What I want to know, is - going fowards - as more and more of these companies start up, and discover more and more unscrupulious ways of 'loading' the search engines with bogus hits/visits/data/etc.
1) Not loosing ad $$ to these folks
and
2) prefenting every search from returning something like www.hotgrannysex.com or www.top50.com as the 1st (or first 15) results for a search on
--Ne auderis delere orbem rigidum meum, non erravi pernicose!
No offense to Mr. Silverstein, but I'm much more interested in Cindy! Beautiful, highly successful nerds are terribly rare!
Just so I'm not off-topic:
Mr. Silverstein, how does Cindy look in tight sweaters?
Drool...
Talisman
"Study your math, kids. Key to the universe." -The Archangel Gabriel
One of the most impressive things about Google to me is how easily you seem to have embraced an open model. I realize the outward view of a company can be quite different from the internal view. How easy is it actually to make decisions such as opening API's. If it's easy can you give some advice on how one might convince their boss.
Thanks,
-Dave
"as plurdled gabbleblotchits on a lurgid bee" - Prostetnic Vogon Jeltz. (One man's humorous is another mans flamebait)
This shouldn't be modded down. The question just needs to be asked in a clearer way.
What job opportunities are there at Google, and what opportunities in the industry as a whole?
Sig (appended to the end of comments you post, 120 chars)
Hi Craig!
I think Google absolutely rocks. It has by far the most intelligent/helpful search engine results. Thanks for the great service.
Now onto the questions- what is the Google vision / strategy for the future? Where can Google go? From a search engine perspective, what are some of the challenges that you have and improvements that can be made (perhaps speeding up crawling to make the latest content available, for example)? How are you going about solving these challenges, and when can we expect them to be implemented?
On a similar note, I've noticed that recently Google announced a "google box" that allows for corporate to take advantage of the google search algorithms and indexing. Any more products like this being planned?
I have a number of web servers, some Unix some Windows, and the number of attempted attacks each day from different IPs must run in to about one hundred. It is mostly people trying to execute commands or using malformed URLs trying to exploit some known past security hole. My question is, how many attempted attacks each day do the Google servers get?
Martin Piper
Owner - ReplicaNet and RNLobby
Google is a great free public resource. My concern is that it has to be expensive running a resource like that. I know Google's strategy is somewhat to use the free resource as a loss leader to promote your search technology, but the key word in "loss leader" is "loss". It's a great theory as long as you are able find people who want and need your search technology.
So my bottom line question is this: Does the web site pay for itself via the advertising? Is there a possibility that someday Google may decide the web site costs too much money to run if you get to a point where your reputation no longer needs the loss leader?
Sometimes it's best to just let stupid people be stupid.
It would be great if you did a documentary feature with TechTv or someone, because its one thing to read about your facility, but it would be another to see it.
Thanks for all of the help I've gotten from Google.com, I don't think I'd still be in schol without it.
Paradesign
PS, even just a photo feature on the site would be nice.
I want 2D games back.
Anyone who has ever needed a piece of information that was on a broken page will agree that the Google page cache is perhaps one of the most underrated and useful parts of your search engine.
There's one problem that everyone has with the cache, however - you don't deep-nest the caching, so that following any links on a cached page will lead to the original (probably broken) site, instead of to another cached page. Is there a technical or legal reason for why it works this way? Any chance we'll see deep caching at some point?
How do you avoid business pressures to make short-sighted solutions, and consistently make good, common sense ideas work instead of adopting ones from marketing sources? Not only does Google have the best search engine technology, but you consistently do the "right" thing. Clean, quick homepage, text only well-identified ads, interesting research projects, etc...This is the way many search engines start, but they all went the way of the "dark" side instead of adopting the "right" solution. In my jobs, it's been very difficult to execute and justify good engineering (or just common sense) under pressure from the people who control the money. Any advice for driving through well-thought-out decisions instead of adopting the "management fad of the month"?
Not to be too "X-File'ish", but does there come a point where too much knowledge is captured in Google? A point where anything that doesn't exist in Google doesn't exist, period? Wouldn't that represent a very tempting target for a bin Laden or a John Ashcroft, to try to control how the modern world thinks?
Kind of far out there, I know, but do you guys worry about this kind of thing?
sPh
Many sites, when referenced by Slashdot, crumble under the load. Can you folks see any difference, either to your "main" servers (www.google.com) or your cache servers?
Stupid job ads, weird spam, occasional insight at
Just curious when mod_google is going to be released for the apache webserver. It would be nice to have the power of Google indexing available to those of us without significant IT budgets (i.e. wife won't let me "buy another #$*@! computer").
What's the worst thing ever to happen to the google server farm? (Besides the pidgeons knawing on cables)
Tim Dorr
Owner/Manger
A Small Orange
How does google deal with denial of service attacks, particularly distributed ones?
The rest of us just suck it up with fat network pipes, but a high-profile target like google would be the holy grail of Internet vandals.
Has anyone ever poisoned your DNSes, effectively taking Google down even though the server are up? Successfully inserted bogus WAN routing info into the Internet, again effectively bringing down Google even though the servers are fine?
What's your worst cracker/net vandal story?
To what do you credit the popularity of google? Do you consider google a "success," or are you holding out for thousands of employees and billions in cash flow?
If you want to know more about the special logos (referred to as "Google doodles"), as well as see an archive of the Google doodles over the years, go here.
Nathan
Do you expect widespread usage of RDF/DAML/OWL/TopicMaps for explicit meta-data annotation of web resources, or will it be used only in small circles of specialized content providers like academia, or maybe not at all?
How will Google react? Do you plan to use meta-data provided by web resources if found, and how will you decide if it isn't just made up to get people on some bogus pr0n site (like with those <meta>-Tags today)? Will it someday render the brute-force approach of full-text-indexing obsolete?
Programming can be fun again. Film at 11.
Google has become such an important part of the Internet for millions of average users. With this in mind, my friends and I often joke about what would happen if (knock on wood) Google were to go out of business. I suggest that ICANN should do something useful for a change, and fund Google as an official, non-profit project for searching the net.
Although I have heard that Google turns a good profit, what exactly is preventing Google from becoming a not-for-profit organization? Couldn't Google take the extra income from licensing its search to create better search technologies and pay the employees, rather than make some shareholders rich? Wouldn't this perhaps make Google a more sustainable organization?
Non-Linux Penguins ?
Google is an incredibly popular and effective website. I'm curious about the amount of pressure you have to expand in order to "stay competitive" or "aptly serve consumer's needs". Is there any kind of a push to go the way of yahoo or amazon and try and include EVERYTHING on that simple page? As things evolve, do you really see Google staying the top engine in 3 to 5 years?
indeed..
I've made some really stupid posts to the newgroups in the past and I used my real name. Can you delete them for me?
How do you guys manage thousands of servers spread throughout multiple datacenters?
How do you handle user accounts? Event notification?
Do you guys use "enterprise" software like Tivoli or Openview, or did you roll your own solution?
Conformity is the jailer of freedom and enemy of growth. -JFK
What would it take to Slashdot Google? What do you do to avoid this? Have you been Slashdotted before, either from Slashdot itself or from some other link?
Carousel is a lie!
How can you possibly test bugfixes/changes that need to get deployed to thousands of machines? Furthermore, how in the heck do you deploy the changes once they're tested. I understand you probably can't describe the exact process, but perhaps you can enlighten us on some principals learned on the subject of CM on such a massive scale.
After the introduction of the Google API, some people, especially from the REST camp, criticized the the use of SOAP, claiming it just adds superflous bloat and is generally "unwebby". What do you think about this?
Programming can be fun again. Film at 11.
Everyone will ask about bandwidth, incoming lines, etc. (All the network capacity and capability stuff). Here's something a little more off the beaten track:
What technologies help to support the Google server farm? What kind of automated monitoring and trouble reporting tools are in use? Are they home brew, open-source, or COTS with some customization (scripts, etc)? And if you had to point to one area of network management and say "we could use some improvement or some better tools", what would that area be?
BTW - Google Rocks! I never use anything else anymore!
-- Mal: "Well they tell you: never hit a man with a closed fist. But it is, on occasion, hilarious."
Are you guys making enough money?
:)
I wish you'd give us some banner ads or something, I feel guilty. I don't want Google to go away.
Seriously, why don't you serve banner ads?
-Dan
There has been much debate about what the practical purpose for Google Voice search might be, could you fill us in? Is it really for use in cars?
Check out this to get most of your answers. Shouldn't we be asking him stuff that isn't sitting on their website?
Why haven't you implemented yet the toolbar for open source browsers? Are there technical difficulties or rather lack of interest from Google?
Can you tell us anything about how you are working
with the various intelligence agencies to provide
them information about seraches that are of interest to them?
Are you thinking of providing SSL access to your
web pages so that these agencies will have to work
with you instead just monitoring your network
traffic?
From one interview...
Jason: What led to Google's decision to use Linux? When did that start?
Sergey: Well, Larry Page and I were in the Stanford PhD program in Computer Science. And we developed Google there. The way the computer science program worked is there was a hodgepodge of computer equipment lying around, and we would grab whatever scraps we could. We had all kinds of computers: HPs, Suns, Alphas and Intel's running Linux. So, we gained a lot of experience with all of those platforms.
When we started Google, we had to make the decision of what we wanted to use. Of course we chose Linux, because it is the most cost effective solution.
PCs are not only much cheaper these days, but we can also get them very quickly, because they're such a commodity item. That's an incredible benefit. We just installed another 1,000 computers and we got that done in a few weeks. That's really hard to do with any other kind of workstation. I think that's an advantage that people don't entirely realize.
Jason: Did you view it as being better, or was cost the main reason?
Sergey: It was better in some ways. Certainly for our purposes, we felt the support was better. For example, the actual kernel authors will respond to problems pretty quickly. They are especially responsive to Google nowadays, since we're so widely used. We can have a 15 minute turnaround. You can't really beat that for support.
That was an important factor, but frankly, the cost was a bigger issue. PCs are so cheap, which is very important. Sun's Solaris is probably more stable than Linux on PCs. It's hard to determine the blame, whether it's the hardware or the operating system. But, it's a minor difference.
Jason: Then, does all of your support come from newsgroups or do you actually pay for it through Red Hat?
Sergey: We have an operations team of about ten people, which helps a lot. And other than that we check newsgroups and e-mail the authors of the code. Usually, if it's a problem we can't figure out, we go straight to the authors.
Jason: Is Linux used on desktops at Google?
Sergey: It depends. Engineering mostly runs Linux. Business development/marketing runs Windows. Actually, I use Linux with VMWare running Windows. Some people have two computers, particularly some people in engineering who do UI development and need to test things out on Windows platforms. I find it better to just use VmWare and have one computer.
Jason: In a technical sense, what does Linux lack? What does it not provide?
Sergey: The 64-bit file system, which I know they are working on. It's slowly coming around. I think there are still occasionally some stability issues. I'm not saying Linux is unique in that respect, but you definitely want to have reliability. There are some issues dealing with higher memory systems. If you get to 2GB, and you try to push it past that, we encounter various problems. I know we've had some trouble with the network stack when we really push it hard. In terms of having lost most connections from lots of different machines.
And from another...
How is Linux used at the Google Projects? Why was Linux choose to improve Google search engine?
Sergey Brin: Actually, we currently run over 6,000 RedHat servers.
Linux is used everywhere...on the 6,000+ servers themselves, as well as desktop machines for all of our technical employees. We chose Linux because if offers us the price for performance ratio. It's so nice to be able to customize any part of the operating system that we like, at anytime. We have a large degree of in-house Linux expertise, too.
Most of our administrative tools were developed in-house, as well.
I don't remember what HTTPd they're running but it sure as hell isn't apache. Someone said that they get 1k hits per SECOND; what do you use to shape that insane amount of traffic? What is the '/search' page coded in? What databases are used to index a terabyte of data? How do those 10,000 nodes find the data they need to quickly? what sort of interlinks are used?
;)
How to you build a cluster like a war machine, in other words?
Are there plans to index audio files (and the audio tracks of video files) so that these could be searched as well? I would guess that existing speech recognition packages could be reused for this purpose so that development would not be too complicated.
Recognizing text in images and videos and indexing that would be a similar task. I know that Google Catalog Search must be doing some OCR already, but I have no idea if this would take too many CPU cycles if applied to all images, or if there are other problems (the images themselves already get downloaded for the image search, so bandwidth should not be the problem).
A big part of Google's strength is in the supported search syntax, most notably that you can search for phrases instead of just keywords, that you can filter OUT certain phrases or keywords, and that you can search for content on specific sites, or NOT on specific sites. The next step for me and probably a lot of other Unix/Perl types is regular expression support.
For example, let's say I'm looking for 80's brat pack member Anthony Michael Hall (not that I would do such a think), but I can't remember his middle name. Looking for "Anthony Hall" will do me little if any good, but looking for "Anthony \w+ Hall" could do the trick nicely.
Another example is that the user can provide their own limited fuzzy searching, by searching for optional prefixes and suffixes along with the root, instead of having to get the word or phrase exactly as it's indexed.
Thanks,
John
What is Google doing to keep itself on top? Do you think there is a lot of room for improvement? How do you think web searching can get better?
So I'm truly surprised no one has asked this one yet, as it's the first thing that popped into my head...
/. story gets posted and people go stampeding to google to find out more? Or is that happening right now? (I'd hate to think of myself as part of a huge herd of individually acting DDOS'ers, but unfortunately, that's about what it ends up being...)
The masses of Slashdotters have slashed and dotted many an unlucky website over the years...Pushing webservers to their limit and often breaking them outright...
With Google's Massive resources, Is there any noticeable difference when a
Sig currently under construction. Mind the gap....
I have heard that Google uses Python extensively to manage its data, grab new data, etc.
As an avid fan of the Python language, I am interested in exactly how Google puts it to use. Can you clue us in?
P.S. - Keep up the good work!
We've all had servers crashing on us just before a deadline. We've all had to go to the office in the middle of the night to prevent a disaster. (we've all been hacked by a scipt-kid, once)
Do you have any stories of disasters or difficult moments in the datacenters that kept you all up for a few nights in a row, but went by unnoticed by the public?
What's the root password?
:)
Kickstart
Is there anything on the internet that you personally couldn't find with google and if so what was it?
p.s.
Thanks for all your help with my school research
How much is your data worth? Back it up now.
Can you talk a bit about how those weights have changed over time? Have there been any surprising shifts?
--- Jason Olshefsky
Karma: Poser (mostly affected by adding this line long after everyone else did)
Does this chain of thought keep you up at night?
Miko O'Sullivan
As an addendum to this, what is it about the corporate culture at Google that makes it work so well while other "hip" dot coms went down the toilet? What's the magic ingredient that made Google turn out differently?
Got Rhinos?
I used to kill hours watching the search requests scroll by on Metacrawler's Metaspy page back when people still used Metacrawler. Any chance we could have something like that on Google? I would *almost* even pay to subscribe to a site where I could watch uncensored Google search requests go by.
With the success and popularity of Google, I find myself using URL's for places less and less and just entering names into Google to find places (they are almost always on the first page...) Do you think that you have almost replaced the URL?
Craig,
When will images.google.com include PNG images in its search base? Why were the image types limited to GIF and JPEG, when most browsers could also display PNG? Now, virtually all non-text browsers support Portable Network Graphics.
Questions done. I'll take this opportunity to thank Google for groups.google.com, the searchable usenet archive. In my opinion, 15% of the total value of the internet is contained therein. Excellent!
- - -
"The sixth sick shiek's sixth sheep's sick."
I was gunna mod... but I ahve a re-quest-ion.
Can you guys put up bandwidth graphs for the public to see. Like mrtg graphs page showing daily google request traffic flow. so we can see what type of overall trends in searching happens during the day.
I would love to be able to see just how massive you traffic is and what it looks like.
and let us know what tools you use to monitor all your stuff.
When you were selecting the OS to run Google, why did you choose Linux? I'm partial to FreeBSD but I'm pretty sure that you evaluated it and found something a) that you didn't like or b) something about Linux that you liked better. If so, what?
Second part of this question: Do you continue to evaluate alternative operating systems?
Chris