Behind the Scenes At Google

An anonymous reader writes "University of Wahington TV Presents "behind the Scenes With Google." From the site: 'Search is one of the most important applications used on the internet and poses some of the most interesting challenges in computer science. Providing high-quality search requires understanding across a wide range of computer science disciplines. In this program, Jeff Dean of Google describes some of these challenges, discusses applications Google has developed, and highlights systems they've built, including GFS, a large-scale distributed file system, and MapReduce, a library for automatic parallelization and distribution of large-scale computation. He also shares some interesting observations derived from Google's web data.' "

Comments

OK then where the hell is

[Anonymous Coward]

proximity search (with adjustable range would be extra nice).

i.e.

((gopher OR shrew OR egret) AND -(mole OR newt)) NEAR(range) ((evil OR "satan incarnate") AND (roe AND -chicken))

"In Italy for thirty years under the Borgias they had warfare, terror, murder and bloodshed but they produced Michelangelo, Leonardo da Vinci and the Renaissance. In Switzerland, they had brotherly love; they had five hundred years of democracy and peace and what did they produce? The cuckoo clock." -- Orson Welles (1915--1985).

I use Google at work

[Dancin_Santa]

I was reading an article a year or so ago about the corporate offices of Google and how there is a projection of all the latest searches displayed in real time on the wall behind the receptionist.

Now I have some pretty important lists which I need to keep tight control over. The information really ought not be distributed outside my office. However, because of the nature of my business, I must do frequent searches using various search engines to fill in my lists.

How am I assured that my searches remain anonymous and secure with Google?

Few women in CS.

[Seumas]

So, I'm always reading about how unfair the tech world is, because there are so few women joining it. But if you watch the video, the audience is surprisingly full of them.

Google & Backup

[Anonymous Coward]

I wonder how Google backups its data -- especially the Gmail data. Does the GFS support automatic replication?

Images of clowns

[saskboy]

"Behind the scenes at Google" invokes images of clowns and mimes. Is it just me? Imagine all the people in the world who haven't used the Internet, they probably would get the same impression from the phrase too.

Re:GFS

[AKAImBatman]

At the time OpenMosix had their own distributed filesystem called MFS. But it's proved inadequate, which is why they are switching to GFS

I'm sorry, did I miss the point at which Google made an open source implementation of GFS? Last I knew, the only docs for GFS were the papers that Google published on the concept. And those papers (unfortunately) seemed to lack a few of the finer details of implementation.

Behind the scenes?

[Anonymous Coward]

Disclaimer: my opinions expressed herein are not necessarily those of Google, Inc.

That having been said, as a long time insider I have a pretty good idea about what really happens "behind the scenes" and let me tell you, both conspiracy theories crackpots and our slashdot fanboys are quite amusing, but the boring fact is that we are neither trying to take over the world, nor are we the best thing since the second coming of Jesus.

We used to be a very successful startup, yes, and now we are a fairly successful corporation. Yes, there are a lot of smart people working here, but don't fool yourself, "the most interesting challenges in computer science" are happening in academia, not in corporations. (Besides, anyone who knows Jeff is perfectly aware that he often tends to grossly exaggerate our importance, but to be honest that is a part of his job which he is doing really great.)

All in all, I love to work here, I thing there are a lot of very smart people here, but if you think that we are the only place on the planet where geniuses cluster lately, you are just not being reasonable. If you want to find real discoveries you have to look in places where people don't have shareholders telling them what to do. The point is that we haven't done anything new per se, only the scale of our implementations is unprecedented.

For example, in my 20% time (Google allows us to spend 20% of paid work time on personal projects) I am working with KeyKOS right now and let me tell you, this is what I call innovation. It was done in the '70s and no mainstream OS has implemented its ideas to this day so far. I'm sure that when after a decade or two a Big Corporation (be it Google, Microsoft, Apple, or IBM) reimplements KeyKOS, the Slashdot crowd will wet their pants screaming "wow, what an innovation!" completely forgetting that it was an innovation back in the '70s of the 20th century when Norm Hurdy et al. were working on it quitely with no buzz and fanfares. Please remember that "The Next Big Thing" is always an old idea but this time backed with $$$ and marketing. Please never forget it, or otherwise the people who are worth their salt will only consider you uneducated.

Re:Behind the scenes?

[Fall into This]

This has got to be the best post I've read about Google. I am so friggin' sick of hearing BS about "Google's Gmail is EVIL!!!!!!!!111111!!!!x0rz! Just READ their terms!!" and the such. Woopdie doo, read Yahoo's. Speaking of which, no one seems to be bitching about Yahoo's 'evils.' Seems to me that if Google's actions are so borg...ish, then why have other search engines not been brought up? Google Maps comes out, all I hear is "another step towards monopolization." Yahoo Maps, no one seems to give a crap. Hypocritical if you ask me, especially considering the mail, news, stock, and so forth. Google's a company, and shall act as one, as others before it: be they Yahoo, Microsoft, Apple, etc. Especially in light of all of these fuckedgoogle.com-esque sites popping up all over the place, continuous digg (and even Slashdot) stories about Google, and flame wars started therein, I have to say it's all getting quite old.

On the other hand, far too many people are suporting Google as if it's the best thing since air. Yeah, I use and (platonically) love Gmail, even if for the interface, but to each his own. There are those who use Yahoo, Hotmail, their own ISP, etc., and still like it. Liking Gmail doesn't make it the best thing there is, only the best for you. The only thing I can agree with in certain cases - such as the new Gmail gigabyte or so forth - is the stimulation of competition. Google has been rather masterful at doing it thus far, at least to the point of causing those "vindictive Google" sites to spring to life. Which is another thing. The only thing that can be left for argument's sake in so many of these "Google is run by Satan" pages is stuff that happens that doesn't really mean anything. They shout things like "GOOGLE'S STOCK IS OVERPRICED, BUT SUFFERING IN THE MARKET" as if it actually proves something; yeah...stock...it rises, falls, and comes at various prices. So? What's so evil about the stock market? Would MSFT decreasing in value be susbtantial "dirt" on Microsoft? Or then there's the firing of the infmaous blogger who worked at Google; yay, he got fired for violating certain terms he agreed to. Guess we should consider dismantling that friggin' Constitution: it's only been a basis for punishing people when it is supposed to promote goodness! The truth is that it'd be difficult for any average employee of any major company to tell the world about his/her experiences, given the confidentiality of certain company info. Interesting to note, too, how Google-hating sites are listed in that naughty search engine in order to get their (in my humble opinion) meaningless word out.

I'm frankly sick of reading all the flamewars, though I suppose this is then hypocritical of me, for I have 1) contributed to such a discussion, 2) read it when I could've chosen not to, and 3) um, I dunno, but three is just one of those standard numbers to use...whatever. I guess I'm still interested in the workings of Google, though, since it seems like a nifty place to work - regardless if my view was shaped by all the subliminal hype Google is subjecting me to (I love Satan...I love him good). I'm just sick of the "miracles"/"totemic worship" occuring at Google; all I want is information on how they work, and I'll make my decision, which incidentally sits somewhere in a comfortable middle...er...actually leaning to the positive side, at least. The post to which I am replying coincides with the information I like to hear: truthful stuff concerning what goes on in the company, assuming of course that the poster is working at Google. That's my two cents, anyhow.

University Recruiting Talks

[stevemm81]

Google is constantly giving talks like this at universities. I saw one at Harvard back in the fall.
They aren't really news worth reporting on slashdot, since they all contain the same content.

Re:Dirt? That more like modelling clay

[Tibe]

You think Google make their money from AdSense? AdWords? etc.?

Google has resorces and expertise beyond most companies, possibly including Redmond.

They have at their fingers the most up-to-date information, opinions, numbers, rantings of most of the world. Do they use this to make income? I bet.

Banks already analize thier data and invest accordingly, Google are bound to do the same. (A la Google news.) With their expertise it is likely to be far more advanced and therefore more profitable.

They don't need dirt. Google can react to the world market before the world knows it's reacting, and well before it understands.

OK that's enough. Time for me to put my tin-foil-nightcap on and head to bed.

50% female is the goal

[Flamesplash]

When google was recuiting at Georiga Tech they stated that one of their founders had the 'vision' of having half of google female in the near future.

One of the thecnical female googerls mentioned how that was probably impossible, but by shooting for the impossible you acheive a lot more than you would have otherwise.

Re:50% female is the goal

[Anonymous Coward]

And who is going to shoulder all of the extra work and risk for having such a significant part of your workforce leaving to become stay at home moms, stay away for maternity leave and decide not to come back, take time off to go pick up the kids, take them for shots, go to school meetings, etc?

Seems like a risky business move, if you ask me. Since women (on average) do not put in the same hours, risk, dedication and sacrifice that men do.

Anyone who thinks I'm full of shit sound look for a book called "WHY MEN EARN MORE". It's written by the former head N.O.W. in NY.

Background

[Pan T. Hose]

A quick search on KeyKOS makes one wonder: Does it have anything in common with GNU's microkernel efforts? Anyone cares to post a brief overview of KeyKOS, possibly in connection and/or comparison to Mach/HURD?

Short answer: yes it does, and it is actually one of the main reasons why I look forward to use Debian GNU/Hurd in the future. Let me quote my old post from January with some background and interesting links to more informations about KeyKOS:

Still, you can't block every hole in security. Sometimes you just have to hope, right?

Yes, you can. No you don't. Software is just an applied form of discrete mathematics. "Beware of bugs in the above code; I have only proved it correct, not tried it," as Donald Knuth once said. It is possible to present a formal proof of correctness for any algorithm. It is nearly impossible and certainly impractical when you have a big mess of spaghetti code like with most of software that is utter crap, but it is possible nonetheless when you know what are you doing and design appropriately, with very clean, small and isolated parts of your system responsible for enforcing its security policies. Take a look at such operating systems as KeyKOS and EROS. E.g. read Verifying Operating System Security paper by J. S. Shapiro and S. Weber: "This paper presents a proof of correctness of the EROS operating system architecture with respect to confinement." Read some essays by Norman Hardy, especially those on Capability Theory. This is hardly a new idea, see GNOSIS: A Prototype Operating System for the 1990s paper by Bill Frantz, Norm Hardy, Jay Jonekait and Charlie Landau, written more than 25 years ago. The bottom line is: it is certainly possible to have a 100% secure system, but developers don't bother because users don't care.

And here is a newer post of mine asking exactly your question about KeyKOS capabilities in connection to the recent development of The Hurd, in the First Program Executed on L4 Port of GNU/HURD discussion two months ago:

When the first programs run, it is just a matter of time before there is a functional L4 port of Debian GNU/Hurd (or just Debian GNU?). I really like the design of the Hurd, but what I'd like to see the most are not the "POSIX capabilities" but the real capabilities as described in the 1975 paper by Jerome Saltzer and Michael Schroeder, The Protection of Information in Computer Systems. (For those who don't know what am I talking about, I recommend starting from the excellent essay What is a Capability, Anyway? by Jonathan Shapiro, and then reading the capability theory essays by Norman Hardy. As a sidenone I might add that I find it amusing that people who say that there are other advantages than only Digital Restrictions Management of using TCPA/Palladium-like platforms usually quote security fe

Google innovates? It's news to me.

[danila]

May be Google has done some nifty things with their file-system, but can't we forget about it already? Their search hasn't changed much http://www.google.com/">in the past six years. Of course, the fanboys will salivate over Google calculator and Google unit converter, but on the scale of Internet these "innovations" barely register.

Some of the other search engines are comparable in quality to Google (Teoma, Vivisimo), and may be better, depending on how many points you take away from Google for spam-infested results, too many blogs, too many Wikipedia clones, too many commercial sites, etc. And some sites are so much further on the innovation scale (meet BrainBoost, an artifically intelligent Internet reference desk answering any questions asked in natural English, with amazing quality and accuracy in a very friendly and usable interface) that they put Google to shame.