Slashdot Mirror
Inside the Google-Plex
tappytibbins writes "Baseline magazine has an in-depth story about how Google manages its own IT infrastructure. From the article: 'In general, Google has a split personality when it comes to questions about its back-end systems. To the media, its answer is, "Sorry, we don't talk about our infrastructure." Yet, Google engineers crack the door open wider when addressing computer science audiences, such as rooms full of graduate students whom it is interested in recruiting.'"
print friendly version, because their page layout is a little too far on the "hey, if we add more adverts, we'll make more money!!1!"/WiReD-more-colors-are-good end of the scale.
I want inside the google party plane!!
Take the cheese to sickbay, the doctor should see it as soon as possible - B'Elanna Torres, "Learning Curve"
I'm still waiting for pictures of the "party plane", though.
What I'm listening to now on Pandora...
Its nice to know that some companies are willing to open their doors to the Tech comunity. Reminds me of Open Source Software... but only with hardware
It still worries me that google will soon know everything about everyone. I hope they dont share that data with ANYONE.
Mod others as you would have them mod you.
Most journalists and business analysts are notable for doing a half-assed job and taking credit for cut & paste jobs. Journalists who actually spend time researching their stories are a dying breed, so my take on this is that Google would rather not waste their time answering stupid questions from people who don't even understand what they're publishing. Their time is much better spent recruiting smart people or just talking to grad students in some sort of academic goodwill.
An alum of my university who works at Google recently visited and gave an informative lecture with a long Q&A session. I can vouch for the fact that we were told more than I've ever been able to read online about the way Google manages various issues, like their IT infrastucture. However there were still limitations to what he would/could tell us (sorry I won't go into specifics). It seemed (as you would expect) the better our questions, the better his answers, and if we asked questions that were too good, then it was likely that he did not feel liberated to answer.
Also, Google was cool enough to sponsor a Programming Contest and a Graduate Research Conference we held. Our alum attended our little conference and had great feedback and questions for our presenting students. With respect to knowledge, intelligence, and humor this guy was all I would imagine and/or hope for one of our alums working at Google.
On the otherhand, I was very unimpressed with certain issues concerning lack of professionalism in the lecture. As one example, though this is only an impression, it seemed that he felt he could just get away with wearing jeans and a Google t-shirt for the few days that he was with us because he worked at the ever prestigious Google. It seemed a bit arrogant. Also keep in mind that his position at google is higher than a solutions engineer.
Just thought I'd share.
Falun Dafa is good!
If it is anything like their web-presence, half the stuff must have 'Beta' appended to it.
New GPayRoll-Beta!
If this signature is witty enough, maybe somebody will like me.
Aha! He has revealed that the Google-plex contains a 'door.'
This slip of the mind will prove invaluable in my Google-imitation plots.
If this signature is witty enough, maybe somebody will like me.
In other news, Google has started tagging new employees on probation with 'BETA' labels.
Help a man when he is in trouble and he will remember you when he is in trouble again.
Everyone is talking about GoogleFS. But no one is talking about how they manage structured data. How do they do it? Some SQL stuff, some homegrow potion, or have they managed to create a sensible interface for structured data on top of GoogleFS?
Leandro Guimarães Faria Corcete DUTRA
DA, DBA, SysAdmin, Data Modeller
GNU Project, Debian GNU/Lin
What background do you have in software development? Currently Google interviews people for "core basics", which are the basic skills you would learn by going to university or trade school in the field related to your position. For example, when interviewing for the Java Developer Software Engineering position, you'll get a lot of questions about the collections library, synchronization, and core computer science questions like semaphores and two-phase commit. My experience with Java, and I know I'd completely fail a system administration interview.
Perhaps you should have informed your recruiter about your background?
my blog
Here are some good papers about Google's technologies:
Sawzall (simplified scripting on top of MapReduce)
MapReduce (Google's massively parallel system based on the concept found in functional programming. The system takes care of managing jobs, parallelism, and fault tolerance, allowing engineers to more quickly produce code.)
GFS (Google's File System)
Google's Cluster (An older paper describing how Google's search cluster works. The cluster described in this paper is a few generations out of date.)
BigTable (Google's semi-structured database. There haven't been any papers released, but this is my write up based on a talk given in October 2005.)
And here are some videos:
The Google Linux Cluster. This is an older video where Urs Hoelzle talks about their system and focuses more on the hardware side of things.
Google: A Behind-the-scenes Look. Jeff Dean gives an overview of most of the technologies mentioned in papers above. I thought the demonstration of Google's internal word clustering was interesting (and funny).
Perspectives on the Information Industry. This is a technology-light (IIRC) talk given by Eric Schmidt.
BigTable: A Distributed Structured Storage System. The talk from which I created my BigTables notes (above).
Andrew
I'm sure they need more than that. Google representatives often say (when talking about cheap commodity hardware), "With 1000 machines, you can expect one to fail everyday." Therefore, if they have 450,000 machines, you can expect about 450 to fail a day. Not only that, but they are probably adding machines like crazy and replacing old machines as they become cost-inefficient (the numbers I've heard say they keep computers for 2-3 years). I think it would take more than two guys to do all that. I'm sure they have a huge ratio of computers to sysadmins, but they still need a bunch of folks to replace the dead machines and add new ones. I imagine their servers are easy to manage on the software side, however.
i seem to remember reading they don't replace the failed ones, they just junk the entire rack when it becomes not worth running anymore.
i'm guessing google are big enough to have thier own datacenters and thus not have space at such a premium as smaller operations. If space isn't at a premium replacing a machine in a rack probablly isn't worth it (it means you have a machine whose remaining usefull life is out of sync with the rest of the rack its in).
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
That's an interesting point. I think a lot of companies are actually that way. I work for *undisclosed faceless corporation* and people show up here in shorts and birkenstocks. A guy on our team walks around the offices barefoot. Invader Zim posters, figurines, calendars of cheerleaders, etc. are all over the place. You could show up in flip flops if you wanted to... but people choose not to. There's something about that Google mentality that sounds neat at first, but then you realize that you're not in college anymore, and even though you CAN wear flip flops to work, you probably don't want to.
So is it neat to have a trendy office space? Sure. Is it neat to have communal centres scattered around the building, and be encouraged to stay afterhours to play games? I guess. But it's the kind of thing that gets old once you realize you've got a family and a life outside of work. Working for Google sounds like working in a basement with a bunch of friends, but that only really works if you don't have other things you want to be devoting time to. Once their workforce matures a bit, I'd guess their "kooky, trippy workspace" won't work quite so well. Don't forget, they're still basically a glorified startup. I'm sure Microsoft had a lot of the same feel back in '86.
Slashdot needs a "-1, Wrong" moderation option.
The Urban Hippie
It's not the highly parallel clustered racks of custom-designed linux servers that makes Google Google. That's an enabling feature. Rather, it's their custom engineered application-level operating environment, if you will, which runs on top of that. It's good at keeping data, indexing attributes, finding it, breaking problems down, and intelligently routing work and results. The search engine and all their other apps are built on top of this, and it allows their engineers to leverage this common distributed platform in all of their external and internal applications.
=== End Elevator Summary ===
Not many companies are willing to write their own application layers to deploy services. Most companies CAN'T. It's just not worth it. It's worth it to Google because developing and deploying world-wide information retrieval services is their business.
However, a standardized Application OE that can run and take advantages of the resources of many potentially unreliable computing resources would be very valuable to many businesses.
Grid technologies, web services, J2EE, and clustering technologies are just scratching the surface.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
I went through the majority of the process (phone interviews, then the face time at the New York facility and a trip to California) before I told them I wasn't interested in the job. My reasons for turning them down were three. First, the 80/20 deal was a myth if you were going to pursue something they were really interested in. I wanted to complete my PhD. Now, this PhD was not in a field they cared much about (despite their glaring need of my skills for one particular service), so that part of the deal was mumbled everytime it was mention. Second, the pay wasn't good enough to basically live at the facility. Third, the interview process was abusive in many respects. The first phone call was with a guy consumed by asking me about my doctoral research and my knowledge of how inodes work. He kept shifting between the two. When I asked him why this was even necessary given the position I was applying for, he got irritated and said that you had to know the ins-and-outs of how a file system works in order to configure (something I wouldn't be doing) any part of their infrastructure.
This lead to my observation of part of their file storage system which is quite possibly the most tweaked NFS nightmare/genius/what-the-fuck I'd ever seen. My past experience with networked file system was, I admit, very limited compared to what they had going on. Now, again, I wasn't even going to have anything to do with this system or any sysadmin work at all, but it was obvious that they wanted you to at least have knowledge of the system on some level beyond the user. It also came across as a showing-off culture too. I am glad I didn't take the job for various reasons, but if you are a sysadmin freaker who loves dinking with shit, you'd fit in; especially if you like to show it off too. Just be prepared to have some middle manager there fuck with you for a hour or two on the phone before you get to the outer part of the inner sanctum.
Comparing it to Windows will be a moot point, since El Dorado is going to have a 40% larger code base than XP.