How Google Uses Linux

postfail writes 'lwn.net coverage of the 2009 Linux Kernel Summit includes a recap of a presentation by Google engineers on how they use Linux. According to the article, a team of 30 Google engineers is rebasing to the mainline kernel every 17 months, presently carrying 1208 patches to 2.6.26 and inserting almost 300,000 lines of code; roughly 25% of those patches are backports of newer features.'

Comments

Release the patches already

[Dice]

They monitor all disk and network traffic, record it, and use it for analyzing their operations later on. Hooks have been added to let them associate all disk I/O back to applications - including asynchronous writeback I/O.

I. Want. This.

Low memory conditions

[jones_supa]

Google makes a lot of use of the out-of-memory (OOM) killer to pare back overloaded systems. That can create trouble, though, when processes holding mutexes encounter the OOM killer. Mike wonders why the kernel tries so hard, rather than just failing allocation requests when memory gets too tight.

This is something I have been wondering too. Doesn't it just lead to applications crashing more often than them normally reporting they cannot allocate more memory?

Re:Is it worth it?

[dingen]

Ooooh... efficiency.. I'm curious what the net savings is.. compared to buying more cheap hardware.

We're talking about Google here. They have dozens of datacenters all over the globe, filled with hundreds of thousands of servers. Some estimate even a million servers or more.

So lets assume they have indeed a million servers and they need 5% more efficiency out of their server farms. Following your logic, it would be better to add 50,000 (!) cheap servers which consume space, power and require cooling and maintenance, but I'll bet you paying a handful of engineers to tweak your software is *a lot* cheaper. Especially since Google isn't "a project" or something. They're here for the long run. They're here to stay and in order to make that happen, they need to get the most from their platform as possible.

Re:Is it worth it?

[LordNimon]

Porting patches from one kernel version to another is not innovation.

A while back I got an invitation to work for Google as a kernel developer. I declined to interview, because I already had a job doing just that. This article makes me glad I never accepted that offer. I feel sorry for those kernel developers at Google. Porting all that code back-and-forth over and over again. Now *that's* a crappy job.

Re:Does Google give coade back

[marcansoft]

Andrew Morton, Google employee and maintainer of the -mm tree, contributed the vast majority of the changes filed under "Google" (and most of those changes aren't Google-specific - Andrew has been doing this since before he was employed there). If you subtract Andrew, Google is responsible for a tiny part of kernel development last I heard, unfortunately.

Real example...

[Fished]

Back in the 90's, we had a customized patch to Apache to make it forward tickets within our intranet as supplied by our (also customized) Kerberos libraries for our (also customized) build of Lynx. It all had to do with a very robust system for managing customer contacts that ran with virtually no maintenance from 1999 to 2007--and I was the only person who understood it because I wrote it as the SA--when it was scrapped for a "modern" and "supportable" solution that (of course) requires a dozen full-time developers and crashes all the time.

Not really bitching too much, because that platform was a product of the go-go 90's, and IT doctrine has changed for the better. No way should a product be out there with all your customer information that only one person understands. But it was a sweet solution that did its job and did its job well for a LONG time. Better living through the UNIX way of doing things!

But, anyway, I never bothered to contribute any of the patches from that back to the Apache tree (or the other trees) because they really only made sense in that particular context and as a group. If you weren't doing EXACTLY what we were doing, there was no point in the patches, and NOBODY was doing exactly what we were doing.