Slashdot Mirror


How Google Uses Linux

postfail writes 'lwn.net coverage of the 2009 Linux Kernel Summit includes a recap of a presentation by Google engineers on how they use Linux. According to the article, a team of 30 Google engineers is rebasing to the mainline kernel every 17 months, presently carrying 1208 patches to 2.6.26 and inserting almost 300,000 lines of code; roughly 25% of those patches are backports of newer features.'

32 of 155 comments (clear)

  1. A New Culture by Anonymous Coward · · Score: 3, Funny

    Hmmm... Techno-Amish? (i.e. "We'll use your roads, but not your damned cars!")

    1. Re:A New Culture by MichaelSmith · · Score: 4, Informative

      Funnily enough the roads were there before the cars.

  2. Release the patches already by Dice · · Score: 5, Interesting

    They monitor all disk and network traffic, record it, and use it for analyzing their operations later on. Hooks have been added to let them associate all disk I/O back to applications - including asynchronous writeback I/O.

    I. Want. This.

    1. Re:Release the patches already by Anonymous Coward · · Score: 5, Informative
    2. Re:Release the patches already by Darkness404 · · Score: 3, Informative

      Its kinda common sense that Google would see how much disk space is used or how much CPU time is used. I mean, what admin -doesn't- know that 2 Gigabytes of space is used by xxxx@gmail.com? Even if all the data was super-encrypted you would still know how large the file is.

      --
      Taxation is legalized theft, no more, no less.
    3. Re:Release the patches already by Anonymous Coward · · Score: 5, Funny

      Can we donate some money and buy these people a site that _doesn't_ look like a goatse link?

  3. Togh by Anonymous Coward · · Score: 3, Informative

    Google does not distribute the binaries, so they are not obliged to publish the source.

    1. Re:Togh by pathological+liar · · Score: 5, Insightful

      Yeah great work Linus.

      The distros STILL stick with older versions and backport fixes, because who in their right mind is going to bump a kernel version in the middle of a support cycle? It's even MORE broken because the kernel devs rarely identify security fixes as such, and often don't understand the security implications of a fix, so they don't always get backported as they should.

      The Linux dev model is NOT something to be proud of.

    2. Re:Togh by grcumb · · Score: 5, Funny

      The Linux dev model is NOT something to be proud of.

      Indeed:

      "The Linux dev model is the worst form of development, except for all those other forms that have been tried from time to time." - Winston Churchill

      ... Oh wait, no. That was me, actually.

      --
      Crumb's Corollary: Never bring a knife to a bun fight.
    3. Re:Togh by Anonymous Coward · · Score: 3, Insightful

      Oh actually I think the form of development used by the BSDs is a lot better. At least it is a lot more efficient. They don't just crap software and deprecate it as soon as it remotely works (hal).

  4. Re:Open source is the coat tails that Google rides by i_ate_god · · Score: 3, Insightful

    you missed the point of open source then

    --
    I'm god, but it's a bit of a drag really...
  5. Re:Is it worth it? by Rockoon · · Score: 5, Insightful

    This company had about a million servers last time I cared to find out. I dont think 'more cheap hardware' means the same thing to you as it does to Google.

    --
    "His name was James Damore."
  6. Low memory conditions by jones_supa · · Score: 5, Interesting

    Google makes a lot of use of the out-of-memory (OOM) killer to pare back overloaded systems. That can create trouble, though, when processes holding mutexes encounter the OOM killer. Mike wonders why the kernel tries so hard, rather than just failing allocation requests when memory gets too tight.

    This is something I have been wondering too. Doesn't it just lead to applications crashing more often than them normally reporting they cannot allocate more memory?

    1. Re:Low memory conditions by IamTheRealMike · · Score: 4, Insightful

      Well, most programs are not OOM safe. It turns out to be really hard to write programs that behave gracefully in OOM scenarios. Killing a sacrificial process when the system is out of memory works OK if you have a pretty good idea of priority ordering of the processes, which Google systems do.

  7. Re:Is it worth it? by Sir_Lewk · · Score: 4, Insightful

    They are already running absolutely absurd amounts of cheap hardware. "Just buying more" is something that I'm sure they are already doing all the time but clearly that only goes so far.

    (30 * kernel engineer salary) / (generic x86 servers + cooling + power) = ?

    I suspect the answer to that is a very very small number.

    --
    "linux is just DOS with a UNIX like syntax" -- Galactic Dominator (944134)
  8. Re:Is it worth it? by coolsnowmen · · Score: 4, Insightful

    You are clearly not an engineer of scientist. Aside from the fact that some people just like to solve technical problems, I am betting google's logic goes something like this:
    We have a problem that is basically only costing us $0.01*10,000computers/day. While that seems low, we plan on staying in business a long time, we could pay someone to solve the problem. Then there is that X factor, that if you don't do it, if you stop innovating, your competitors will, and they will get more and you will get less from the pool of money that is out there. In addition to that, the CS guy you paid to solve that is now worth more to your company (if you employed him) because [s]he now has a better understanding of a complex bit of code (the linux kernel) that you rely on heavily.

  9. Does Google give coade back by TorKlingberg · · Score: 4, Insightful

    Does Google give any code and patches back to the Linux kernel maintainers? Since they probably only use it internally and never distribute anything they are not required to by the GPL, but it would still be the right thing to do.

    1. Re:Does Google give coade back by MBCook · · Score: 5, Informative

      Yes, they do. Since they use older kernels and have... unique... needs, they aren't a huge contributor like RedHat, but they do a lot.

      During 2.6.31, they were responsible for 6% of the changes to the kernel.

      --
      Comment forecast: Bits of genius surrounded by a sea of mediocrity.
    2. Re:Does Google give coade back by marcansoft · · Score: 4, Interesting

      Andrew Morton, Google employee and maintainer of the -mm tree, contributed the vast majority of the changes filed under "Google" (and most of those changes aren't Google-specific - Andrew has been doing this since before he was employed there). If you subtract Andrew, Google is responsible for a tiny part of kernel development last I heard, unfortunately.

    3. Re:Does Google give coade back by CyrusOmega · · Score: 3, Informative

      A lot of companies will also use a single employee for all of their commits too. I know the company I used to work for made one man look like a code factory to a certain open source project, but, in fact, it was a team of 20 or so devs behind him doing the real work.

    4. Re:Does Google give coade back by marcansoft · · Score: 4, Informative

      Andrew has been doing a large amount of kernel work for some time now, before his employment with Google. Note that the 6% figure is under non-author signoffs - people that patches went through, instead of people who actually authored them. Heck, even I submitted a patch that went through Andrew once (and I've submitted like 5 patches to the kernel). Andrew does a lot of gatekeeping for the kernel, but he doesn't write that much code, and he certainly doesn't appear to be committing code written by Google's kernel team under his name as a committer.

      Google isn't even on the list of actual code-writing employers, which means they're under 0.9%. I watched a Google Tech Talk about the kernel once (I forget the exact name) where it was mentioned that Google was (minus Andrew) somewhere in the 40th place or so of companies who contribute changes to Linux.

    5. Re:Does Google give coade back by farnsworth · · Score: 4, Informative

      Google is responsible for a tiny part of kernel development last I heard, unfortunately.

      I don't know that much about google's private modifications, but the question of "what to give back" does not always have a clear default answer. I've modified lots of OSS in the past and not given it back, simply because my best guess was that I am the only person who will ever want feature x. There's no point in cluttering up mailing lists or documentation with something extremely esoteric. It's not because I'm lazy or selfish or greedy -- sometimes the right answer is to just keep things to yourself. (Of course, there are times when I've modified something hackishly, and had been too lazy or embarrassed to send it back upstream :)

      Perhaps google answers this question in a different way than others would, but that doesn't necessarily conflict with "the spirit of OSS", whatever that might be.

      --

      There aint no pancake so thin it doesn't have two sides.

    6. Re:Does Google give coade back by itzdandy · · Score: 3, Insightful

      If you subtract search engines google is responsible for a a tiny portion of the internet. Andrew gets benies from google so I suppose they do get some credit for the quantity of his work as he needs to eat and pay rent so that he can code.

    7. Re:Does Google give coade back by marcansoft · · Score: 3, Informative

      By that I meant "developed for Google, useful to other people".

      We can divide Andrew's potential kernel work into 4 categories:

      1. Private changes for Google, not useful for other people.
      2. Public changes for Google, deemed useful to other people but originally developed to suit Google's needs.
      3. Public changes of general usefulness. Google might find them useful, but doesn't drive their development.
      4. Maintaining -mm and signing off and merging other people's stuff

      Points 1 and 2 can be considered a result of Andrew's employment at google. Points 3 and 4 would happen even if he weren't employed at Google. From my understanding, the vast majority of Andrew's work is point 4 (that's why he's listed under non-author signoffs as 6%, along with Google). Both Andrew's and Google's commit-author contributions are below 0.9%.

      So what we can derive from the data in the article, assuming it's accurate, is:

      • Google's employees as a whole authored less than 0.9% of the changes that went into 2.6.31
      • Andrew authored less than 0.8% of the 2.6.31 changes
      • Andrew signed off on 6% of the 2.6.31 changes
      • Besides Andrew, 3 other changes were signed off by Google employees (that's like .03%)

      So no, Google doesn't contribute much to the kernel. Having Andrew on board gives them some presence and credibility in kernel-land, but they don't actually author much public kernel code. Hiring someone to keep doing what they were already doing doesn't make you a kernel contributor.

  10. Re:Is it worth it? by dingen · · Score: 5, Interesting

    Ooooh... efficiency.. I'm curious what the net savings is.. compared to buying more cheap hardware.

    We're talking about Google here. They have dozens of datacenters all over the globe, filled with hundreds of thousands of servers. Some estimate even a million servers or more.

    So lets assume they have indeed a million servers and they need 5% more efficiency out of their server farms. Following your logic, it would be better to add 50,000 (!) cheap servers which consume space, power and require cooling and maintenance, but I'll bet you paying a handful of engineers to tweak your software is *a lot* cheaper. Especially since Google isn't "a project" or something. They're here for the long run. They're here to stay and in order to make that happen, they need to get the most from their platform as possible.

    --
    Pretty good is actually pretty bad.
  11. Re:Open source is the coat tails that Google rides by IamTheRealMike · · Score: 5, Insightful

    Hmm, you realize that Android alone is over 10 million lines of code right? That's a pretty big open source contribution right there. But then there's also over a million lines of code across 100+ smaller projects too. So I am not sure what your definition of "table scraps" is but it's significantly more lines of code than most companies do.

  12. Re:Is it worth it? by Rockoon · · Score: 5, Insightful

    Also consider the fact that Google has been basically deploying new servers non-stop for many many years. They are already purchasing cheap hardware at a very high rate. Even a tiny 1% improvement in efficiency for the existing and future servers is a huge huge win for them.

    That could amount to hundreds of millions of dollars saved over the next decade, and it doesnt take a genius to realize that a couple dozen programmer salaries will be a hell of a lot less than that.

    --
    "His name was James Damore."
  13. Re:Is it worth it? by LordNimon · · Score: 4, Interesting

    Porting patches from one kernel version to another is not innovation.

    A while back I got an invitation to work for Google as a kernel developer. I declined to interview, because I already had a job doing just that. This article makes me glad I never accepted that offer. I feel sorry for those kernel developers at Google. Porting all that code back-and-forth over and over again. Now *that's* a crappy job.

    --
    And the men who hold high places must be the ones who start
    To mold a new reality... closer to the heart
  14. Are you nuts by Anonymous Coward · · Score: 3, Insightful

    I'm not a huge goog fan, I never take their cookies so I don't use anything but search..but JUST search is way more "give back" than table scraps. If they announced tomorrow their search would now cost x-dollars a year, as long as it was somewhat reasonable,like an extra 5 bucks a month on top of my ISP bill, I'd pay for those table scraps. Google search has done more than anything else to make the web actually *useful* since the invention of the hyperlink.

    Sure, there are other search engines, but if you actually learn to *use* the features and filters present wih google's, it just stomps all the others flat.

    Whatever they give back in terms of code is just gravy on top of that.

  15. Real example... by Fished · · Score: 4, Interesting

    Back in the 90's, we had a customized patch to Apache to make it forward tickets within our intranet as supplied by our (also customized) Kerberos libraries for our (also customized) build of Lynx. It all had to do with a very robust system for managing customer contacts that ran with virtually no maintenance from 1999 to 2007--and I was the only person who understood it because I wrote it as the SA--when it was scrapped for a "modern" and "supportable" solution that (of course) requires a dozen full-time developers and crashes all the time.

    Not really bitching too much, because that platform was a product of the go-go 90's, and IT doctrine has changed for the better. No way should a product be out there with all your customer information that only one person understands. But it was a sweet solution that did its job and did its job well for a LONG time. Better living through the UNIX way of doing things!

    But, anyway, I never bothered to contribute any of the patches from that back to the Apache tree (or the other trees) because they really only made sense in that particular context and as a group. If you weren't doing EXACTLY what we were doing, there was no point in the patches, and NOBODY was doing exactly what we were doing.

    --
    "He who would learn astronomy, and other recondite arts, let him go elsewhere. " -- John Calvin, commenting on Genesis 1
  16. Re:So about 1/10th Sun's contribution by Again · · Score: 4, Funny

    That's a drop in the bucket compared to what Sun has contributed to open source. Of course, slashdot appears to be perversely against Sun for some reason I cannot fathom.

    Names are very important. The name Sun reminds of that place on the other side of the door where if we go, our skin gets red and burns. Google reminds us of that friendly homepage that would load under 5 seconds on dial-up.

  17. Re:Open source is the coat tails that Google rides by petrus4 · · Score: 3, Insightful

    They take and take from open source and throw back a couple of table scraps and you people all kiss their ass for it.

    300K lines of code? Yep, table scraps.

    For people who wonder why I continue to want to see the end of the FSF, the above attitude is the reason why. Stallman and his organisation are the reason for it.

    Aside from being ugly and spiritually bankrupt, reciprocity paranoia is based on completely erroneous reasoning, as well. The same people who talk about how music piracy isn't harming anyone, because it doesn't physically take away from a finite supply of copies, are also those who express the above paranoia about people "taking," from FOSS, as if that is somehow a physically finite resource, when music isn't.

    Get rid of your fear.