Android and the Linux Kernel Community

An anonymous reader links to Greg Kroah-Hartman's explanation of a rift (hopefully mendable) in the development culture of Google's Linux-based Android OS and the Linux kernel itself. "As the Android kernel code is now gone from the Linux kernel, as of the 2.6.33 kernel release, I'm starting to get a lot of questions about what happened, and what to do next with regards to Android. So here's my opinion on the whole matter ..."

Comments

Lots of comments on LWN.net's coverage

[chrisd]

If you head over to LWN, we've already gone back and forth on this a bit. http://lwn.net/Articles/372419/. The short form is that if they don't like how we use the kernel, we're unlikely to be accepted upstream. It's all still released as source code to the world, but the mainline is not interested in most of what we've with to the kernel.

Re:Lots of comments on LWN.net's coverage

[Rene S. Hollan]

Okay. I got off my ass and actually looked at waitlock. It isn't, as I thought, a lock with a restoration of execution context when held, so that one can temporarily surrender other locks, and reacquire them. It is a lock on keeping the system out of various low-power states.

Mainline has an (arguably, according to Google-folk, inefficient), lock on guaranteed service latency (low, medium, high?). and Google wants a lock on specific activity level abstractions (idle, suspend, etc.). Did I get that right?

Well, if the present implementation is inefficient (a linear search...), fix that: use an AA-tree or heap, or something.

I suspect the latency time guarantees are coarse at best and probably map to specific power states (idle, suspend, etc.) anyway. I think BOTH models should be supported (heck, you could always map levels to latencies in some tunable fashion, if you had to fake it: "responsiveness SUSPEND" vs. "hardware SUSPEND") but only because there are already standard abstract notions of what IDLE, SUSPEND, and POWEROFF are.

What may have happened is that the existing implementation was inefficient, not quite fitting the expected model, and Google-folk found it faster to roll their own driven by time-pressure, instead of reworking the existing inefficiencies, thinking "heck, we can layer our API over the existing implementation later" and finding the models are more different than thought.

Reconciling that is called "refactoring" and sometimes it breaks APIs as much as one does not want it to.

Now, who has a vested interest in doing that?

Google? Not really. They figure they can handle the overhead of a kernel fork, at least for a while.

Mainline kernel devs? No. The existing model "works".

I'll tell you who: Google's downstream customers. One of them will do it.

Re:Google

[gyrogeerloose]

It is called Embrace, Extend and Extinguish. I thought it only applies to MS. Well I think, Do no Evil is gone through the window :)

Well, we know what Steve Jobs said about Google's "Don't Be Evil" mantra--"It's bullshit." Or, "a load of crap," depending on your source.

Re:Google

[h4rr4r]

If linux was under the BSD license it would be in the same shape FreeBSD is in, nearly no one using it in the server room.

So sure we have to wait for btrfs instead of using ZFS, but only because SUN decided to make sure they chose an incompatible license.

Re:One sentence to say it all...

[minsk]

it wouldn't build as there were items Google never released that it was dependent upon.

Are you sure that statement is true? It seems inconsistent with other information posted here, and in the LWN discussion.

Technical aspects

[shutdown -p now]

"OMG fork!" and other political issues aside, I think it's interesting to look at the technical side of the problem. What is the exact nature of Google's changes to the kernel, why did they feel they need them, and are they actually a good idea or not? Can someone with kernel hacking experience enlighten us?

I'm particularly curious after reading the comments on LWN, and specifically this:

Kernel developers (including other embedded developers who have achieved good power savings modes) don't believe that the Android way of doing things is good.

and this:

The code could be mainlined if Google were willing to consider that their wakelock approach was suboptimal and adapt to a more reasonable one. ...

The wakelock patches were first posted to the linux-pm list on the 13th of January 2009, which is just over a year ago. http://lwn.net/Articles/318611/ gives a good overview of how they were received - there's basically no buy in, even from other developers in the embedded Linux field. ...

the major sticking point (ie, wakelocks) were posted for public review a year ago and most of the substantive issues people had with them weren't addressed at all in the four months of intermittent discussion that followed. If the entire world suggests that you do something in some other way and you refuse to, that doesn't constitute a genuine effort to work with them. Nokia have managed to obtain the same level of power management without such invasive changes, so any assertion that wakelocks are required for Android to achieve its goals seem pretty baseless.

So apparently the issue at the heart of this is a questionable design decision by Google.

Re:Technical aspects

[mjg59]

Some background on this:

Android is written to automatically attempt to enter a sleep state whenever possible. So, for instance, when your phone is in your pocket the hardware is suspended. You get an incoming call. You pick up a phone and press a button. The button press fully wakes up the phone. What stops the phone from suspending again?
Wakelocks are Google's solution to this. Pressing the button takes a named wakelock. While any wakelocks are held, the phone will not go back to sleep. Once the call is complete, userspace can release the wakelock and the phone will suspend again.

The problem here is that your wakelocks are tied to your userspace. Userspace needs to know which wakelocks the kernel takes in order to be able to release them. Which wakelocks you want will depend on how your platform is designed. Google have implemented the wakelocks they need to solve the problems they face - other vendors may wish to use different wakelocks. The end result is that you end up with a kernel that can only usefully run with a specific userland, and userland modifications require kernel modifications. It's broadly like saying that it'd be fine for some kernel drivers to require gnome and some others to require KDE.

What's depressing about this is that it's an entirely unnecessary approach. You don't need the system to fully suspend. Modern ARM hardware supports something generally referred to as retention mode, which is where the hardware enters a power state equivalent to suspend without actually performing a full state transition. Rather than explicitly suspending, if there are no tasks to run then the kernel idle routine will put the CPU into this state. When a task is scheduled to run (either because of a timer or in response to an event), it runs.

The stated objection to this is that applications can't be trusted to behave themselves. If I write an Android application that repaints itself by sleeping for a frame and then drawing, that'll keep waking up the CPU. In the wakelock scenario, the system isn't running and so this isn't a problem. In the retention scenario, it'll keep getting scheduled and battery life will suffer.

There's two easy solutions to this. The first is to use the range timer functionality in the kernel. This allows applications to say "I want to sleep for about a second, and can afford this much inaccuracy". The kernel uses this to attempt to merge wakeups - if an application sleeps for a second and a millisecond later another application does the same, they can both be woken up at once rather than causing two wakeups. But we can also take this to more extreme lengths, for instance by changing the accuracy value to a millennium when the screen is turned off. The application won't get scheduled again until we receive a "wakeup" event and set the timers back. The second approach is even simpler - just SIGSTOP all non-system tasks when we'd otherwise release the wakelocks.

So I wouldn't frame this as refusing to meet Google halfway. People have tried to come up with solutions that would meet the usecases that Google want to deal with and which don't end up forcing people to tie their kernel to their userland. Nokia achieve comparable power consumption without any of the wakelock infrastructure, so it's clearly not a requirement in order to hit these targets. What would you expect a meeting halfway solution to look like?

Re:Technical aspects

[GooberToo]

I love how they blast Google for not being willing to meet halfway, but they're doing exactly the same thing.

It doesn't appear that's the case. It seems Google coded a bunch stuff without consulting other, more knowledgeable kernel developers. Google then announces what they've done and throw it over the wall. The feedback is the design stinks and is very fragile. They also ask why Google didn't leverage the existing infrastructure which is already working, and which seems to have been in place long before Google started, which addresses their very issue. Nokia via their Linux/OMAP ARM efforts don't seem to have trouble using the existing infrastructure.

Google responds by saying the existing infrastructure isn't a complete solution for their needs. The kernel guys ask why Google didn't simply update the existing code to address their specific needs rather than add poorly conceived new code/interfaces. Even the Linux/Nokia guys step forward and state the existing infrastructure is plenty useful and powerful and is easily able to address Google's need with only minor changes. Furthermore, the general consensus is that the required changes can be made such that its fully compatible with their userspace applications.

Google response? Who gives a shit?!?!? We don't need it upstream. Let everyone else fend for themselves. We don't care if it ads bloat and is unusable for anyone else.

Now the kernel guys are left with unmaintained code, which nobody wants, which exists only to add bloat to the kernel, which benefits nobody but Google. Seems to me, Google doesn't care to meet in the middle and they've made it very clear its Google's way or no way. And as result, Android's kernel is more bloated because of it. Seems like a lose-lose for everybody while highlighting just how huge Google's ego can be.

Re:Google

[Temporal]

Erm, no. While I don't consider Google to be a particularly charitable organization, they do regularly open source their products (though mostly minor ones, as you rightly pointed out) when there is no legal obligation on them to do so.

The reason for that is perfectly clear, too: it strengthens the image of Google as both "geeky" and "open" tech company, which are both important parts of Google's public image.

It's not just public image. There's also the fact that Google is a company full of geeks, many of whom are open source fans in their own right.

I was primarily responsible for Google releasing Protocol Buffers. I did it not for the sake of improving my employer's public image, but because I thought it was a useful tool that should be shared, and those around me agreed. Because of the bottom-up nature of decision making at Google -- and given that I was willing to do the work -- I had no trouble pushing this through.

So yeah, it's pretty upsetting to me to see people say things like "Google does only care about OSS when it suits them and drops out instantly when it doesn't.". This kind of statement completely misunderstands how Google even works. This just isn't the kind of company where orders comes down from executives on high with the only motive being profit -- anyone who thinks otherwise obviously doesn't work here.

Honestly, I think the main reason we haven't released more stuff is because it's kind of a lot of work (as I have learned). Dumping code over a wall does not please the open source community -- you have to maintain it; document it; test it on a zillion platforms; answer e-mail from people who think they are not just entitled to your code, but are doing *you* a favor by using it; review patches from college kids who don't really know what they're doing; etc.

(Oblig. disclaimer: These are my own personal opinions; I am not authorized to speak for my employer.)

Re:Google

[Temporal]

But what comes to TFA, the guy was ready to work with Google to put those patches in the main Linux kernel

I don't think the Android team would describe it that way. TFA doesn't really cover their side of the story. As is usually the case when two sides don't agree, the details are complicated. I'm not on Android so I'm not going to attempt to get into it, but they aren't particularly happy with the situation either.