Android and the Linux Kernel Community

An anonymous reader links to Greg Kroah-Hartman's explanation of a rift (hopefully mendable) in the development culture of Google's Linux-based Android OS and the Linux kernel itself. "As the Android kernel code is now gone from the Linux kernel, as of the 2.6.33 kernel release, I'm starting to get a lot of questions about what happened, and what to do next with regards to Android. So here's my opinion on the whole matter ..."

Comments

Re:One sentence to say it all...

[advocate_one]

So then maybe they shouldn't have kicked the android code out of mainline in the first place?

it never made it out of staging into mainline... it wouldn't build as there were items Google never released that it was dependent upon.

Lots of comments on LWN.net's coverage

[chrisd]

If you head over to LWN, we've already gone back and forth on this a bit. http://lwn.net/Articles/372419/. The short form is that if they don't like how we use the kernel, we're unlikely to be accepted upstream. It's all still released as source code to the world, but the mainline is not interested in most of what we've with to the kernel.

Re:Lots of comments on LWN.net's coverage

[Rene S. Hollan]

Okay. I got off my ass and actually looked at waitlock. It isn't, as I thought, a lock with a restoration of execution context when held, so that one can temporarily surrender other locks, and reacquire them. It is a lock on keeping the system out of various low-power states.

Mainline has an (arguably, according to Google-folk, inefficient), lock on guaranteed service latency (low, medium, high?). and Google wants a lock on specific activity level abstractions (idle, suspend, etc.). Did I get that right?

Well, if the present implementation is inefficient (a linear search...), fix that: use an AA-tree or heap, or something.

I suspect the latency time guarantees are coarse at best and probably map to specific power states (idle, suspend, etc.) anyway. I think BOTH models should be supported (heck, you could always map levels to latencies in some tunable fashion, if you had to fake it: "responsiveness SUSPEND" vs. "hardware SUSPEND") but only because there are already standard abstract notions of what IDLE, SUSPEND, and POWEROFF are.

What may have happened is that the existing implementation was inefficient, not quite fitting the expected model, and Google-folk found it faster to roll their own driven by time-pressure, instead of reworking the existing inefficiencies, thinking "heck, we can layer our API over the existing implementation later" and finding the models are more different than thought.

Reconciling that is called "refactoring" and sometimes it breaks APIs as much as one does not want it to.

Now, who has a vested interest in doing that?

Google? Not really. They figure they can handle the overhead of a kernel fork, at least for a while.

Mainline kernel devs? No. The existing model "works".

I'll tell you who: Google's downstream customers. One of them will do it.

Re:Lots of comments on LWN.net's coverage

[Eunuchswear]

Might be rather hard to "clean up". Back in 2008 Mathew Garret wrote:

Google was going to be an interesting case of a large company hiring people both from the embedded world and also the existing Linux development community and then producing an embedded device that was intended to compete with the very best existing platforms. I had high hopes that this combination of factors would result in the Linux community as a whole having a better idea what the constraints and requirements for high-quality power management in the embedded world were, rather than us ending up with another pile of vendor code sitting on an FTP site somewhere in Taiwan that implements its power management by passing tokenised dead mice through a wormhole.

To a certain extent, my hopes were fulfilled. We got a git server in California.

Re:Google

[gyrogeerloose]

It is called Embrace, Extend and Extinguish. I thought it only applies to MS. Well I think, Do no Evil is gone through the window :)

Well, we know what Steve Jobs said about Google's "Don't Be Evil" mantra--"It's bullshit." Or, "a load of crap," depending on your source.

Re:Google

[h4rr4r]

If linux was under the BSD license it would be in the same shape FreeBSD is in, nearly no one using it in the server room.

So sure we have to wait for btrfs instead of using ZFS, but only because SUN decided to make sure they chose an incompatible license.

Re:One sentence to say it all...

[Lumpy]

And what for the interface? Android is at least 9000 years ahead of anything else for a limited attention easy to use touchscreen interface. Plus I get a pile of apps ready to go including navigation.

Re:One sentence to say it all...

[minsk]

it wouldn't build as there were items Google never released that it was dependent upon.

Are you sure that statement is true? It seems inconsistent with other information posted here, and in the LWN discussion.

Technical aspects

[shutdown -p now]

"OMG fork!" and other political issues aside, I think it's interesting to look at the technical side of the problem. What is the exact nature of Google's changes to the kernel, why did they feel they need them, and are they actually a good idea or not? Can someone with kernel hacking experience enlighten us?

I'm particularly curious after reading the comments on LWN, and specifically this:

Kernel developers (including other embedded developers who have achieved good power savings modes) don't believe that the Android way of doing things is good.

and this:

The code could be mainlined if Google were willing to consider that their wakelock approach was suboptimal and adapt to a more reasonable one. ...

The wakelock patches were first posted to the linux-pm list on the 13th of January 2009, which is just over a year ago. http://lwn.net/Articles/318611/ gives a good overview of how they were received - there's basically no buy in, even from other developers in the embedded Linux field. ...

the major sticking point (ie, wakelocks) were posted for public review a year ago and most of the substantive issues people had with them weren't addressed at all in the four months of intermittent discussion that followed. If the entire world suggests that you do something in some other way and you refuse to, that doesn't constitute a genuine effort to work with them. Nokia have managed to obtain the same level of power management without such invasive changes, so any assertion that wakelocks are required for Android to achieve its goals seem pretty baseless.

So apparently the issue at the heart of this is a questionable design decision by Google.

Re:Technical aspects

[mjg59]

Some background on this:

Android is written to automatically attempt to enter a sleep state whenever possible. So, for instance, when your phone is in your pocket the hardware is suspended. You get an incoming call. You pick up a phone and press a button. The button press fully wakes up the phone. What stops the phone from suspending again?
Wakelocks are Google's solution to this. Pressing the button takes a named wakelock. While any wakelocks are held, the phone will not go back to sleep. Once the call is complete, userspace can release the wakelock and the phone will suspend again.

The problem here is that your wakelocks are tied to your userspace. Userspace needs to know which wakelocks the kernel takes in order to be able to release them. Which wakelocks you want will depend on how your platform is designed. Google have implemented the wakelocks they need to solve the problems they face - other vendors may wish to use different wakelocks. The end result is that you end up with a kernel that can only usefully run with a specific userland, and userland modifications require kernel modifications. It's broadly like saying that it'd be fine for some kernel drivers to require gnome and some others to require KDE.

What's depressing about this is that it's an entirely unnecessary approach. You don't need the system to fully suspend. Modern ARM hardware supports something generally referred to as retention mode, which is where the hardware enters a power state equivalent to suspend without actually performing a full state transition. Rather than explicitly suspending, if there are no tasks to run then the kernel idle routine will put the CPU into this state. When a task is scheduled to run (either because of a timer or in response to an event), it runs.

The stated objection to this is that applications can't be trusted to behave themselves. If I write an Android application that repaints itself by sleeping for a frame and then drawing, that'll keep waking up the CPU. In the wakelock scenario, the system isn't running and so this isn't a problem. In the retention scenario, it'll keep getting scheduled and battery life will suffer.

There's two easy solutions to this. The first is to use the range timer functionality in the kernel. This allows applications to say "I want to sleep for about a second, and can afford this much inaccuracy". The kernel uses this to attempt to merge wakeups - if an application sleeps for a second and a millisecond later another application does the same, they can both be woken up at once rather than causing two wakeups. But we can also take this to more extreme lengths, for instance by changing the accuracy value to a millennium when the screen is turned off. The application won't get scheduled again until we receive a "wakeup" event and set the timers back. The second approach is even simpler - just SIGSTOP all non-system tasks when we'd otherwise release the wakelocks.

So I wouldn't frame this as refusing to meet Google halfway. People have tried to come up with solutions that would meet the usecases that Google want to deal with and which don't end up forcing people to tie their kernel to their userland. Nokia achieve comparable power consumption without any of the wakelock infrastructure, so it's clearly not a requirement in order to hit these targets. What would you expect a meeting halfway solution to look like?

Re:Technical aspects

[GooberToo]

I love how they blast Google for not being willing to meet halfway, but they're doing exactly the same thing.

It doesn't appear that's the case. It seems Google coded a bunch stuff without consulting other, more knowledgeable kernel developers. Google then announces what they've done and throw it over the wall. The feedback is the design stinks and is very fragile. They also ask why Google didn't leverage the existing infrastructure which is already working, and which seems to have been in place long before Google started, which addresses their very issue. Nokia via their Linux/OMAP ARM efforts don't seem to have trouble using the existing infrastructure.

Google responds by saying the existing infrastructure isn't a complete solution for their needs. The kernel guys ask why Google didn't simply update the existing code to address their specific needs rather than add poorly conceived new code/interfaces. Even the Linux/Nokia guys step forward and state the existing infrastructure is plenty useful and powerful and is easily able to address Google's need with only minor changes. Furthermore, the general consensus is that the required changes can be made such that its fully compatible with their userspace applications.

Google response? Who gives a shit?!?!? We don't need it upstream. Let everyone else fend for themselves. We don't care if it ads bloat and is unusable for anyone else.

Now the kernel guys are left with unmaintained code, which nobody wants, which exists only to add bloat to the kernel, which benefits nobody but Google. Seems to me, Google doesn't care to meet in the middle and they've made it very clear its Google's way or no way. And as result, Android's kernel is more bloated because of it. Seems like a lose-lose for everybody while highlighting just how huge Google's ego can be.

Re:Google has just TIVO-ized the kernel

[AndrewNeo]

Hardly. Nobody's forcing you to run their binaries on your phone. You can still compile the kernel and other applications from code, forked or not.

Re:Google

[Temporal]

Erm, no. While I don't consider Google to be a particularly charitable organization, they do regularly open source their products (though mostly minor ones, as you rightly pointed out) when there is no legal obligation on them to do so.

The reason for that is perfectly clear, too: it strengthens the image of Google as both "geeky" and "open" tech company, which are both important parts of Google's public image.

It's not just public image. There's also the fact that Google is a company full of geeks, many of whom are open source fans in their own right.

I was primarily responsible for Google releasing Protocol Buffers. I did it not for the sake of improving my employer's public image, but because I thought it was a useful tool that should be shared, and those around me agreed. Because of the bottom-up nature of decision making at Google -- and given that I was willing to do the work -- I had no trouble pushing this through.

So yeah, it's pretty upsetting to me to see people say things like "Google does only care about OSS when it suits them and drops out instantly when it doesn't.". This kind of statement completely misunderstands how Google even works. This just isn't the kind of company where orders comes down from executives on high with the only motive being profit -- anyone who thinks otherwise obviously doesn't work here.

Honestly, I think the main reason we haven't released more stuff is because it's kind of a lot of work (as I have learned). Dumping code over a wall does not please the open source community -- you have to maintain it; document it; test it on a zillion platforms; answer e-mail from people who think they are not just entitled to your code, but are doing *you* a favor by using it; review patches from college kids who don't really know what they're doing; etc.

(Oblig. disclaimer: These are my own personal opinions; I am not authorized to speak for my employer.)

In summary...

[rickb928]

Google has silently forked the kernel. There is an 'Android' kernel, and the mainline kernel

Is this the first time this has happened?

Will it matter?

Apparently, this is reasonably well understood.

I, for one, welcome our Android kernel overlords. My phone doesn't need server optimizations.

Re:Google

[Thantik]

Care to explain why I don't see symbian or rim browsers on that list at all? It's because those are WEB BROWSER STATISTICS...Oh, yeah, that's why you posted as AC.

The mobile phone market in regards to *web page statistics* is a horribly inaccurate way to measure penetration of said devices. Sure, I can pull up any damn statistic I want and say that it means what I want it to mean, but it doesn't mean that it holds water.

I'd guess there is some very close competition in terms of sales of *NEW* devices, between iPhone and Android. The benefit android has is that you can chose ANY carrier you wish. iPhone your just stuck with AT&T (in the USA) and my own anecdotal evidence amongst friends shows that they're ditching iPhone in droves. But that's just anecdotal evidence. Linux is indeed a contender.

Re:Google

[MichaelSmith]

Eh, trolling much? If you look outside US, Nokia is dominating. iPhone is nowhere as successful as it is in US. In top of that, Nokia holds patents (that they really deserve) over many technologies used with GSM, 3G and so on.

And Nokia offers a real Linux phone, not just Android or locked-down iPhone shit.

Well I live in Australia and the iPhone is very popular. Most people who want a smart phone will have an iPhone. Many older people who just want to make calls will have a $50 nokia. I use an openmoko, which is also a real Linux phone.

Re:Google

[Temporal]

But what comes to TFA, the guy was ready to work with Google to put those patches in the main Linux kernel

I don't think the Android team would describe it that way. TFA doesn't really cover their side of the story. As is usually the case when two sides don't agree, the details are complicated. I'm not on Android so I'm not going to attempt to get into it, but they aren't particularly happy with the situation either.

Re:Google

[unfunk]

As I said, the only groups being hurt by this are Google and those dumb enough to rely on Android for their future, anyone else with a brain will take a look at the competition and more open platforms.

Newsflash: Consumers don't care about whether Google's playing nice with the Linux community or not.

Re:One sentence to say it all...

[jeremyp]

How is this +5 informative? It's completely false.

The reason it didn't make it to mainline is because the Google code was reviewed and found to have problems that stopped it being accepted into mainline. Because there are user space items in Android that would be affected, only Google could make the changes without breaking Android.

Think about it, if you were unable to build the Android kernel because Google were withholding stuff, it would be in direct violation of GPL v2. Do you see Greg KH complaining that Android violates the Linux licence? No.

Re:up merge justification has to be Android-agnost

[Chousuke]

I'm not sure what you mean by "rapidly changing driver/userlevel API". The kernel userspace backwards compatibility may not be broken, so it's quite stable compared to in-kernel interfaces.

If you mean that the interfaces *need* to change, then perhaps those parts of the drivers should not be in the kernel (as is the case for 3d drivers)

I'm not at all qualified to talk about virtualisation, but regarding 3d drivers, my understanding is that a significant portion of the stack actually lives in MESA, not the kernel.

The kernel side's responsibility (assuming a "modern" one instead of the traditional "X does everything" kind of driver) is to handle details the kernel cares about such as initialisation, mode-setting, power and memory management, while providing a mostly static interface to the userspace component of the driver, which contains most of the messy 3d stuff.

I think malleable in-kernel APIs allow your driver *not* to be perfect when you merge it. Perfection is required only for any user-space interfaces.

Furthermore, if your driver suffers from bad design in some kernel subsystem, it is possible go and fix things. Or conversely, if some driver depends on an interface that you wrote, but you need to change it, that is also feasible.

Now, one might argue that it's possible to preserve the old kernel APIs, but while that is possible, it greatly increases the maintenance burden. You will need to test code that only outdated drivers use. That's a waste of resources.

Re:Google

[salesgeek]

Wrong. Let's not act like a product category that had been around for 10 years prior to iPhone was suddenly created by Apple. Apple did not create the smart phone. They did not even define the product:

Palm, Blackberry and Microsoft (to a lesser extent) defined the product. Apple came out with the best model in the 2008 model year. That is the extent of the achievement, and to give Apple more credit is simply an insult to the people that created iconic products like the Blackberry, Treo and to a lesser degree WinMo smartphones that defined the platform that Apple added incremental improvements to.

It is telling that even my lowly G-1 has more features than the same generation iPhone (keyboard, removable battery, 3G, removable memory, metal detector, etc...).

Finally, Android has *already won the war* with Apple. It's pretty much exactly like Mac vs. PC in the early 90s. Developers who pass on Android for iPhone will be seen as shortsighted by the end of this year as marketshare expands.