Android and the Linux Kernel Community

An anonymous reader links to Greg Kroah-Hartman's explanation of a rift (hopefully mendable) in the development culture of Google's Linux-based Android OS and the Linux kernel itself. "As the Android kernel code is now gone from the Linux kernel, as of the 2.6.33 kernel release, I'm starting to get a lot of questions about what happened, and what to do next with regards to Android. So here's my opinion on the whole matter ..."

Comments

One sentence to say it all...

[Jorl17]

"Because of this, Google has now prevented a large chunk of hardware drivers and platform code from ever getting merged into the main kernel tree. Effectively creating a kernel branch that a number of different vendors are now relying on."

That's all. It's obvious that Google doesn't care about it that much. And yet nobody demanded them to do so -- if Google wants it its own way, why shouldn't it be able to?
I may be a crazy open-source lunatic, but I am tired of all of this "It's a world conspiracy against Linux"-thing. Let's get a grip, talk less and code more.

Re:One sentence to say it all...

[minsk]

it wouldn't build as there were items Google never released that it was dependent upon.

Are you sure that statement is true? It seems inconsistent with other information posted here, and in the LWN discussion.

Re:Google

[killmenow]

Apparently Google employees have mod points today.

Re:Google

[Colin Smith]

Google has a major advantage here as one of the largest companies in the world

Nokia has the major advantage that they are *the* largest phone producer on the face of the planet and have *the* largest world market share by a large percentage. Google and particularly, Android are small fry in comparison, despite their size in other markets. Those companies jumping on Android are dumb because their horizon is limited to the US market and a single platform.

For irony, Google "Maemo" and "Nokia N900".

As I said, the only groups being hurt by this are Google and those dumb enough to rely on Android for their future, anyone else with a brain will take a look at the competition and more open platforms.
 

Lots of comments on LWN.net's coverage

[chrisd]

If you head over to LWN, we've already gone back and forth on this a bit. http://lwn.net/Articles/372419/. The short form is that if they don't like how we use the kernel, we're unlikely to be accepted upstream. It's all still released as source code to the world, but the mainline is not interested in most of what we've with to the kernel.

Re:Lots of comments on LWN.net's coverage

[Rene S. Hollan]

Okay. I got off my ass and actually looked at waitlock. It isn't, as I thought, a lock with a restoration of execution context when held, so that one can temporarily surrender other locks, and reacquire them. It is a lock on keeping the system out of various low-power states.

Mainline has an (arguably, according to Google-folk, inefficient), lock on guaranteed service latency (low, medium, high?). and Google wants a lock on specific activity level abstractions (idle, suspend, etc.). Did I get that right?

Well, if the present implementation is inefficient (a linear search...), fix that: use an AA-tree or heap, or something.

I suspect the latency time guarantees are coarse at best and probably map to specific power states (idle, suspend, etc.) anyway. I think BOTH models should be supported (heck, you could always map levels to latencies in some tunable fashion, if you had to fake it: "responsiveness SUSPEND" vs. "hardware SUSPEND") but only because there are already standard abstract notions of what IDLE, SUSPEND, and POWEROFF are.

What may have happened is that the existing implementation was inefficient, not quite fitting the expected model, and Google-folk found it faster to roll their own driven by time-pressure, instead of reworking the existing inefficiencies, thinking "heck, we can layer our API over the existing implementation later" and finding the models are more different than thought.

Reconciling that is called "refactoring" and sometimes it breaks APIs as much as one does not want it to.

Now, who has a vested interest in doing that?

Google? Not really. They figure they can handle the overhead of a kernel fork, at least for a while.

Mainline kernel devs? No. The existing model "works".

I'll tell you who: Google's downstream customers. One of them will do it.

Re:Google

[gyrogeerloose]

It is called Embrace, Extend and Extinguish. I thought it only applies to MS. Well I think, Do no Evil is gone through the window :)

Well, we know what Steve Jobs said about Google's "Don't Be Evil" mantra--"It's bullshit." Or, "a load of crap," depending on your source.

Re:Google

[Colin Smith]

Is that why we already succeeded at Year Of Linux On Desktop back in 2003?

Haven't you noticed? The desktop is irrelevant. It's been abstracted to an Internet access platform. It's the phone in the pocket which is the current battleground, and Linux has won that already.

 

Re:Google

[GenP]

I want ZFS for my home server, zfs is not the answer

Very Zen.

No, I don't think so

[Concern]

Google has frankly set a new standard as far as how companies can become very successful by embracing the open and free software communities. I honestly don't think you can point to many other companies that are doing better, nor could you realistically expect to. In the mobile space, pretty much the next nearest competitor in terms of openness is Apple (Darwin, et al) - in other words, a joke. Meanwhile Google not only has a wonderfully organized system for playing with all the Android code, but a broad commitment across products. Look at Wave, for instance - wide, wide open, and very deliberately (because they know it cannot succeed any other way). Google has probably done more for Linux and its credibility than most other companies in the world.

I think this is something totally different - a disagreement about direction between the mainline maintainers and Google's Android team. Corporate developers, even well-intentioned ones, have a conceptual hurdle to get over when someone Not Their Manager is telling them "you must spend x man-months refactoring your code thusly."

Many, many companies have run into this issue with Linux (and other projects) before, and many will again. It usually goes something like this:

Step 1: Whatever. We're Google. Am I going to rearrange my whole development roadmap to follow the directions of some whiny nerd in his mom's basement? LOL.
Step 2: Oh. Crap. Wow it is kind of a lot work maintaining my own entire fork of the Linux kernel/KHTML/etc. all by myself.
Step 3: Either A) capitulation - the last guy is fired or smacked with the clue stick, and the cooperation restarts, or B) a true fork. These usually stagnate and die, and are also riddled with bugs and security holes btw... unless, the fork is really more interesting than what it forked from, in which case, the community switches to the fork and justice is served.

Often between Step 1 and 2, the maintainer will attempt to play a little corporate politics by embarrassing said middle manager in the media. By the way, this is pretty smart and it often works - especially with companies as large but otherwise savvy as Google, a slashdot story can jumpstart efforts to mend the rift by bringing more senior eyes on the problem. Cooperating is in everyone's interest, and they will realize it.

Technical aspects

[shutdown -p now]

"OMG fork!" and other political issues aside, I think it's interesting to look at the technical side of the problem. What is the exact nature of Google's changes to the kernel, why did they feel they need them, and are they actually a good idea or not? Can someone with kernel hacking experience enlighten us?

I'm particularly curious after reading the comments on LWN, and specifically this:

Kernel developers (including other embedded developers who have achieved good power savings modes) don't believe that the Android way of doing things is good.

and this:

The code could be mainlined if Google were willing to consider that their wakelock approach was suboptimal and adapt to a more reasonable one. ...

The wakelock patches were first posted to the linux-pm list on the 13th of January 2009, which is just over a year ago. http://lwn.net/Articles/318611/ gives a good overview of how they were received - there's basically no buy in, even from other developers in the embedded Linux field. ...

the major sticking point (ie, wakelocks) were posted for public review a year ago and most of the substantive issues people had with them weren't addressed at all in the four months of intermittent discussion that followed. If the entire world suggests that you do something in some other way and you refuse to, that doesn't constitute a genuine effort to work with them. Nokia have managed to obtain the same level of power management without such invasive changes, so any assertion that wakelocks are required for Android to achieve its goals seem pretty baseless.

So apparently the issue at the heart of this is a questionable design decision by Google.

Re:Technical aspects

[mjg59]

Some background on this:

Android is written to automatically attempt to enter a sleep state whenever possible. So, for instance, when your phone is in your pocket the hardware is suspended. You get an incoming call. You pick up a phone and press a button. The button press fully wakes up the phone. What stops the phone from suspending again?
Wakelocks are Google's solution to this. Pressing the button takes a named wakelock. While any wakelocks are held, the phone will not go back to sleep. Once the call is complete, userspace can release the wakelock and the phone will suspend again.

The problem here is that your wakelocks are tied to your userspace. Userspace needs to know which wakelocks the kernel takes in order to be able to release them. Which wakelocks you want will depend on how your platform is designed. Google have implemented the wakelocks they need to solve the problems they face - other vendors may wish to use different wakelocks. The end result is that you end up with a kernel that can only usefully run with a specific userland, and userland modifications require kernel modifications. It's broadly like saying that it'd be fine for some kernel drivers to require gnome and some others to require KDE.

What's depressing about this is that it's an entirely unnecessary approach. You don't need the system to fully suspend. Modern ARM hardware supports something generally referred to as retention mode, which is where the hardware enters a power state equivalent to suspend without actually performing a full state transition. Rather than explicitly suspending, if there are no tasks to run then the kernel idle routine will put the CPU into this state. When a task is scheduled to run (either because of a timer or in response to an event), it runs.

The stated objection to this is that applications can't be trusted to behave themselves. If I write an Android application that repaints itself by sleeping for a frame and then drawing, that'll keep waking up the CPU. In the wakelock scenario, the system isn't running and so this isn't a problem. In the retention scenario, it'll keep getting scheduled and battery life will suffer.

There's two easy solutions to this. The first is to use the range timer functionality in the kernel. This allows applications to say "I want to sleep for about a second, and can afford this much inaccuracy". The kernel uses this to attempt to merge wakeups - if an application sleeps for a second and a millisecond later another application does the same, they can both be woken up at once rather than causing two wakeups. But we can also take this to more extreme lengths, for instance by changing the accuracy value to a millennium when the screen is turned off. The application won't get scheduled again until we receive a "wakeup" event and set the timers back. The second approach is even simpler - just SIGSTOP all non-system tasks when we'd otherwise release the wakelocks.

So I wouldn't frame this as refusing to meet Google halfway. People have tried to come up with solutions that would meet the usecases that Google want to deal with and which don't end up forcing people to tie their kernel to their userland. Nokia achieve comparable power consumption without any of the wakelock infrastructure, so it's clearly not a requirement in order to hit these targets. What would you expect a meeting halfway solution to look like?

Re:Technical aspects

[GooberToo]

I love how they blast Google for not being willing to meet halfway, but they're doing exactly the same thing.

It doesn't appear that's the case. It seems Google coded a bunch stuff without consulting other, more knowledgeable kernel developers. Google then announces what they've done and throw it over the wall. The feedback is the design stinks and is very fragile. They also ask why Google didn't leverage the existing infrastructure which is already working, and which seems to have been in place long before Google started, which addresses their very issue. Nokia via their Linux/OMAP ARM efforts don't seem to have trouble using the existing infrastructure.

Google responds by saying the existing infrastructure isn't a complete solution for their needs. The kernel guys ask why Google didn't simply update the existing code to address their specific needs rather than add poorly conceived new code/interfaces. Even the Linux/Nokia guys step forward and state the existing infrastructure is plenty useful and powerful and is easily able to address Google's need with only minor changes. Furthermore, the general consensus is that the required changes can be made such that its fully compatible with their userspace applications.

Google response? Who gives a shit?!?!? We don't need it upstream. Let everyone else fend for themselves. We don't care if it ads bloat and is unusable for anyone else.

Now the kernel guys are left with unmaintained code, which nobody wants, which exists only to add bloat to the kernel, which benefits nobody but Google. Seems to me, Google doesn't care to meet in the middle and they've made it very clear its Google's way or no way. And as result, Android's kernel is more bloated because of it. Seems like a lose-lose for everybody while highlighting just how huge Google's ego can be.

Re:Technical aspects

[Anonymous Coward]

I'm commenting anonymously because I'm closely involved with Android. I'm not going to comment on whether wakelocks are good or bad.

But mjg59's comment about ARM retention state being as good as suspend is blatantly wrong at a SoC level. There are several good reasons why a suspend details is better than retention but I can't go into the details due to NDAs. But I can vouch that I have personally seen power usage shoot up on well known Android phones when suspend is disabled and only retention is done.

So, mjg59, I would kindly request that you stop making such claims. I doubt you have worked on embedded devices -- I believe you work on server level stuff -- but you most certainly haven't worked extensively on the SoCs being used on Android phones.

Just to make my stance clear, I'm all for having the Android kernel merged into mainline.

Re:Technical aspects

[pslam]

So, mjg59, I would kindly request that you stop making such claims. I doubt you have worked on embedded devices -- I believe you work on server level stuff -- but you most certainly haven't worked extensively on the SoCs being used on Android phones.

I have worked on embedded devices - Linux based and others - for over a decade, and mjg59 is right. If you need something as brutal as hard suspend mode, you're doing it wrong. Any serious low power optimized ARM SoC can run down to very low current when idle, and has peripherals which can be individually clocked down and/or gated. I did work in a periphery way on the G1 at Google, and was very surprised at the way power gating was done. Put simply: the only other embedded OS in the same class as Android which does power gating like this is Windows Mobile. Everyone else learned it was unnecessary a long time ago. I was fairly shocked how few engineers had ever done serious embedded work before, and the result shows.

I know the Qualcomm parts have horrendously stupid design decisions in them which prevent decent idle current, but it's a wash compared to the other sources of battery drain. It's also a wash compared to the damage it does to your code design to support full suspend as part of normal per-second operation.

The Linux maintainers are right: Android is just doing it the wrong way. If there's any one feature I think Android could have done without, it's wake-locks. Learn how to use fine-grained clock switching and gating, not brutally-coarse-grained suspend. This isn't a bloody laptop. And no, I'm not saying this as a back seat driver: I really have done this kind of crap for a decade.

Re:Google

[Temporal]

Erm, no. While I don't consider Google to be a particularly charitable organization, they do regularly open source their products (though mostly minor ones, as you rightly pointed out) when there is no legal obligation on them to do so.

The reason for that is perfectly clear, too: it strengthens the image of Google as both "geeky" and "open" tech company, which are both important parts of Google's public image.

It's not just public image. There's also the fact that Google is a company full of geeks, many of whom are open source fans in their own right.

I was primarily responsible for Google releasing Protocol Buffers. I did it not for the sake of improving my employer's public image, but because I thought it was a useful tool that should be shared, and those around me agreed. Because of the bottom-up nature of decision making at Google -- and given that I was willing to do the work -- I had no trouble pushing this through.

So yeah, it's pretty upsetting to me to see people say things like "Google does only care about OSS when it suits them and drops out instantly when it doesn't.". This kind of statement completely misunderstands how Google even works. This just isn't the kind of company where orders comes down from executives on high with the only motive being profit -- anyone who thinks otherwise obviously doesn't work here.

Honestly, I think the main reason we haven't released more stuff is because it's kind of a lot of work (as I have learned). Dumping code over a wall does not please the open source community -- you have to maintain it; document it; test it on a zillion platforms; answer e-mail from people who think they are not just entitled to your code, but are doing *you* a favor by using it; review patches from college kids who don't really know what they're doing; etc.

(Oblig. disclaimer: These are my own personal opinions; I am not authorized to speak for my employer.)

Re:Google

[Microlith]

And how is Android not open? The entire thing is FOSS.

I wondered that myself when looking into the N900, and the explanation as to why Android is not open is because while it is Open Source, it does not have an open community.

If you look at the OHA FAQ, they explain why an open platform is good for everyone EXCEPT the end-user. Unless you mistake who your customer is, you'd realize that the end-user's input is just as important as those who apply your OS to their device. Both the OHA and the cell vendors make that mistake, as the OHA thinks the user of Android is the handset maker, who themselves think their customer is the carrier. The only true customer is the person who pays for and uses the device in the end, and they should be able to have input into their device and its OS should they care.