Slashdot Mirror
Android and the Linux Kernel Community
An anonymous reader links to Greg Kroah-Hartman's explanation of a rift (hopefully mendable) in the development culture of Google's Linux-based Android OS and the Linux kernel itself. "As the Android kernel code is now gone from the Linux kernel, as of the 2.6.33 kernel release, I'm starting to get a lot of questions about what happened, and what to do next with regards to Android. So here's my opinion on the whole matter ..."
Because Google doesn't have their code merged into the mainline, these companies creating drivers and platform code are locked out from ever contributing it back to the kernel community.
Google shows no sign of working to get their code upstream anymore.
Oh come on, was it really a surprise to anyone that Google does only care about OSS when it suits them and drops out instantly when it doesn't. All of their own sites, business and back-end technology is just as closed as Microsoft's.
I see someone coming along and saying "but they contribute to open source!". Sure, they do, they release little snippets of code and open source those products they base on OSS code because they have to by GPL. One could seriously argue if their open sourcing efforts are making better open source community in general, or not. Like TFA states, their ignorance has caused more turmoil than ever before in Linux land. Companies are obviously going to create support and drivers for Android-branch of Linux kernel, but cant contribute the same code back to real Linux kernel. And possibly never will because it costs them too much work, money and time. Even those companies that previously did develop linux drivers. That's not harming Linux and OSS community?
Get off your lazy ass and see what's really happening.
"Because of this, Google has now prevented a large chunk of hardware drivers and platform code from ever getting merged into the main kernel tree. Effectively creating a kernel branch that a number of different vendors are now relying on."
That's all. It's obvious that Google doesn't care about it that much. And yet nobody demanded them to do so -- if Google wants it its own way, why shouldn't it be able to?
I may be a crazy open-source lunatic, but I am tired of all of this "It's a world conspiracy against Linux"-thing. Let's get a grip, talk less and code more.
Have you heard about SoylentNews?
You don't merge new features into a robust low-level code base.
You merge support for abstractions the new features rely upon into the low-level code base, and build on them.
Make a case for kernel support of the "new lock" and "security model" independent of Android's reliance on it, and remove as many of the Android drivers using these facilities OUT of the main tree. IMHO, drivers really don't belong there, but should be available in "driver package sets" aggregated by distro providers.
In Liberty, Rene
This isn't about some anti-Linux conspiracy, it's about conflicting business interests.
Google has made modifications to make their software work. Those modifications can not be integrated as is with the kernel. Cleaning up their open source code can be done, but their internal closed code would likely also need modifications to account for that cleanup. So for Google, merging their code would likely be expensive and require extensive testing and a large download and upgrade system for all of the live phones.
On the other hand, companies that are developing drivers and software that depend on the new kernel are now hosed because the kernel is no longer exposed as part of the Linux kernel. So they have to deal directly with Google for any changes to their kernel, and if they also want to release their products for other Linux platforms, they have to re-engineer much of the os-interaction functionality. Obviously a pricey situation for them.
If Google updates their code to get it into the kernel, it makes it cheaper and easier for other developers to work it, but it costs them money. If they don't then it costs the developers more money.
-Rick
"Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
You can use Android, but you're now dependent on Google for maintenance, bug-fixes, and trusting that they don't include any evil bits.
Courtesy of the google translator ...
Still no idea what it meant. :-P
Lost at C:>. Found at C.
If you head over to LWN, we've already gone back and forth on this a bit. http://lwn.net/Articles/372419/. The short form is that if they don't like how we use the kernel, we're unlikely to be accepted upstream. It's all still released as source code to the world, but the mainline is not interested in most of what we've with to the kernel.
Co-Editor, Open Sources
Open Source Program Manager, Google, Inc.
Google has frankly set a new standard as far as how companies can become very successful by embracing the open and free software communities. I honestly don't think you can point to many other companies that are doing better, nor could you realistically expect to. In the mobile space, pretty much the next nearest competitor in terms of openness is Apple (Darwin, et al) - in other words, a joke. Meanwhile Google not only has a wonderfully organized system for playing with all the Android code, but a broad commitment across products. Look at Wave, for instance - wide, wide open, and very deliberately (because they know it cannot succeed any other way). Google has probably done more for Linux and its credibility than most other companies in the world.
I think this is something totally different - a disagreement about direction between the mainline maintainers and Google's Android team. Corporate developers, even well-intentioned ones, have a conceptual hurdle to get over when someone Not Their Manager is telling them "you must spend x man-months refactoring your code thusly."
Many, many companies have run into this issue with Linux (and other projects) before, and many will again. It usually goes something like this:
Step 1: Whatever. We're Google. Am I going to rearrange my whole development roadmap to follow the directions of some whiny nerd in his mom's basement? LOL.
Step 2: Oh. Crap. Wow it is kind of a lot work maintaining my own entire fork of the Linux kernel/KHTML/etc. all by myself.
Step 3: Either A) capitulation - the last guy is fired or smacked with the clue stick, and the cooperation restarts, or B) a true fork. These usually stagnate and die, and are also riddled with bugs and security holes btw... unless, the fork is really more interesting than what it forked from, in which case, the community switches to the fork and justice is served.
Often between Step 1 and 2, the maintainer will attempt to play a little corporate politics by embarrassing said middle manager in the media. By the way, this is pretty smart and it often works - especially with companies as large but otherwise savvy as Google, a slashdot story can jumpstart efforts to mend the rift by bringing more senior eyes on the problem. Cooperating is in everyone's interest, and they will realize it.
Tired of Political Trolls? Opt Out!
Somehow I managed to forget Nokia for being more open than Apple - and arguably - Google. I guess because so few people use, or will likely ever use, their smartphones. :)
Tired of Political Trolls? Opt Out!
Make a case for kernel support of the "new lock" and "security model" independent of Android's reliance on it, and remove as many of the Android drivers using these facilities OUT of the main tree.
That's exactly right, we can't just be having code merged into the kernel mainline because a 'big business' depends on it. The answer to the question on it's presence there should be determined on the merit of the code itself, not the weight of the company proposing it.
"OMG fork!" and other political issues aside, I think it's interesting to look at the technical side of the problem. What is the exact nature of Google's changes to the kernel, why did they feel they need them, and are they actually a good idea or not? Can someone with kernel hacking experience enlighten us?
I'm particularly curious after reading the comments on LWN, and specifically this:
Kernel developers (including other embedded developers who have achieved good power savings modes) don't believe that the Android way of doing things is good.
and this:
The code could be mainlined if Google were willing to consider that their wakelock approach was suboptimal and adapt to a more reasonable one. ...
The wakelock patches were first posted to the linux-pm list on the 13th of January 2009, which is just over a year ago. http://lwn.net/Articles/318611/ gives a good overview of how they were received - there's basically no buy in, even from other developers in the embedded Linux field. ...
the major sticking point (ie, wakelocks) were posted for public review a year ago and most of the substantive issues people had with them weren't addressed at all in the four months of intermittent discussion that followed. If the entire world suggests that you do something in some other way and you refuse to, that doesn't constitute a genuine effort to work with them. Nokia have managed to obtain the same level of power management without such invasive changes, so any assertion that wakelocks are required for Android to achieve its goals seem pretty baseless.
So apparently the issue at the heart of this is a questionable design decision by Google.
"Embrace & extend"!
Has there ever really been a serious option B effort to just pick a time and do a major kernel fork and maintain it forever by any company as large as google, with their level of developers and resources/cash behind them? Any precedent there?
I suspect this isn't clear cut and there is more than a little "Google is too big we must hate them now" attitude. The fact is Android is open source and anyone can take their code do what they want with it just as Google can take any other open source code and do what they want with it.
That's the point of open source code. Claiming something is open and all about freedom and then trying force everyone into doing what you think should be done is neither really free or open.
"Redhat Enterprise Linux 5" is essentially a massive kernel fork at 2.6.18. Redhat is the most plausible contender for doing this, since they employ a really significant number of the world's kernel devs, including Alan Cox (until last year). That split was even acrimonious at times IIRC. But then again, you can just call it similar to Canonical's LTS - people choose whether to go with a version of something that's more or less mature - and most distros going with the "more mature" option frequently cheat and backport all kinds of things in the meantime (with greater and lesser success).
Depending on who you ask, RHEL can be more risky than mainline. I've definitely had RHEL panics take down production, only to later discover linux kernel bugs that had been fixed in mainline for a while, but that redhat hadn't backported to their ancient linux fork. But then again, people get burned going the other way all the time too. I don't know if anyone's really independently studied what's ultimately safest. My guess is that you are usually safest inside the biggest crowd - i.e. closer to mainline - but not too near the very latest version.
Tired of Political Trolls? Opt Out!
Google has silently forked the kernel. There is an 'Android' kernel, and the mainline kernel
Is this the first time this has happened?
Will it matter?
Apparently, this is reasonably well understood.
I, for one, welcome our Android kernel overlords. My phone doesn't need server optimizations.
deleting the extra space after periods so i can stay relevant, yeah.
Kernel development isn't driven by the product needs but by egos people maintaining it. In that sense it is no different to any closed source kernel. The only option to get things done quickly is to fork it and then hope to get it merged later. Linus isn't exactly known as "accommodating" and "reaching out" individual and merging code upstream isn't exactly a priority on Google product schedule so there.
I can't see what the issue is, if Google wants to make their own fork of Linux go for it, that is what FOSS is about, "CHOICE!!". Apple have done this with their own version of BSD (Darwin) and added lots of things not in the mainstream flavours of BSD and they are doing fine thank you.
Android is open source so does it really matter?
It's not black-and-white. It's better then Android being closed source, but it's not as good as it could be. It'd have been better (for the public, not necessarily Google) if Google got the android-specific changes to the Linux kernel in the mainline kernel. This would require quite a bit of back-and-forth with the kernel devs, though, and a lot more time and money then Google would truly have to spend to get Android on the market.
"A witty saying proves nothing." - Voltaire
the french reply means "Does anybody understand what AC said ? I am really clueless"
What that looked like to me was someone who was furious that a person who self-taught himself C in less than a few years implemented a kernel scheduler that kicked the crap out of anything the community came up with to date.
No way that dude could be better than me...
Ego's abound everywhere, my friend...
GPL says contribute your changes. Doesn't say one thing about contribute your changes, and make sure they work in the main tree. He himself said they were doing it, until people started complaining about how things were written or what not. So they stopped. I don't blame them one bit. It is kind of like writing a library to be in PEAR, you have a piece of code that works. And and you see people asking for a library and you submit it for a vote, they first don't like that you used the PHP license, they want you to use the BSD license, then they don't like your naming of the methods. Hell, has the people running PEAR, looked at the naming conventions of PHP (addslashes, mysql_connect) I basically said flip that and put it on google code. There are forks all the time in the linux kernel, they usually die out, but wouldn't it be fun if googles kernel was the only one to survive?
http://www.codestrom.com/wandering/2009/03/zfs-vs-btrfs-comparison.html
help humanity get knowledge inovation
Man, if you have posted here 6 years, just please try to learn to type like a normal person. No one would be trolling you if you didn't post like this. Your text is really, really hard to read. It itself comes out as trollish. Replace the @'s with at and &'s with "and" and don't use so many unnecessary ". Your points might be good but no one bothers to read them like this.
Cell phones need a different kernel than servers, with different drivers, graphics and security requirements. Maybe this fork is justified? Maybe this doesn't need to be in the mainline?
We've had no improvements in real productivity on the desktop in 15 years. It's long past time it was replaced with something else. Since we've tried WinTel, and that gave no progress, it's time we tried another path.
If it turns out that these new platforms deliver the same productivity or better on 1% of the watts and 10% of the capital cost, they'll drive a revolution in IT that shifts influence from the people who have unlimited watts (the US) to more fairly share influence with people who are Watt and capital challenged (India, Pakistan and others). That's an advantage to the new markets actually, since they leverage the volume production of innovations they haven't paid for. And then there's the side-effect that power efficient computing reduces the need for hydrocarbon fuels and carbon emissions, which is a good thing even if you don't believe in AGW because if you use it up, then it's gone.
The US can win this one by innovating, rather than milking the market they inherited. It's not that hard, and we have the experience advantage until we forfeit it.
Help stamp out iliturcy.
Without a doubt it's no the best outcome and I would hope that eventually something changes for the better but freedom does give people the choice to do what they want even if Google didn't make the best decision.
For that reason I would support their move. I think giving people the impression that open source only has freedom for those who think a certain way will stop businesses from taking on open source code because they'll be labeled as bad.
It is wise to discuss it with Google and try to get the best outcome but if it doesn't happen then as long as they're not violating that particular licence, it's all fine imo.
I'm hiding the easy answer here deep in the thread under your question.
Google has Android slates (tablet format ARM-powered machines). If Google took 1000 of them ($400K worth) and just gave them to select Kernel developers (several each) to do with what they would, the problem would solve itself. The Kernel developers would adapt their preferred distro to work on the first slate they had, and they'd keep the rest to work as they were intended until it limited them - and then they'd want to break the limit by improving the Android code, which they would then want back in the Kernel tree because they would want their improvements to persist. It's all about motivation, and you motivate a Kernel geek with cool tech he wants to continue to use.
HP did this. Why do you think Kernel.org runs on HP servers and has for many years? It's because HP donates servers strategically, and yields huge benefits therefrom.
Help stamp out iliturcy.
And to a lot of people outside the Linux kernel development community, that "stable API nonsense" document is such a blindingly wrong example of software design that advocating it would get you shown the door in a job interview.
The "nonsense" docuemnt assumes that the purpose of all drivers is to converge to a single less-buggy ideal; this is true for a hardware driver where the hardware is frozen when it goes out the door. This is blatantly untrue for software drivers (e.g. 3d graphics, virtualization) where the interface between the driver and other (non-kernel) software changes far more rapidly than the interface between the driver and the kernel (and especially for 3d graphics, where the driver contains an entire thread stack, JIT compiler, etc.).
Good programmers will tell you that the driver belongs next to the component to which it is more tightly bound by a rapidly-changing API. Linux programmers will tell you the driver belongs in the Linux kernel so LKML can keep it up-to-date with changes to the mostly-unchanging kernel/driver API (and you can only make changes to the more-rapidly-changing driver/userlevel API during the merge window every three months). And to an extent Linux programmers are right - in an ideal world, the driver should be refactored until the driver/userlevel API is not so rapidly changing, at which point the driver should indeed live in the kernel. It's very easy for a Linux programmer to demand such ideal code as the cost of merging it into the kernel; it's much harder to take a "good enough" system and achieve such a high level of perfection, especially when merely achieving "good enough" is a target that moves fast enough to consume almost all of a programmer's time.
The two camps will never see eye-to-eye; both are completely convinced that they are correct.
A witty [sig] proves nothing. --Voltaire
Yo mama's company so fat it's too big to fail?
Somehow I managed to forget Nokia for being more open than Apple - and arguably - Google. I guess because so few people use, or will likely ever use, their smartphones. :)
You also forgot Palm, who have WebOS, which is more "Linux like" than Android (No forking here!) Palm's WebOS is often thought of by Android fans as being so Apple-like that they immitated the closed source nature of Apple. Nothing is further from the truth.
By the way, Nokia's market is limited to Europe (and the third world, but they don't buy smartphones in India, Africa, and the Far East), but there, they are the Blackberry of Europe with regards to smartphones. In other words, there are arguably more N and E series Nokia owners in Europe than any other smartphone. (Those *are* smartphones, albeit some of the older Symbian ones are more on the Palm Treo type of Smartphone technology than iPhone, very capable devices but very old-fashioned interface. Though it should be noted that they had a mobile WebKit browser before Apple did on the iPhone for Symbian's "Nokia browser"!)
I think describing anything that Greg Kroah-Hartman says as 'reasoning' is stretching the definition of the term somewhat...
I am TheRaven on Soylent News
I'm not sure what you mean by "rapidly changing driver/userlevel API". The kernel userspace backwards compatibility may not be broken, so it's quite stable compared to in-kernel interfaces.
If you mean that the interfaces *need* to change, then perhaps those parts of the drivers should not be in the kernel (as is the case for 3d drivers)
I'm not at all qualified to talk about virtualisation, but regarding 3d drivers, my understanding is that a significant portion of the stack actually lives in MESA, not the kernel.
The kernel side's responsibility (assuming a "modern" one instead of the traditional "X does everything" kind of driver) is to handle details the kernel cares about such as initialisation, mode-setting, power and memory management, while providing a mostly static interface to the userspace component of the driver, which contains most of the messy 3d stuff.
I think malleable in-kernel APIs allow your driver *not* to be perfect when you merge it. Perfection is required only for any user-space interfaces.
Furthermore, if your driver suffers from bad design in some kernel subsystem, it is possible go and fix things. Or conversely, if some driver depends on an interface that you wrote, but you need to change it, that is also feasible.
Now, one might argue that it's possible to preserve the old kernel APIs, but while that is possible, it greatly increases the maintenance burden. You will need to test code that only outdated drivers use. That's a waste of resources.
Sez who? My phone runs a pretty much standard kernel. Nokia have worked with the Kernel developers getting it all in to mainline.
This is great for me - if (when) Nokia abandon it some time in the future I (or "the community") can carry on supporting it forever.
Which is the whole point about the GPL (see RMS vs the printer driver).
Watch this Heartland Institute video
Sez common sense.
Android beats the crap out of whatever Nokia is capable of producing. And it's 100% open source, so you (or "the community") can just as well "carry on supporting it forever".
More details. What features of mainline Linux make it incompatible with portable phones?
How, exactly?
Watch this Heartland Institute video
If you mean that the interfaces *need* to change, then perhaps those parts of the drivers should not be in the kernel (as is the case for 3d drivers)
That's the ideal. Now step out of the ivory tower and look around. Drivers that are simple enough to conform to that ideal are already in the tree (or will always be out of the tree because of reasons that have nothing to do with "stable API nonsense"); Greg K-H has a neat program to work with such drivers so long as enough documentation is present. Drivers that are too complex to easily evolve to that design are universally OUT of the tree - the complexity coming from the driver being cross-platform or having code maturity behind it that rivals the Linux kernel itself. (And, bluntly, Linux has never had to deal with a codebase comparably mature to Linux - Linux has never learned how). No one disagrees that the ivory-tower ideal is better; the disagreement is in whether the effort to achieve it is worthwhile (or even feasible).
To make an analogy: the design improvements necessary to make a graphics / virtualization / smartphone codebase totally acceptable to LKML are (very roughly) equivalent in scope to the design improvements that would be necessary to make Linux *natively* run Windows drivers (ignoring the wisdom of doing so). This isn't obvious to LKML because most of the redesign would be in non-kernel code, but the scope of work really is that large. Google is not rejecting a better design simply out of spite; Google is rejecting the better design because the amount of effort to implement it is something that LKML would balk at doing to their own codebase.
A witty [sig] proves nothing. --Voltaire
This isn't obvious to LKML because most of the redesign would be in non-kernel code, but the scope of work really is that large. Google is not rejecting a better design simply out of spite; Google is rejecting the better design because the amount of effort to implement it is something that LKML would balk at doing to their own codebase.
Citation needed. Kernel people are not oblivious to userland needs. I can't say how much work it would be for Google to fix Android, but at least according to one LWN comment, Android could be made mainline-palatable without breaking userland interfaces: http://lwn.net/Articles/372631/
Some drivers would need to be modified as well, of course, but I think it's better to do the work early than late when you realise you really should've listened to the mainline developers.
Here's an interesting report from one of the realtime tree developers: http://lwn.net/Articles/372877/
it's not like Apple ever "borrowed" from the open source community without giving back is it?
The BSD crowd shouldn't care about that, Apple can do almost what they want with the code. But for Linux not getting the drivers for 3D accelerators, SOC models or other kind of hardware, just because Google have to reinvent the wheel, is really sad.
From TFS:
Android, while Google popularized it (after purchasing Android, Inc.) and provides its own apps that are used on most Android phones, is the Open Handset Alliance's Linux-based OS now, not Google's.
If you go off and read in more depth, one of the features of the Android waitlock is how waitlocks can be acquired or released by writing a waitlock name to a magic device (/proc/wait/lock and /proc/wait/unlock or something); this means userlevel can release a lock the kernel acquired, which is a feature Android uses but kernel developers reject as bad design. (And I happen to agree isn't ideal design, but taking a shortcut like that which only root can exploit does make a lot of sense.) Still think no userlevel code needs to be rewritten?
Greg K-H may be technically right in that Android could use a different mechanism without changing the kernel-user ABI. But that's only because his definition of stable ABI/API only covers syscalls and conveniently ignores special device nodes (/proc, /sys, /dev) where power management interfaces live. The Linux kernel ABI/API is really 10x larger than he realizes; those special device node interfaces are quite wide.
Linux kernel developers really are smart people, and their statements tend to be true - from a certain point of view. That point of view often has little to do with bringing in useful functionality and more to do with protecting that developer's personal interest - in Greg K-H's case, promoting the kernel as the one true repository of all code.
A witty [sig] proves nothing. --Voltaire