Slashdot Mirror
Should Linux Have a Binary Kernel Driver Layer?
zerojoker writes "The discussion is not new but was heated up by a blog entry from Greg Kroah-Hartman: Three OSDL Japan members, namely Fujitsu, NEC and Hitachi are pushing for a stable Kernel driver layer/API, so that driver developers wouldn't need to put their drivers into the main kernel tree. GKH has several points against such an idea." What do you think?
Linux, with the stability of Windows prior to the advent of certified drivers.
Lets put it in Kernal 2.7.
Oh wait... OOPS!
Having a kernel API for drivers allows developers to stay away from the mainstream kernel. This will enhance the stability of the kernel in general and also allow hardware vendors to support Linux with less effort.
To Terminate, or not to Terminate, that's the question - SCSIROB
No thanks, this is just a great way to promote closed source inside the linux kernel and to make debugging problems totally impossible.
Shadus
I have been waiting for that for a long time. The lack of a stable interface is hampering adoption of Linux. Not all of the manufacturers are willing to open source drivers or for that matter to continuously change them as the APIs change. This is very welcome but unfortunately, I think they'll fail. There is just too much politics surrounding Linux these days.
http://www.kroah.com/log/2005/11/07/#osdl_gkai2
:-)
Some misunderstandings were made. But of course, if they posted this link, there'd be no point to posting TFA or the arguments that will almost certainly follow.
They'll think I've lost control again and leave it all to evolution. -- Supreme Being, Time Bandits
I gave up Linux mostly because I was tired of getting punished for having new hardware, which is often unsupported. Especially on laptops.
If you don't force the manufacturers to include their driver source in the kernel, you might get them to release actual drivers for their new hardware.
Toronto-area transit rider? Rate your ride.
It's a bad idea because what happens when the driver ABI changes? You have to wait umpteen months for the company to get off thier asses and fix it - like nVidia.
It also precludes anyone else from fixing bugs in the broken, half assed crap most corporates spit out these days.
feh. stuff.
Not just Heresy, but Linus has said directly that he doesn't want a stable binary kernel driver API percisely so that people *can't* write binary drivers for Linux.
I am unamerican, and proud of it!
one of the main problems for getting device manufacturers to support linux is the fact that they either have to release a new version of their driver every time the linux kernel changes some esoteric internal API, or be badmouthed for not having good linux support.
would it really hurt so much to guarantee a stable DKI? doesn't have to freeze the whole kernel, just a subset of functions that will be guaranteed to work as they do now in perpetuity.
backwards compatibility is just as important to driver writers as it is to app writers.
doesn't even have to be binary backwards compatible, source level would be sufficient for most.
The difference between Theory and Practice is greater in Practice than in Theory.
One of Linux's great strengths is the flexibility of changing to meet new needs and not being hobbled by rigid backwards compatibility. This can only be done if all source is open and anyone can update drivers to meet new needs. When someone comes up with a patch to streamline a certain minor part of the kernel, it frequently has repercussions elsewhere in kernel land. It is these small changes which have made linux better and better with breathtaking speed. A "stable" binary API removes the possibility of keeping everything up to date and would dramatically show down the adoption of new features and general improvements.
Continual refactoring is worth far more than some supposed binary API which prevents changes. Get rid of binary drivers! If companies are so paranoid that they want binary drivers, then the hell with them. Linux can advance better without that baggage.
Infuriate left and right
These companies want a binary layer so they can build binary drivers.
What people tend to forget about this is that it's a bad idea- from most every perspective.
The Linux kernel was written as a Free Softwate alternative to the existing *nix systems.
We have thousands of drivers in the kernel from a combination of development efforts. Sometimes a driver is written by an independant kernel developer, and sometimes it's written from the company producing the hardware, working alongside the community.
What these companies want is to be able to have thier cake without giving back to the community. This is a very slippery slope at the least, and illegal at best, since these sorts of links to binary kernel drivers have been long known to be illegal to distribute alongside the kernel (unless special previsions are made, such as a userland driver).
Also, binary drivers have been known to be buggy and essentially removie the kernel developers from a position where they have control over the kernel as a whole project. I won't even go into the issues associated with a possible security hole in a binary driver, or a binary driver with, for example, spyware in it.
The arguement for it is, of course, that this might mean more drivers. This is a test of our strength as a community. Doing the right thing is harder. It means we won't have all the hardware at all times, and certainly not the newest thing. But we retain control over our computers.
It's hard to say no, but this looks like a clear case where we have to.
Please read The Linux Kernel Driver Interface (all of your questions answered and then some) by the same author before commenting...
IANAKH, but couldn't more drivers be moved into userspace (or other lower rings) --- especially for things like USB printers and miscellaneous gizmos? I think it would also be nice to not bundle thousands of drivers and support for architectures I don't have with the kernel. The kernel itself could provide a very minimal layer of hardware protection (like an exokernel?) and there'd be libraries exporting generic abstractions for particular classes of hardware. Is the context-switching penalty really so great these days? Educate me!
One of Linux's biggest problems is the lack of device drivers for common devices, especially newer video cards. Let's face it, companies like ATI and NVIDIA aren't going to release fully open-source drivers. It would be wonderful if they would, but it would also be wonderful if we had flying cars.
Having a stable binary driver interface would make it easier for hardware manufacturers to embrace Linux, give things like wireless chipsets more usability on Linux and drive further adoption of Linux as a viable competitor to more proprietary solutions
The perfect is the enemy of the good, and the more Linux gains a foothold the better it is for open source. Insisting that device manufacturers need to have on-staff kernel hackers in order to keep ahead of a frequently-changing kernel makes it that much harder for manufacturers to support Linux as a viable alternative.
Provided Linux can have a stable binary driver infrastructure that doesn't harm stability, it would greatly help in the adoption of Linux worldwide.
But we retain control over our computers.
Tell me, what BIOS do you run ? Do you have the source to the firmware in your IDE disk drive ? In your CD-ROM/DVD-ROM drive ? Do you have the source to your SCSI controller's firmware ?
If you think you have control over your computer you are suffering under a delusion.
If a company is developing an embedded Linux ap for their own hardware. All of a sudden, all of the communications with the board-specific hardware is being done through binary drivers, resulting in an effectively closed system.
No more hacking WRT54G's for you, chump.
"Eve of Destruction", it's not just for old hippies anymore...
Linux was, is and hopefully will always be "open". I don't want closed drivers in the kernel (even via an API layer) any more than I want a Sony rootkit masquerading as DRM.
It isn't about "politics". It's about policy and philosophy.
If the hardware doesn't work with Linux, don't buy the hardware/pester the vendor for an open driver, or don't run Linux.
It is not welcome. Linux is about Open Source, and allowing people to link-in binary closed drivers goes against this.
Bypassing the dogma of the above, there are numerous pragmatic reasons why this would be better for linux, even if you don't include support for binary third-party drivers.
Sure, some of these are extreme cases. You can usually get away with just re-compiling the driver, and occasionally, you can even use the binary from the existing version.
The point is you should *always* be able to do this wihtin the same major kernel version. There is no technical reason, aside form the politicis of not wanting to ever allow binary drivers, to not have a stable driver API.
Imagine if the Mozilla plugin API changed with every new version of Firefox. And look at all the complaints when a new Firefox version doesn't work with all the old extentions. It is the exact same.
If you are going to take the strategy of "We will do things to attempt to force you to do thigns our way," don't be supprised if the response of companies is "Fine, then we will ignore you."
The simple fact of the matter is that most companies are not willing to go open source, for software or drivers. You can argue that's a bad thing, but it is the reality of the situation. So, if open source is out in their book, either because of contractual obligations or mentality or whatever, they are left with two choices:
1) Do Linux drivers, and update them every time the interface changes, which can be as often as every minor kernel revision.
2) Ignore Linux, and let the community write the drivers if they want.
The problem is that Linux is a bit player. They are larger than the other bit players, but they are still tiny, less than 10%. Given that the continous rewrites can get expensive, the choice for many will simply be not to write the driver.
So if you are ok with that, then great, but don't get mad at companies when they won't play by your rules. Are they being unaccomidating? Sure, but so are you.
In the end, it comes down to needing to make a decision of what you want Linux to be. If you want Linux to try and become the next big thing in OSes and start to really make an entrance in the home market, standardisation is needed. Standard APIs, standard UIs, inter-version consistencies, etc. In essence, it needs to become more like OS-X. Now if you are ok with Linux being more of a geek/server OS then that's not necessary, but you can't demand the world change around you.
the question I have for this is "why?" wouldn't a stable binary API likely result in far more third party hardware support for linux? possibly more laptops that are actually compatible with linux?
This seems like a case of open source programmers shooting themselves in the foot because they want everything to be open source... not every application and driver is going to be made open source just to suit the desires of the linux development community. It seems to me that sticking to a hard party line against closed source software instead of trying to co-exist with said software is bound to keep linux in relative obscurity and pretty much ensure that it never becomes a viable competitor in the desktop market.
"In America, first you get the sugar, then you get the power, then you get the women..." -H. Simpson
The BIOS is indeed an issue, and there are efforts underway to make a Free BIOS.
But why not try our best to have as much control as we can?
Wrong. It is closed source companies who put the code before the user. They protect the code more than they protect the users. Open source is about protecting the user by allowing unhindered access to code for modification and redistribution.
It's funny how you warp things around.
They don't release the source. There is no source. That's the whole point. They just release the programming docs so anyone can write a driver. There is no IP for them to protect.
The day you can get third party IP licensors (e.g. that nice crossbar architecture used in previous-gen nVidia chipset memory control blocks wasn't developed in house by nVidia and they have contractual obligations not to release interface specifications for it) to agree to have their interfaces open by the licensees is the day you'll have fully open register-level documentation for consumer 3D graphics chips.
"The Devil does not know a lot because He's the Devil, He knows a lot because he's old." -- unknown
"Basically, I want people to know that when they use binary-only modules, it's THEIR problem. I want people to know that in their bones, and I want it shouted out from the rooftops. I want people to wake up in a cold sweat every once in a while if they use binary-only modules."
- Linus Torvalds on linux-kernel
And many people forgets that non-gpl drivers may be very well impossible to write at all (at least some lawyers think this), drivers are not at all like an app is WRT to gtk, drivers are more like "plugins". Plus, a closed driver module makes MUCH HARDER to debug bugs if the driver is doing bad things, and you can't know that (which makes harder to stabilize and/or develop the kernel. Several closed drivers can make it a hell or impossible at all.
Linux is actually much better at this than windows - you can see what the kernel does. Microsoft's test suite means nothing, as explained by a (great) microsoft programmer: http://blogs.msdn.com/oldnewthing/archive/2004/03/ 05/84469.aspx
"In a comment to one of my earlier entries, someone mentioned a driver that bluescreened under normal conditions, but once you enabled the Driver Verifier (to try to catch the driver doing whatever bad thing it was doing), the problem went away. Another commenter bemoaned that WHQL certification didn't seem to improve the quality of the drivers.
Video drivers will do anything to outdo their competition. Everybody knows that they cheat benchmarks, for example. I remember one driver that ran the DirectX "3D Tunnel" demonstration program extremely fast, demonstrating how totally awesome their video card is. Except that if you renamed TUNNEL.EXE to FUNNEL.EXE, it ran slow again.
There was another one that checked if you were printing a specific string used by a popular benchmark program. If so, then it only drew the string a quarter of the time and merely returned without doing anything the other three quarters of the time. Bingo! Their benchmark numbers just quadrupled.
Anyway, similar shenanigans are not unheard of when submitting a driver to WHQL for certification. Some unscrupulous drivers will detect that they are being run by WHQL and disable various features so they pass certification. Of course, they also run dog slow in the WHQL lab, but that's okay, because WHQL is interested in whether the driver contains any bugs, not whether the driver has the fastest triangle fill rate in the industry.
The most common cheat I've seen is drivers which check for a secret "Enable Dubious Optimizations" switch in the registry or some other place external to the driver itself. They take the driver and put it in an installer which does not turn the switch on and submit it to WHQL. When WHQL runs the driver through all its tests, the driver is running in "safe but slow" mode and passes certification with flying colors.
The vendor then takes that driver (now with the WHQL stamp of approval) and puts it inside an installer that enables the secret "Enable Dubious Optimizations" switch. Now the driver sees the switch enabled and performs all sorts of dubious optimizations, none of which were tested by WHQL.
(IOW: it doesn't guarantee stability or quality at all. It's just a false sense of "stability")
First, I think you're missing the fact that, overall, Linux doesn't care that you can't put your binary-only drivers on it.
Linus has said publically many times that the reason that there is not ABI is because he doesn't want one. Binary drivers for *anything* end up screwing stuff up. When they do, there is NOTHING anyone but the original author can do. The code stagnates, users get shut out without any help and nobody is any the wiser as to how that hardware actually worked in the first place.
That's WHY there is stuff like the kernel module license tainting, so that the kernel developers look at a problem, see that you have the massive unknown of a in-kernel binary loaded and can instantly filter your report out. They don't care that your binary driver doesn't work. They can't help you.
Additionally, setting anything into a static position means that development of it ends and stagnates. You'll never get a static interface that you can use to extend the drivers when new features come along. You end up with all sorts of kludges and interface versioning to try to take account of new things.
Linux is developed as an independent operating system, not Windows 2006. No-one wants to make you use it if you don't want to. I doubt Linux was ever intended as anything other than a "pure", almost theoretical, system; that is, one that can be constantly redesigned from the ground up to the way it should have been, not kludged to make it fit your eight-year-old driver (which the author is no longer available to update) for a mouse that happens to still use the old interface.
"Frankly, linux desperatly needs both a kernel debugger, and an ABI to be a REAL alternative for many customers."
Whoa, magic word customers. Linux doesn't have customers. Your company may have customers. There's no obligation on Linux to help you get/keep your customers. People use Linux because they want to. How often does Linus appear on your telly begging you to buy into Linux? Never. Because he doesn't care if you do or not. However, I do imagine it feels pretty nice to him that you do want to use it.
"It also needs the ABI for driver developers so that we can write a single driver and expect it to work on the dozens of flavors of linux we are expected to support."
*You* are expected to support whatever you decide to make. Unfortunately, the linux kernel developers are expected to support YOU, your hardware and everyone else in the world. They don't because they cannot and have no reason to. Even if they had your complete source code, they cannot be expected to maintain your driver for you (which is what will happen when your company goes bust / gets bored with OS).
Sometimes the best-written drivers in the world are not taken into the kernel because they don't quite fit and the maintainance involved in keeping them in the kernel is too difficult. Your driver, if it is to have any support in the kernel, needs to be able to be updated on any kernel-coder's whim in order to make the whole a better system. You can't do that with binaries, you can't even do it with stable interfaces. You have to have the source.
The kernel coders have never promised that your stuff will always work (unless it is designed to run purely from userspace... several times Linus has says that userspace interfaces will not MUCH change over time.). They haven't because they cannot.
The nature of the system is changes to bring improvements, from the interrupt system to the IDE interfaces, from the schedulers to the userspace interfaces such as sysfs or procfs, Linux changes and evolves over time and they cannot guarantee that anything other than userspace syscalls and the like will not be broken, changed or improved between one kernel release and the next.
When the linux kernel people discover a new way to write drivers that sees enhancements across the board, chances are that they are going to break any of your "single driver" models. That's why they won't give you one. Them im
In other words, it would turn Linux into the same kind of piece of shit that Windows is, and defeat the entire purpose of using it!
"[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz
Initial costs associated with a manufacturer-supported driver:
Ongoing costs associated with a manufacturer-supported driver:
These costs exist even if a version of the driver is merged into the mainline kernel. The only problem solved by such source-level merging is compatibility with the latest kernel version. It is not acceptable to the manufacturers' customers to be required to update to the latest kernel/distribution to be able to use the device.
Here's the key point: If there is no binary interface between the driver and the kernel, all of the above costs skyrocket. You have M kernel versions against N distributions, with the total increasing over the life of the product. If there is a binary interface guarantee from the kernel development team to change only very slowly and only extremely rarely breaking compatibility -- like the guarantee Windows provides -- then the incremental costs are containable. It is reasonable to expect that 95% of their testing on 2.6.5 is valid on 2.6.14.
The perfectly reasonable response from kernel developers is that with closed-source drivers they get stuck debugging problems that are't kernel-related (I don't hold ideology to be economically significant so I'll ignore it here, without insult to people's strong opinions on the subject). Their proposed solution is to require the driver's source before they'll help with the debugging.
From the manufacturers' point of view that's a very draconian requirement. They are justifiably concerned about intellectual property (availability of the source makes it much easier for competitors to reverse-engineer the hardware/firmware). Surely there must be a middle ground. Is there some way to have a relationship between the device manufacturers and the kernel developers that minimizes everyone's costs?
I think there is. Note that all of the above costs and issues are just as valid in the Windows world as in the Linux world. Microsoft doesn't want to deal with bad drivers crashing their systems, costing them both development/debugging time and reduced perceived stability (--> lower sales). Their solution is the Windows Hardware Quality Lab (WHQL).
The WHQL is a separate entity from Microsoft. Device manufacturers are required to submit their driver source (effectively under NDA) along with their device. The WHQL staff runs the driver through a battery of tests, probably mostly automated. If the device and driver meet stability standards set by Microsoft, the driver is signed by WHQL. Windows checks for this signature at installation time and warns the administrator if it is not present. Microsoft can reasonably refuse to support non-WHQL-signed drivers when crashes occur, for exactly the same reasons that Linux kernel developers refuse to support drivers without the source. This system has been the single most important factor in Windows' significan
Bandannarama
Oh, where do I start... I could talk about every one of your points.. lets just pick a few.. Firstly, the lack of hardware support for things like grandma's web cam, and caching SATA controllers hurts Linux much more than it hurts adaptec or joe blow's web cam co. If linux is to be just used by hackers for hackers that's probably fine. That isn't the impression I get from "linux people" who are constantly whining about lack of hardware support.
Additionally, setting anything into a static position means that development of it ends and stagnates.
Really? Could have fooled me... Windows can still run DOS and Windows applications from 20 years ago, linux can still run application from 10 years ago. When an API is created, usually there are hooks for extending it in the future. I wouldn't say either one has stagnated, over the course of the last 10 years. If that were the case windows wouldn't run on my quad 64-bit athlon.
*You* are expected to support whatever you decide to make. Unfortunately, the linux kernel developers are expected to support YOU, your hardware and everyone else in the world.
But its our brand which gets damaged, when plugged into a linux box, and it works at 1/4 speed, or looses data because the developer who implemented the driver forgot some important edge case.
They really don't care about you and your binary drivers
No, but the weekly calls by linux users asking for them says that Linus and friends are only a small part of the community.
Stick a Knoppix CD in a computer and see how much of the hardware is supported by default. 90% if not more? Include binary-only winmodem/USB ADSL/philips webcam etc. drivers and you get close to 99%.
Sure, linux supports a lot of hardware, but there is a lot it doesn't support, or supports in a seriously half ass way. I would venture a guess that about 40% of the drivers in the kernel don't actually work based on my experience running linux since the early '90s. When I say don't actually work, i'm saying the driver cannot recover from rare hardware error conditions (like FC cable pulls for example), doesn't fully work. Doesn't work in all situations, for example SCSI adapters that work fine with harddrives but don't work with tape drives because they cannot deal with the rare condition of a tape drive returning check conditions and data at the same time. Or, works fine at some fraction of its rated capability. This isn't counting the fact that the kernel developers often break functionality in the kernel drivers when they change stuff and it doesn't get discovered for 6 months, because the developer making the change didn't test the driver that was affected. Now, your average slashdot weeny doesn't see this because they have standard whitebox or dell machines, that are the same as the other 90% of slashdot weenies. In real life there is a vast amount of strange hardware out there, professional audio cards, hardware encryption engines, 4 port fiber channel cards, ficon, 10G ethernet cards, large disk arrays, big memory systems, 10's of different types of system management interfaces for monitoring things like, system ECC soft errors, redundant power supplies status, etc. The list goes on. In many cases the linux driver was written and tested a year ago, a bunch of people are using it in a production environment and it has been broken for 6 months in the mainstream kernels. I was responsible for fixing a number of race conditions in the VMM a couple of years ago, that only occurred with a SMP box that had more than a couple G of RAM (a rarity back then). This was suppose to work, but it didn't because the people with 6G of ram didn't test it with SMP applications and the people with >2 CPU's didn't test with more than a few hundred megs of RAM. Recently I tried bench marking a 3TB disk array in linux, only to discover that it didn't work in over half of the file systems I tried (XFS, Reiser, JFS, ext2, ext3) and in at least one case worked but was so slow it was unusabl
I know you're joking, but how about this for an idea:
... rant away open source crazies. ;)
A hybrid kernel. Open source drivers are compiled into the kernel. There is a API for closed-source drivers to run in user-space.
Does not violate GPL.
Little compromise to stability.
Developers who only want to do closed-source drivers can do so.
Developers have incentive to open source their drivers in order to have better performance and take advantage of newer kernel features (the internal APIs are updated with the kernel, the external APIs stay fixed and fall behind the feature curve).
Win.
Win.
Win.
Unless its just a philosophical question, in which case