Linux Kernel To Have Stable Userspace Drive

liquidat writes "Linus Torvalds has included patches into the mainline tree which implement a stable userspace driver API into the Linux kernel. The stable driver API was already announced a year ago by Greg Kroah-Hartman. The last patch to Linus' tree included the new API elements. The idea is to make life easier for driver developers: 'This interface allows the ability to write the majority of a driver in userspace with only a very small shell of a driver in the kernel itself. It uses a char device and sysfs to interact with a userspace process to process interrupts and control memory accesses.'"

Comments

Better drivers and more of them

[Billly Gates]

A stable kernel api for drivers is what linux needs as proprietary driver writers make poor quality and buggy implementations. While this is not a kernel one its a good compromise as proprietary drivers are here to stay as much as it would be great if we had free gnu ones inside the kernel.

I wonder if it would be easy to port it to windows and macosx? IT would be cool for hardware makers to have a driver that works with all operating systems with minimal effort in porting. Costs are one of the issues besides the difficulty in porting windows drivers to linux which many makers do not bother doing.

Re:Better drivers and more of them

[whomeyup]

There's a userspace drive framework on Windows, but the vast majority of Windows drivers reside entirely in kernelspace.

Re:Better drivers and more of them

[chromatic]

Costs are one of the issues besides the difficulty in porting windows drivers to linux which many makers do not bother doing.

If only there were some magical pool of experienced labor just waiting to write and maintain, in perpetuity, Linux drivers for any manufacturer of any hardware....

Re:Better drivers and more of them

[howlingmadhowie]

no one wants the driver. what the developers what is documentation for the connectors to the motherboard. surely this should be a legal requirement for the manufacturer anyway. i would regard it as the minimal acceptable documentation for the product. it is, after all, my computer the hardware will be attached to.

i would personally like to see all pieces of hardware sold with schematics for the hardware. with other products, like cars, this is trivial anyway--anybody can open up the car and see what bit goes where. with computer hardware, because of things like microcode, this is impossible.

and as for intellectual property. what strikes me is how this phrase is always used to protect the financial interests of a company against the greater good for society and the individual. if someone would instead be honest and say "companies are allowed to require society to install software (which does what exactly?) and use a particular operating system if the user wants to use their hardware, so ensuring (at the least) that the product will soon be unusable" then i'd have less against the people who champion this position

in the best case, this is built-in obsolescence. one thing i find repugnant is the attitude that it is morally okay to force society into this position.

Re:Better drivers and more of them

[adamofgreyskull]

They are hardware manufacturers. They sell hardware. What the fuck do they have to worry about if they provide documentation on the interface between their hardware and the hardware under my desk? Am I going to suddenly be able to infer the entire design of their hardware (or the source of the drivers they've written for windows), just because I have that knowledge?

Am I missing something here?

Re:Better drivers and more of them

[ZorbaTHut]

Keep in mind how much of a modern graphics card's abilities are now located in software. More than once, a driver update has come out that has *massively* boosted graphics card speed. I suspect that modern graphics cards are really just ultra-high-speed multicore floating-point coprocessors, and most of the scene logic happens on the CPU. I wouldn't be surprised in the least if the interface between CPU and graphics card was a tightly guarded secret - main bus bandwidth, and bandwidth in general, is one of the major bottlenecks on graphics systems right now.

To make it even worse, I'm told that many wifi cards are only legal because they're not open-source. Sound bizarre? When they're sold, they're sold with certain restrictions on frequency and power. These restrictions are located entirely within the drivers. If they distributed open-source drivers, those restrictions would either have to be moved into hardware (expensive) or disabled (causes horrific problems with the FCC.)

We're well past the point where hardware interfaces can be described in half a dozen pages. We're well past the point where "hardware devices" even exist entirely in hardware. Most interesting hardware devices have complex interfaces that depend on functioning backend software.

Re:Better drivers and more of them

[Jah-Wren Ryel]

Am I missing something here? Yeah. Patents.

They are afraid that by providing documentation on interfaces, it may tip-off a patent holder to start looking for infringement where they might not otherwise have done so.

After all, when the prevailing legal advice is to actively not look for pre-existing patents, it is inevitable that companies will independently create infringing hardware. It's like we get the worst of both worlds - patents might as well be trade-secrets since reading them is a legal mine-field if you are working in the same area, but we also get government enforced monopolies that stifle competition.

At least the lawyers get paid for their contributions,
and that's all that should really matter in the end. Right?

Re:Better drivers and more of them

[ray-auch]

Am I going to suddenly be able to infer the entire design of their hardware (or the source of the drivers they've written for windows), just because I have that knowledge?


No - but that isn't their argument.

Suppose you get an email from Ferrari saying their new API for their F1 car has control functions for a moveable floor.

Will that enable you to infer the entire design of their car ? - No.

No suppose Maclaren get that email. Will it affect the competition between them and Ferrari ? - quite possibly.

In both cases, I don't know enough to decide if the argument is completely valid - but both are credible.

Re:Better drivers and more of them

[Haeleth]

To make it even worse, I'm told that many wifi cards are only legal because they're not open-source. Sound bizarre? When they're sold, they're sold with certain restrictions on frequency and power. These restrictions are located entirely within the drivers. If they distributed open-source drivers, those restrictions would either have to be moved into hardware (expensive) or disabled (causes horrific problems with the FCC.)

This makes no sense whatsoever. Why can't they just release the driver source code with a note adding that it is illegal to use the driver if you remove the restrictions?

You might as well say that it is illegal to make open-source FTP programs, because only closed-source FTP programs could include effective restrictions to stop them being used to download child pornography.

Re:Better drivers and more of them

[ZorbaTHut]

But does that mean software that runs on a CPU on the graphics card, or software that runs on the system CPU, stealing cycles from it? The latter is what some manufacturers are doing, and should not be doing.

Why not?

Seriously, why not? Do you honestly think they should be building an entire separate general-purpose CPU, and putting that on the graphics card? If they can achieve better FPS - overall FPS, including what's "stolen" from the CPU - by putting heftier more special-purpose hardware on the video card, and falling back to the CPU for the stuff the video card can't do well, why shouldn't they?

I fork over money for graphics cards because they make my games look better and play better. Fundamentally, I don't much care how they do that.

Yes, obviously, moving more stuff to the graphics card would be faster. It would also be more expensive. If there are better ways for them to use the transistors, it's fine by me.

That interface should be nothing more than the information of what the system and applications expect the graphics card to display, an encapsulation protocol to organize it into messages and responses, and a basic way to stream it across the bus (like PCI-Express x16, for an example with high performance). Those messages may possibly be a reflection of the graphical API calls done by the applications.

What we have these days in designs are the result of companies trying to cut their costs with the consumers be damned. These are bad designs, not so much because they steal CPU power from the consumer's computer, but more so because they create these massively complex interfaces that keep changing all the time, and driver code that is so buggy it is frequently the source of systemwide crashes or data corruption. At least if that buggy code is moved into a process, it can do its thing without taking down the whole system.

But we shouldn't have to be doing that. The hardware specific code should be inside the hardware, running on the CPU that comes as part of that hardware. Upgrades can be provided by the system CPU as a checksummed and, if necessary, cryptographically signed, blob (via a unified firmware image upload interface design that all devices should share that includes device match checks to be sure the correct image is loaded). Then maybe we'll start getting some real value add out of things like video cards, instead of getting cards that result in a net loss of CPU power when added in.

Why? You're making claims, but you're not backing them up.

Here's what I see. I see that I can go to the store and pick up a new graphics card. I can plug it into my computer and suddenly my games run significantly faster. Now, if I have a choice between Card A and Card B, and they're identical in every effective respect except that Card B makes my games run 10% faster with better shadows and uses some nasty techniques to do so that I completely don't have to care about, I'm buying Card B. There might be philosophical arguments to be made about whether Card A or Card B is better, but in all honesty, I see these philosophical arguments as similar to the whole Hurd vs. Linux debate. Okay, Hurd might be "better" - but Linux works.

What they're doing with the cards right now works. They've poured millions of man-hours into making them as fast as humanly possible within budget. If they determine that it's best to use CPU power for some things, make the GPU into a half-FPGA, and use a complex binary protocol that changes based on how the GPU is programmed, and they can make all of that stable - and my computer literally hasn't bluescreened in a year, so I'll give them that much - I think, just maybe, you should respect that they're making the right decisions for the priorities that people have. Which, generally, is game performance per dollar.

If you want to keep metaphorically extolling the virtues of Hurd because it's "better", though, I'm certainly not going to stop you.

Re:Better drivers and more of them

[level_headed_midwest]

They could, and with the signed drivers being required for at least 64-bit Vista, they could enforce it. But I'd be VERY willing to bet that if Microsoft even hinted at this, the hardware maker would just have to threaten to call the DOJ and Microsoft would backpedal faster than you could say "antitrust."

And you also mentioned the other solution if Microsoft does make threatening noises. With both the Windows and Linux kernel driver APIs being stable, it should be trivial to make a translation wrapper between the two. MS would have their hands tied in keeping drivers off Linux if that happens as they'd have to stop their own driver development. MS needs as many good drivers as they can get for even 32-bit Vista, let alone anything 64-bit. If I were Microsoft, I'd be helping out all I could to get a stable wrapper or translation layer so that a "universal" driver for both Linux and Vista could be made by device manufacturers. Vista notoriously lacks drivers, especially Vista 64-bit, and Linux has enough to make most things work, especially for corporate machines and in server rooms. Both of those areas are ones that are much more likely to consider switching to Linux from W2K/XP than to Vista than the ordinary Joe. Or they will sit on XP like many sat on W2K until around the time XP SP2 came out. Both results would give MS no sales, and since they are also MS's most profitable markets, not upgrading to Vista would be a serious blow to MS. So I think that taking a risk that some previous 2K/XP switch to Linux because of more driver support is far outweighed by the increased number of Vista sales because of better driver support.

Full circle?

[MichaelSmith]

Perhaps in ten years time Linux will be a microkernel

Re:Full circle?

[Tumbleweed]

Microkernels are the wave of the future.

And always will be. :)

I thought I read Uberspace

[TheRistoman]

So I immediately thought of a spaceship running Linux... I mean, could it really be otherwise?

Re:Performance

[corychristison]

If you'd have read the article, you'd notice that it states firmly:

However, DMA transfer between userspace and kernelspace is not yet implemented. This means essentially that drivers which involve high traffic are not an option yet. So graphic drivers as well as file system drivers and similar cannot use this API at the moment.

Finally...

[Statecraftsman]

The lack of a stable userspace driver api was that last thing stopping soccer moms and grandmothers from running Linux on the desktop.

My sarcasm is so extreme, I think what I said above may have actually been true.

Re:Performance

[jd]

There's a 21us hit every time you context-switch, which would hurt on very high-performance drives but is probably below the threshold of being obvious for network-based storage and really slow drives. However, a few intelligent drives supposedly support total kernel bypass and zero-copy - basically the drive remote DMAs the data into and out of memory, once told where things are. This would only require kernel access for initializing the transfer and locking down the pages. I seriously doubt, though, that any of these will be common uses for the userspace drivers.

The most common use, I would imagine, would be as a testbed platform. Writing things directly into the kernel has many unquantifiable variables - I'm highly respectful of all who develop kernel code on a regular basis, that is no small achievement. Developing the same code in userspace with an API to link over eliminates many of the possible ways you can screw up a machine, although the code would still need to be written with an eye to being used in kernel space. For much of the writing and testing, though, you'd be in a more predictable environment.

The second-most common use would be for proprietary closed-source drivers to be written for userspace. Writing them for the kernel is problematic as the kernel internals change too much, and many such companies spend so little on maintenance that the drivers rapidly become obsolete - requiring users to either use inferior kernels or different technology, with the latter often not being possible or practical. I don't imagine older Linux drivers to be ported this way, any more than they've been maintained by the pathetic commercial vendors who pull such stunts, but newer such drivers should now be less pathetic and marginally more portable, which will be good.

Oh, wrt comments by others, Linux should absolutely never become a microkernel. Message-passing as a methodology is barely adequate for networks - RPC and CORBA are hardly famed for their elegance or performance, and when was the last time you saw Globus or MPI being used to link machines in a LAN gaming session? For that matter, STREAMS has been available for Linux since about Linux 1.2, if I recall correctly. I can't think of a single driver - even outside any of the standard or experimental trees - that uses it. I like the idea of such a patch, as I like the idea of maximum flexibility, but if it were truly useful, it would be used. It isn't.

Like QNX, although not as clean

[Animats]

QNX has had user space drivers along somewhat similar lines for many years. In QNX, all the drivers are in user space, which makes for a much smaller kernel. Here's a simplified article on QNX driver writing.

The Linux approach has the problem that Linux doesn't have the message passing primitives that QNX does. So there's a special purpose mechanism to hook up these new user-space drivers to the I/O system calls. In QNX, "open", "close", "read", and "write" are actually C library functions that call MsgSend to do the work. (System V IPC isn't really suitable; it's asynchronous, which means a few extra scheduler events for every message pass when you try to use it for something that works like a subroutine call. Long story.)

Unfortunately, on x86 hardware, you can't protect the system from a user level driver and still give the driver direct hardware access. IBM VM mainframes get this right, but x86 does not. On the other hand, you can have channel drivers for the various types of x86 channels (SCSI, FireWire, USB, etc.) and make other drivers work through them.

User-level drivers cost you at least one extra memory copy of the data. That's not too bad in practice. I did a FireWire video camera driver for QNX, and when transmitting 640x480 24 bit images at 15fps, it took about 3% of a Pentium 4 CPU.

Vista already has this (not trolling, read on)

[AlphaWolf_HK]

FWIW, not trying to troll, but thought I would point out that this feature is one of Vista's improvements over XP, and simultaneously the primary reason why Vista's compatibility isn't that great right now, and thus the primary reason why many people don't switch to Vista yet. Most of the hardware vendors have to make big changes to their drivers in order to accommodate this, especially nvidia who has to make about 4 different user space drivers (one for d3d, one for opengl, and an SLI version of both of those.) This is a good thing to have for both security and stability reasons, and I was waiting for when somebody would add this to the Linux kernel.

Linux has the advantage in that with Linux you can use both the old "kernel only" drivers, and the user space drivers at the same time. Vista could have done this as well, however Microsoft felt that if they allowed this to happen, then most hardware vendors would be lazy and continue to use their old kernel mode drivers, thus defeating the purpose. And to be honest, I agree with them. Linux doesn't need this on the other hand, as eventually somebody who is interested will make these kinds of changes to all of the open source drivers anyways as needed, which can't really happen because most windows drivers are binary only, so Linux can more or less take the "phased change" approach.

Disclaimer: I use both Vista and Linux (gentoo is my preferred distribution,) and am not afraid to say that I don't hate either of them, and rather like both of them.

Re:Vista already has this (not trolling, read on)

[Anonymous Coward]

You're half right.

Vista has partially user-space drivers for graphics, where the majority of the driver is in user space, and the kernel-mode component just allows communication between the driver and the hardware. Linux already has a similar architecture, as does MacOS X.

Second, it has user-space USB drivers. Which Linux and MacOS X have both had for ages.

It also has user-space printer drivers, which is no big deal - printer drivers hae been user-space only for years on most operating systems.

No other driver is user-space. They're all still in the kernel. They have modified the API and ABI for a lot of them, particularly sound (by removing all hardware acceleration) and network (because it interacts with the newer network stack).

Re:Vista already has this (not trolling, read on)

[wwahammy]

Actually it was there before Vista. Windows Media Player 11 came with the first version of the userspace driver framework. I think its used for media players that sync with WMP.

My understanding was that Microsoft recommended companies move to userspace not that it was required. To be fair though, I know very little about WDDM so they might have different requirements.

When I read the headline, the first thing I couldn't help but think was if the roles were reversed there would be hundreds of people saying "Good to hear you got something Linux had for a year already." Good ideas are good ideas. Why can't people just be happy when their ideas are recognized as good by others?

Useless API, for simple drivers only

[kscguru]

Maybe I'm unique in that I not only RTFA, but browsed the patches themselves.

Which led me to the conclusion that this patch set is worthless. It allows remapping of memory-mapped I/Os to a userspace app, and allows a thread to "wait" on an interrupt. Both are nice ideas, and it would be very easy to implement a nice serial port driver with the new APIs. (As any kernel hacker knows, serial port is one of the simplest device drivers; it's easy.)

The new API is completely useless for binary-only drivers. I/Os / IRQs are enough for extremely simple devices - these are, after all, the primitives for talking to hardware. But if this were all a driver needed, don't you think Nvidia / ATI would have used this model for shim drivers a long time ago? Simple things like DMA and PCI configuration access are not present - but to be fair, those are implementable with these primitives. Reality check: real world drivers are a lot more complicated. What is impossible is fast thread switching, kernel synchronization primitives, access to the network stack (wireless!), ring-0 CPU instructions, real-time timing access, and the huge reduction in context switches / cache flushes that comes from running within the kernel (moving code to user-mode increases latency by a factor of 3, roughly). Kiss the lag-free desktop goodbye as hard drive latency skyrockets, watch your 3D framerate drop by 70%, see your webcam stutter into unusability.

The kinds of drivers this API can support are the simplistic ones, the kind that are already GPLed and are already in the kernel, the 80% of devices in this world Linux has always had good support for. The kinds of drivers this API cannot support are 3D graphics, high-performance disk or networking, wireless networking, latency-sensitive USB or Firewire, the virtual devices (VMware, KVM, Xen, even /dev/tty) - notice that most of the devices Linux supports poorly (and all the common binary-only drivers) fall into this list.

To be fair, the official (e.g. from Linus) announcements I've seen only claim this interface is useful for embedded devices (which tend to code for a specific kernel, and not get updated). No official announcement claims the new API will help binary-only drivers. It's just the OSS-zealot crowd making unwarranted assumptions. Yes, this is the bad news: the stable userspace driver API will do nothing to solve binary-only driver dilemmas. Sorry.

Re:Useless API, for simple drivers only

[suv4x4]

What is impossible is fast thread switching, kernel synchronization primitives, access to the network stack (wireless!), ring-0 CPU instructions, real-time timing access, and the huge reduction in context switches / cache flushes that comes from running within the kernel (moving code to user-mode increases latency by a factor of 3, roughly). Kiss the lag-free desktop goodbye as hard drive latency skyrockets, watch your 3D framerate drop by 70%, see your webcam stutter into unusability.

Nice rant there. Let me summarize it:

"What is impossible in user space driver is kernel space features".

No shit. That's the point of a user-space driver. If you give a user-space app access to ring-0, it's no longer user-space. Or did you imagine there's some sort of unwet water that the stupid developers of the kernel keep missing.

The user-space driver is not set to replace all kernel mode drivers. Just like Vista, it's set to replace *some* of them, for example USB devices with low traffic. It's not a solution from heaven, it's just a reduction of fail-prone pieces that lurk in your system.

If you RTFA you probably had to read the summary as well where it's said user-space drivers aren't suitable for high-performance gear such as graphics cards.