Linux Kernel Goes Real-Time

Several readers wrote to alert us to the inclusion of real-time features in the mainline Linux kernel starting with version 2.6.18. (Linus Torvalds had announced 2.6.18 on September 19.) Basic real-time support is now mainline. This will ease the job of developers of embedded Linux applications, who for years have been maintaining real-time patch sets outside of the mainline kernel. The announcement was made by TimeSys Corp., a provider of developer services. Much of the work was done by Thomas Gleixner at TimeSys and Ingo Molnar at Red Hat.

Comments

What about media?

[stonedyak]

The article only talks about these new patches from the embedded devices point of view. So can anyone tell me if these new features would be useful for improving the responsiveness of media applications in Linux? I'm talking about video/audio playback, as well as authoring and recording.

What other benefits would the desktop see from this?

Re:What about media?

[SmileR.se]

No. Real-time is about being able to guarantee that tasks complete within a certain deadline (and ofc being able to predict this deadline), not doing things fast.

Re:What about media?

[fossa]

But desktop users, or at least me, don't care about doing things "fast". I care about things feeling fast. I care about latency. So, do these patches help the audio on my computer not skip when I move a window? I've tried the premptive kernel patches in the past with no noticeable difference. How are these patches similar, different, or complimentary to Ingo Molnar's (whose patches are mentioned in the article). Thanks.

Re:What about media?

[Bacon Bits]

Desktops and most servers do not get any benefit from a RTOS. RT makes it so that the system purposefully downgrades less-useful things like user input for maximum priority things like, say, polling a fetal heart monitor every n milliseconds or responding to an automobile collision to deploy an airbag.

RTOS in Linux is primarily useful for Linux-based routers. However, seeing that QNX has been in the industry for 25 years, has an extremely good reputation (it's the de facto standard in the auto industry), and is already POSIX compliant, Linux still has a long way to go. The price for QNX might be USD $10,000+, but if you actually have a need for a RTOS, licensing cost is not a major obstacle.

Re:What about media?

[novus ordo]

You are right, to a certain extent, but take audio/video work as an example. If I am using my computer as a synthesizer, I want to be able to hear the keys as they are being pressed, not 0.5-2 sec later. Even if the mixing part would take longer than the time I press my next key, the kernel should process that key anyway. Music is a weird thing.. a couple of milliseconds difference is enough to change the perception of a note. So in this case, it will 'seem' faster, even though it will have less throughput(context switches the mixing process out even though it has the mixing data in the cache which conveniently gets invalidated, swapped, etc.) I guess it gives certain applications the super-duper dependent on time factor that would otherwise be overlooked in the ways kernel measures "fast."

Re:What about media?

[flithm]

I get what you're saying, but if I didn't know better what you said could be a little misleading. To be totally correct a real time operating system has nothing to do with task completions, but rather with providing the ability (often mathematically proven) to respond to interrupt and thread context switches within a pre-determined (and known) time constant. This is generally referred to as the interrupt or thread switch latency.

And actually the grand parent is right to wonder about whether or not this will allow for a more responsive feeling system, because in general, it will. Much more so than the pre-emptive kernel routines already in = 2.6.17.*. If you've ever used a real time OS you'll know what I mean -- nothing feels so nice to use. The processor can be pegged, and have a ridiculous load average of say... 100 or something (100 tasks are trying to use 100% of the CPU), and you won't really notice any sluggishness... but, of course, tasks will just take a long time to complete.

It's not without it's downfalls though. Obviously being able to guarantee low latency interrupt responses is going to require some overhead. You'll definitely slow your computer down by using a real time OS, although unless you're a gamer you might not even notice.

All that said and done, unless you're using your computer for some very specific things like embedded devices, critical applications (medical, power station management, etc), and audio / video stuff, you'll probably never notice the difference.

I know a lot of audio people are happy about the real time patches because the delay between turning a dial, or moving the mouse, and the noises that come out of the speakers is quite noticeable even with really small delays.

Re:Open source at it's finest

[AchiIIe]

>>Only took 15 years.

Do you even know what Real-Time computing means? Do you know what an interrupt is? Can you name even a single one RT OS ?

sigh... from wikipedia: http://en.wikipedia.org/wiki/Real-time Computing

History

The term real time derives from its use in early simulation. While current usage implies that a computation that is 'fast enough' is real time, originally it referred to a simulation that proceeded at a rate that matched that of the real process it was simulating. Analog computers, especially, were often capable of simulating much faster than real time, a situation that could be just as dangerous as a slow simulation if it were not also recognized and accounted for.

The Saturn rocket had a 50 Hz hard-coded loop.

Once when the MOS Technology 6502 (used in the Commodore 64 and Apple II), and later when the Motorola 68000 (used in the Macintosh, Atari ST, and Commodore Amiga) were popular, anybody could use their home computer as a real-time system. The possibility to deactivate other interrupts allowed for hard-coded loops with defined timing, the low interrupt latency allowed the implemention of a real-time operating system, giving the user interface and the disk drives lower priority than the real time thread. Modern Desktop PCs will usually fail at such tasks, if no specialised real-time operating system is installed. [citation needed]. The Motorola 68000 and subsequent family members (68010, 68020 etc) also became popular with manufacturers of industrial control systems thanks to this facility. This application area is one in which real-time control offers genuine advantages in terms of process performance and safety.

Well, the meaning has changed somewhat

[jd]

Real-time, these days, provides a certain guarantee of service. The "harder" the real-time, the more firmly defined and the more firmly guaranteed any paramater can be. For example, you may want to guarantee that process X has a timeslice of T1 to T2 milliseconds that must start within a range of S1 to S2 milliseconds after the start of every wall-clock second, no matter what. This means if a hardware interrupt occurs that would prevent the guarantee being met, too bad. The interrupt will have to wait.

Typically, it is indeed used in simulations, but very rarely on a 1:1 simulated-time-to-wall-clock-time basis. Simulating the formation of a solar system, or the results of runaway nuclear fission cannot be done on a 1:1 basis. Instead, real-time's guarantees permit you to firmly quantify a constant ratio between wall-clock time and simulated time - provided the programmer can firmly guarantee that a given simulated time interval will never require more than a certain amount of wall-clock time to simulate and a barrier can always be established to prevent more than the desired amount from being simulated. This rules out all event-driven simulations, herustics, variable N-body problems, fixed-precision variable-mesh CFDs, etc.

As soon as the number of variables or steps becomes indeterminate for a given simulated unit of time, you cannot pre-determine the wall-clock time needed to simulate it. If, however, the guarantees are enforced, it is STILL real-time in the accepted sense, even though you have no idea prior to running the program as to when a simulated unit of time will occur. An OS does not stop being real-time if it is fed a non-computable problem, it merely stops being able to tell you when it'll get done.

ADEOS - the microkernel used in RTAI - is "hard real-time", as is VxWorks. TimeSys' Linux patches are soft real-time. Soft real-time is good for audio/video, but is pretty naff when it comes to providing any guarantees. It's usually faster than hard real-time, as it can spend more time processing than synchronizing, so is more useful on a desktop system. With suitable patches (nanosecond timers, real-time clock chip drivers, etc), Linux can be a damn good hard real-time system, but there are performance penalties. Besides, broadcast-quality images are 30 fps in the US and 25 fps in the UK. Being able to align a frame on a nanosecond boundary is thus a complete waste of time. It's damn useful if you're running a particle accelerator ring or even a maglev train (350 mph requires absolute sub-microsecond guarantees on timings), but not many home users have these yet. A pity - it might be fun to have a synchotron source.

Does Linux need hard real-time, then? Oh, certainly. It's popular in the very groups of people who are interested in nuclear science, mass transit or quantum cosmology. I firmly believe that patches that provide such mechanisms are not merely useful but extremely desirable. Bleeding-edge needs drive technology precisely because that is where technology isn't, so keeping in reach of those folk is vital.

Does Linux need real-time in the kernel? Certainly, though the soft real-time that has been added is probably sufficient. (I hate saying things like that. I'm a geek and I'm proud of it, and therefore want Linux to have the maximum flexibility possible. However, poorly maintained code is worse than no code at all, and there just isn't the userbase to keep hard real-time in the kernel at an acceptable standard.) As I said, soft real-time is all you need for multimedia. It's also all you need for games and other stuff that Linux hasn't been nearly hot enough in in the past.

Hard real-time distributions are another matter. For that matter, a distribution that provided a good set of hard real-time tools, clustering tools and CAM drivers would probably be very popular nor only with the scientific community but also with industry. A distro that pushed Linux into markets that couldn't be reached without effort from any mainstream project would undoubtedly be a good thing. It would be very much a minority effort, and yet would require far more sophisticated QA and technical support, so no sane person would attempt such a thing. Of course, I've never been accused of being sane...

Re:It's progress, but not everything you need yet

[Ingo Molnar]

Linux has made major progress in the real-time area. But it still doesn't have everything needed.

Many drivers are still doing too much work at interrupt level. There are drivers that have been made safe for real time at the millisecond level, but that's not universal. Load a driver with long interrupt lockouts and your system isn't "real time" any more. This is the biggest problem in practice. There are too many drivers still around with long interrupt lockouts.

That's where my -rt patchset (discussed by Thomas in the article), and in particular the CONFIG_PREEMPT_RT kernel feature helps: it makes all such "interrupt lockout" driver code fully preemptible. Fully, totally, completely, 100% preemptible by a higher-priority task. No compromises.

For example: the IDE driver becomes preemptible in its totality. The -rt kernel can (and does) preempt an interrupt handler that is right in the middle of issuing a complex series of IO commands to the IDE chipset, and which under the vanilla kernel would result in an "interrupt lockout" for several hundreds of microseconds (or even for milliseconds).

Another example: the -rt kernel will preempt the keyboard driver right in the middle of sending a set of IO commands to the keyboard controller - at an arbitrary instruction boundary - instead of waiting for the critical section to finish. The kernel will also preempt any network driver (and the TCP/IP stack itself, including softirqs and system-calls), any SCSI or USB driver - no matter how long of an "interrupt lockout" section the vanilla kernel uses.

Is this hard technologically? Yes, it was very hard to pull this off on a general purpose OS like Linux (the -rt kernel still boots a large distro like Fedora without the user noticing anything) - it's the most complex kernel feature i ever worked on. I think the diffstat of patch-2.6.18-rt5 speaks for itself:

613 files changed, 22401 insertions(+), 7903 deletions(-)

How did we achieve it?

The short answer: it's done via dozens of new kernel features which are integrated into the ~1.4MB -rt patchset :-)

A longer technical answer can be found in Paul McKenney's excellent article on LWN.net.

An even longer answer can be found on Kernel.org's RT Wiki, which is a Wiki created around the -rt patchset.

Performance overhead of the -rt patch-set

[Ingo Molnar]

RT has a pretty big speed penalty.

I can definitely say that unlike some other approaches, the -rt Linux kernel does not introduce a "big speed penalty".

Under normal desktop loads the overhead is very low. You can try it out yourself, grab a Knoppix-based PREEMPT_RT-kernel live-CD from here: http://debian.tu-bs.de/project/tb10alj/osadl-knopp ix.iso.

From early on, one of our major goals with the -rt patchset (which includes the CONFIG_PREEMPT_RT kernel feature that makes the Linux kernel "completely preemptible") was to make the cost to non-RT tasks as small as possible.

One year ago, a competing real-time kernel project (RTAI/ipipe - which adds a real-time microkernel to 'above' Linux) has done a number of performance tests to compare PREEMPT_RT (which has a different, "integrated" real-time design that makes the Linux kernel itself hard-real-time capable) to the RTAI kernel and to the vanilla kernel - to figure out the kind of overhead various real-time kernel design approaches introduce.

(Please keep in mind that these tests were done by a "competing" project, with the goal to uncover the worst-case overhead of real-time kernels like PREEMPT_RT. So it included highly kernel-intensive workloads that run lmbench while the box is also flood-pinged, has heavy block-IO interrupt traffic, etc. It did not include "easy" workloads like mostly userspace processing, which would have shown no runtime overhead at all. Other than the choice of the "battle terrain" the tests were conducted in a completely fair manner, and the tests were conducted with review and feedback from me and other -rt developers.)

The results were:

LMbench running times:

| Kernel............ | plain | IRQ.. | ping-f| IRQ+p | IRQ+hd|

| Vanilla-2.6.12-rc6 | 175 s | 176 s | 185 s | 187 s | 246 s |
| with RT-V0.7.48-25 | 182 s | 184 s | 201 s | 202 s | 278 s |

(Smaller is better. The full test results can be found in this lkml posting.)

I.e. the overhead of PREEMPT_RT, for highly kernel-intensive lmbench workloads, was 4.0%. [this has been a year ago, we further reduced this overhead since then.] In fact, for some lmbench sub-benchmarks such as mmap() and fork(), PREEMPT_RT was faster.

(Note that the comparison of PREEMPT_RT vs. I-pipe/RTAI is apples to oranges in terms of design approach and feature-set: PREEMPT_RT is Linux extended with hard-realtime capabilities (i.e. normal Linux tasks get real-time capabilities and guarantees, so it's an "integrated" approach), while ipipe is a 'layered' design with a completely separate real-time-OS domain "ontop" of Linux - which special, isolated domain has to be programmed via special non-Linux APIs. The "integrated" real-time design approach that we took with -rt is alot more complex and it is alot harder to achieve.)

See more about the technology behind the -rt patchset in Paul McKenney's article on LWN.net, and on Kernel.org's RT Wiki.