Kernel 2.6 Real-Time Benchmarks

An anonymous reader writes "This whitepaper at LinuxDevices.com includes benchmarks comparing the real-time performance of the vers. 2.4 and 2.6 linux kernels, based on LynuxWorks' BlueCat Linux 5.0 beta. The graphs compare the results for average and worst-case measurements of both interrupt response and task switch performance for the two kernels, running on a 1GHz Pentium III under relatively heavy load. Check it out -- there's an enormous improvement. The article also includes a rundown of other features of the new kernel that seem likely to be welcomed by embedded developers."

Comments

Wider Application?

[4of12]

I'm hoping the benefits of these performance improvements aren't limited to single-purpose embedded devices.

Better interactive response (framebuffer, keyboard, mouse) would help desktop users, while some of the enterprise warehouse folks could use less interruption of important I/O tasks.

Maybe those aims can be achieved somewhat by these kernel improvements.

Re:Wider Application?

[selderrr]

i second that. I write software for the psychology dept at our univ, designed to do reaction time experiments. In basic priming experiments, we need accuracy up to at least 3 msec, preferably 1msec. For eye-movement detection I'd like even finer measurements, but under windows that's simply impossible.

accuracy up to 1msec is feasable by using a sufficiently fast machine (1GHz or more) with enough ram (512 at minimum) and VRAM (64min, more if you want speedy pics), removing network adapters & USB equipment and beefing the process priority up to max.

But even under those conditions, I occasionally lose a msec here or there. I wonder how OSX behaves under the same conditions...

Re:Wider Application?

[Krunch]

It seems you need "a system means that performs its functions and responds to external, asynchronous events within a specified amount of time". In other words, a real-time OS. Neither Windows nor Linux are real-time OS. There is some patches that aims at making Linux more real-time.

You can use Tron too.

Re:Wider Application?

[selderrr]

I know. I really need an RT OS, but there are 3 reasons why I can't use the RT linux project :

- the univ has a windows-only network policy. It took me a bucket of sweat just to get authorisation for my OSX tiBook
- I need to develop an app that is used by 10 people. All of them use windows. Getting them to switch would take more effort than simply making them accept an (occasional) possible 1-5msec delay somewhere in their data.
- I have 10 years of experience with win32 development, zero years with linux. The app has a big honking interface with 50 dialogs or something, so I need a RAD IDE like MSVC++ DevStudio.If I were to learn a linux IDE (for a gnome or KDE app) I would probably take an extra year.


The only solution would be to create the authoring app on win32, and a runtime for RTlinux. That way, my collegues could create their experiments in windows, and copy them to the lab machine which runs linux. But then there's the issue of univ policy. Plus, I'm lazy :-)

Re:Wider Application?

[laa]

Try Delphi/Kylix from Borland. You can use the same code to compile both win32 and linux apps. Works quite nicely. There's a free (as in beer) version for personal use that you can test - as a university the licenses should also be reasonable.

Re:Wider Application?

[treat]

the univ has a windows-only network policy.

Did Microsoft pay them to do this? Was it an overt payment or an under the table bribe?

it doesn't say

[Tirel]

which version they were using exactly.. test2-mm2? test3-ac1? test1? this is supposed to be a serious article?

What about the interactivity patches?

[dpilot]

The article (Yes, I RTFA.) talks about the responsivness of the new kernel, showing dramatic improvements in active and maximum latencies. Yet at the same time, there are people working very hard on a set of 'interactivity patches' and complaining of skipping when playing music on 2.6.

I haven't followed those discussions back to the beginning, so I don't know if they're gilding the lilly. But the intense work on interactive scheduling *now* scares me about like the late work on the VM did with the 2.4.0 release.

Just like the VM was the Achilles heel of 2.4 will interactive response, particularly playing music on a system under load, be the bane of 2.6?

Re:What about the interactivity patches?

[oever]

Playing music on a 2.6 kernel is certainly not perfect, especially in the first 30 seconds of a song with xmms. During this time xmms is probably loading the song and the disk access holds back the music stream.

Performance is, however, much better than with 2.4.

Re:What about the interactivity patches?

[vigilology]

2.4 never skips for me. How can skipping in the first thirty seconds be better than never skipping?

Re:What about the interactivity patches?

[StarBar]

Playing music on a 2.6 kernel is certainly not perfect, especially in the first 30 seconds of a song with xmms. During this time xmms is probably loading the song and the disk access holds back the music stream.

That only shows that xmms is not taking advantage of the new possibilities on the 2.6 kernel. It is up to the application to set priorities between tasks/threads, it is not something the kernel will do by itself. Properly set priorities will get you a much better sound experience on the 2.6 kernel. The preemptiveness of the 2.6 kernel will get rid of the lagging cursor movements, among other things, and that has nothing todo with priorities, so you will get it automatically. This might get you a feeling of better performance but infact it is just a better system response to your interaction. // Starbar

Re:What about the interactivity patches?

[irc.goatse.cx troll]

" 2.4 never skips for me. How can skipping in the first thirty seconds be better than never skipping?"
They're refering to skipping under load (think compiling a kernel, transcoding video, decompressing large data, etc).

In 2.6, it skips for the first 30secs before the schedualer can see that its an interactive task.
In 2.4, it skips until the song is done because the kernel isnt much on caring about 'interactive tasks'.

Re:What about the interactivity patches?

[vigilology]

Oh. My 2.6 xmms would skip under no load every so often :-/

Re:What about the interactivity patches?

[Anonymous Coward]

So the audio skipping is a feature, not a bug, right?

Re:What about the interactivity patches?

[irc.goatse.cx troll]

Try 'renice -10 xmms' as root.
A lot of what the schedualing stuff does relies on nice levels.
man nice for a full description, but basicly a lower nice level is higher priority, max is -10.

The kernel also learns what tasks are interactive based on how often they sleep, but they should also respect nice level.

I generaly keep X at -5, mozilla at -5, and xmms at -10

Re:What about the interactivity patches?

[vigilology]

Doh, didn't think of that. I shall try it next time I try 2.6, thanks. Need lm_sensors and the spca webcam module to compile with it first :-)

Re:What about the interactivity patches?

[Anonymous Coward]

xmms is obviously optimized for the behaviour of more traditional schedulers. Someone is going to have to tweak the code to work better under 2.6. You have the same problem when you port code between different OS's.

Re:What about the interactivity patches?

[Anonymous Coward]

XMMS 1.2.8 has been released. You can find it on the download page. You can also check out the NEWS or the more detailed ChangeLog. Enjoy.

Re:What about the interactivity patches?

[Harik]

Um, 2.6.0-test4 blows for X. Any time mozilla is rendering a complex frame the mouse jerks like an epileptic chinchilla.

May I re-ask the question I asked on Monday?

[Anonymous Coward]

May I re-ask the question I asked on Monday? And maybe throw in one of my responses for good measure?

How does this compare with the kernel that was reviewed [and thoroughly trashed] by Dedicated Systems last August?

PDF DOCUMENT: http://www.qnx.com/products/ps_neutrino/benchmarks /elds/qnx62_elds.pdf

**********

NO - I'm not a QNX employee.

I am, however, about to embark on a huge project that involves real-time collection of massive amounts of biometric data, and I've spent the last several weeks investigating the state of the art of RTOS's.

We are looking at five possible candidates: VxWorks, Linux, CE.NET, QNX, and LabVIEW Realtime. VxWorks has a lot of market share, but the consensus seems to be that, under the hood, it's a little shaky. The consensus also seems to be that Linux just ain't ready for primetime; in fact, Linux realtime performance is so bad that people believe the kernel will need to be re-written from the ground up before it's ready to play in this league. Don't know much about LabVIEW Realtime, except that the National Instruments salesdroids are happy to sell it to you. QNX looks like it's got the nicest kernel of all, but it's not clear that it's got the driver & third party application support we might need.

Which leaves us with CE.NET: It's got surprisingly good performance [bests VxWorks in tests], it's got all the multimedia codecs of Windows, and it's got built in support for ActiveDirectory [so technicians could upload data directly into the database and sign their work right then and there].

On a purely performance-based evaluation, I'd probably choose QNX, but because of its flexibility, we'll probably go with CE.NET.

Anyway, back to my original point: How is this new 2.6.x "realtime" kernel any better than the 2.4.x "realtime" kernel that failed so miserably in the Dedicated Systems review?

Again, not trolling - just looking for all the information I can amass.

Thanks.

Re:May I re-ask the question I asked on Monday?

[Samrobb]

Taken from the article you reference:

It has to be said that Red Hat Inc. does not claim any real-time behaviour.

SO... they compared a RTOS with an (admitedly) non-realtime OS? I'm not surprised at the results.

RedHat markets ELDS as an OS for embedded systems. Not all embedded systems require realtime performance. Heck, even systems that require realtime performance don't always require the level of performance that QNX can deliver. There's a large number of embedded systems for which even plain Linux without any performance enhancements is a good choice.

If you're really looking for a version of Linux that supports hard real time requirements, take a look at something other than ELDS - <shameless plug> TimeSys Linux, for example </shameless plug>

(yes, I am a TImeSys employee).

Re:May I re-ask the question I asked on Monday?

[Anonymous Coward]

Neither the standard Linux 2.4 kernel nor 2.6 is a real-time OS.

There are projects out there that attempt to give Linux a real-time kernel, but those are not part of the main Linux development tree.

Re:May I re-ask the question I asked on Monday?

[killmister]

Which leaves us with CE.NET: It's got surprisingly good performance [bests VxWorks in tests], it's got all the multimedia codecs of Windows, and it's got built in support for ActiveDirectory Why one would need Active Directory and MS codecs for biometrics data processing real-time system ?

Re:May I re-ask the question I asked on Monday?

[t4k1s]

The Linux kernel was never intented to compete with QNX or any other realtime operating system.
Try RTAI. It's a patch to allow hard realtime performance on GNU/Linux systems.

Re:May I re-ask the question I asked on Monday?

[Anonymous Coward]

Which leaves us with CE.NET: It's got surprisingly good performance [bests VxWorks in tests], it's got all the multimedia codecs of Windows, and it's got built in support for ActiveDirectory Why one would need Active Directory and MS codecs for biometrics data processing real-time system?

Turns out these medical types are absolutely freakin' paranoid about their double blind tests.

Think Team A/Team B analysis at the CIA, but in a laboratory setting, it's more like Team A prepares the dosages [placebo versus varying concentrations of the compound in question], then Team B administers it and gathers data without knowing what it is they are administering.

If you have something like Active Directory [or Novell Directory Services], then the Team A Group will have full Read/Write access to the first half of a subject's record [subject's prior medical history, nature of the compound administered, etc.], but no access whatsoever to the second half of a subject's record [data recorded after compound was administered], whereas the Team B Group has no access to the first half of a subject's record, but full Read/Write access to the second half of the patient's record.

If your real time OS has support for something like Active Directory, or Novell Directory Services, or iPlanet, and, of course, if your backend database does, as well, then Team B group members can upload the data they collect directly into the database, without you [the programmer] having to write some kludgy, half-assed daemon or service or whatever to perform the authentication for you.

PS: This sort of thing works well in reverse for routers. If a router [or a switch] is aware of a directory, it can authenticate itself to the directory, then query the directory and download all of its parameters [like Access Control Lists] from the directory. In the past, Cisco has done this sort of thing for their routers vis-a-vis Novell Directory Services and Active Directory, but, like all Cisco products, that stuff is mucho expensive.

PPS: As for the codecs, our surgeons will want a realtime display of the data that is being collected, which means drivers for both video cards and video capture cards, and codecs for displaying that video on screen. Surprisingly enough, there do seem to be a fair number of companies who have released QNX drivers for video capture cards.

Re:May I re-ask the question I asked on Monday?

[Rock Ridge]

You could try LynuxWorks' Bluecat and if that is not good enough they have a true realtime offering, LynxOS, which is ABI compatible with Linux.

Beware TCQ on 2.6.0

[tzanger]

LKML verified it and I've experienced it personally -- TCQ on IBM Deathstar drives (mine were 60G, the LKML was 120G IIRC) can cause massive fs corruption.

Apparently a queue depth of 8 (the default it seems) is the specific culprit. LKML seems to say that TCQ of 32 works but I've turned it off entirely now.

It's marked experimental for a reason. :-)

Still performing poorly

[Clockwurk]

From the whitepaper...

The QNX NEUTRINO RTOS v6.2 and Embedded Linux Developer's Suite v1.1 were evaluated against the
same criteria and test suite.

The QNX NEUTRINO RTOS v6.2 performed very well during this evaluation. None of the performance or stress tests revealed any problems and the RTOS was fast, predictable and reliable at all times. The QNX NEUTRINO RTOS is also the only RTOS that has a true message-based client-server architecture well equipped to handle today's requirements concerning distributed processing, high availability, etc. The Red Hat Embedded Linux Developer's Suite v1.1 (based on the Linux kernel 2.6) is clearly not foreseen to be used in a real-time environment. Linux is made as a GPOS and the test results illustrate this. The added value of the Embedded Linux Developer's Suite is questionable: it does not make it easier to generate a custom target platform.

Although the Linux kernel is royalty free, it comes with a price: documentation is poor and the API is not compatible with (POSIX) standards. The learning curve to get the kernel up and running on your custom target platform is steep.

Good Job, SCO!

[Anonymous Coward]

That is some very fine code.

Re:Linux 2.6 != operating system

[CentrX]

It schedules tasks, handles input/output with hardware, allocates storage and memory, manages processes... it's an operating system. Now, it may be more precise to say "Red Hat Linux" or "GNU/Linux", but that doesn't mean that Linux isn't an operating system.

bullshit

[dh003i]

Linux is the kernel of an operating system (of many operating systems, actually). With just the Linux kernel, you basically can't do anything. With just the Linux kernel, you certainly can not operate your system. That's like saying that a brain is a human being because it's the central control region.

Re:bullshit

[CentrX]

You can construct any sort of device that doesn't require human interaction. It performs the functions of an operating system.

It's not quite like saying your brain is a human being, it's like saying the brain is the operating system for your body, which is essentially true in nearly all respects. The human body in your analogy would be the hardware, and in that respect brain is to operating system as human being is to computer.

I'm not saying that the operating system is everything that is software on the computer (which would be analogous to saying that a brain (kernel) is the same thing as a human being (operating system). A human being isn't analogous to an operating system at all.

This is a pointless analogy anyway, though many would argue that a brain is what makes a human uniquely human. We have a lot of physiological systems that are similar to other mammals and animals and function at the similar level or inferior levels, whereas our brain is clearly more advanced than that of other animals. Anyway, it all comes down to this. All of the cloud of nearly essential tools that comprise a Linux system are applications under the kernel, they are managed by the kernel, and to the kernel they are indistinguishable from other applications.

Talk to your computer science department

[Anonymous Coward]

Assumeing you have one, this is a job for your computer science department. They have the people who know how to do this, and you can get plenty of under grads who will work cheap for expirence and their name on a paper.

A true CS department will also not be so windows focused, and help you fight the IS fools^h^h^h^h^hpeople who insist they know the best way to do your job.

Who says (Free)BSD is dead

[Anonymous Coward]

They have version 5.0 already! Thats almost twice as much as Linux!

It would be nice if...

[acidrain69]

IT ACTUALLY COMPILED ON ALPHA.

Sorry. I WANTED to test it, but it doesn't work, and I don't know enough to fix it. Something about previous declarations of define's.