Slashdot Mirror
RTLinux Boasts Single-Digit uSec Responsiveness
An anonymous reader writes "A Linux implementation delivering single-digit microsecond responsiveness on 64-bit dual-core AMD Opteron processors is being demonstrated at the Embedded Systems Conference in Boston this week. From the article: "According to FSMLabs, an AMD Opteron 265 dual-core system running RTLinux can deliver guaranteed interrupt latencies of no more than five microseconds, with scheduling jitter of no more than eight microseconds, even with Linux under a heavy load." Heck, with numbers like that it seems like Linux could run circles around XP Pro for audio/video apps such as streaming, recording, and playback!"
beos did not handle media well because of low latencies, it handled media well because of thoughtful media system design. beos actually has poor latencies and response times in a number of areas, it just accelled at scheduling and prioritisation
http://www.fsmlabs.com/rtlinux-is-unfair-by-design .html
Don't hope you'll get this on your desktop anytime soon. This is RTLinux. Know what RT meants? Real-Time. That's a system in which you can guarantee the responsiveness.
But there's a catch: at development, you control all aspects (hardware and software) of that system. If just one component fails real-time requirements, you card castle crumbles.
In a desktop system, you can't control all aspects. That video card you just bought just added a little latency to your system, and it's not realtime. What is that program?
Never heard of it, but it fails RT requirement.
So, this is cool... but in the embedded systems field. Don't start comparing it to Windows XP and thinking you'll get it on a desktop Linux.
Misleading titles? Inflammatory blurbs? Keep in mind that Slashdot is a tabloid.
NT does claim to be suitable for some soft-realtime applications. Most people knowingly waive it off as useless for anything more than trivial applications. NT makes no guarentees about service time, but it has been making strides in improving priority levels, etc. I'm not sure what XP adds to this, since I'm basically pulling this out of a textbook from an influential researcher and author.
RTLinux is generally in the same class. You mentioned that they're using a desktop beast, but those are actually mainframe/server systems they're benchmarking on! 256 CPUs, Opteron Dual Core chips, etc. These chips feature lots of stuff you can't afford in the bulk of embedded systems.
The only thing this appears to be useful for is low latency financial market trades. If they were going after the embedded system market, perhaps an embedded system benchmark would suit better.
Also, how can your RTLinux feature a RTCore based on ruggedized Linux or BSD? How much of this system is Linux, exactly? Perhaps a trademark litigation is in order over this?
I Browse at +4 Flamebait
Open Source Sysadmin
Apparently the moderators do not either..
This isn't for the desktop (did you guys even RTFA?).
Ignorance is bliss, isn't it?
Second, there's a much better reason that RTLinux is irrelevant, even to the pro audio stuff that wants a real-time kernel: real-time processes in RTLinux can't access the Linux kernel, they can only access the RTLinux core. This means they would need special drivers for audio, MIDI, and so on, and any access to the file system or any other normal OS or userland service would have to go through a pipe to a non-realtime server -- super annoying.
This is why Linux audio developers are still pushing for at least soft-realtime performance in the stock kernel. RTLinux is good at what it does, but it's really only practical for systems that are deployed on limited hardware, such as telecoms systems and other embedded systems, where the appropriate drivers can be built into the application.
RTLinux, last I checked, was a real time OS completely unlike Linux that was able to run a Linux system as a subtask, much the same as Xen would. Linux tasks would have the same scheduling latency they would normally have, but programs running on RTLinux would have higher scheduling precision. However, you give up a lot in order to achieve low latency - the system doesn't have many of the features most programs would expect to be present, so don't expect, say, Open Office or Firefox to benefit from running on such a system.
On the other hand, maybe my understanding is wrong or out of date (if so, then someone please correct me).
I've been using the GPL version of RTLinux for years now, in high-speed data logger and control systems in the geophysics and biomedical fields. The problem (difficulty) is not commodity hardware (we've used off-the-shelve P1:P4, AMD, and Via boxes), it's writing the RTLinux drivers, or kernel modules, to communicate with and control the devices, such as a/d cards, serial devices, etc. That is, you can't take an application written for a non-RTLinux system, plop it on an RTLinux box, and expect hard realtime response -- you have to rewrite it.
Exactly - almost all of the large movie houses use Linux for their rendering pipelines (see http://linuxmovies.movieeditor.com/software/index. html for more) and there is a LONG list of software available for pretty much every task you could ever want regarding creating / editing / processing video on Linux.
;}
Many of these tools are easily commercial quality and/or created by commercial companies for use on linux - and many of them are 'linux-only' which blows the 'not for linux' argument out of the water.
ignorance is bliss
Gekido's Lair
Ugh... of course, 5usec is 1/200,000th of a second, so it's actually 10.000 clock cycles... *slaps himself*
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
To me the numbers announced are on par with hard real-time constraints
One would hope so, from a hard real-time OS.
In any case, hard real-time does not mean "really low latency." It means "failure to meet latency guarantee is fatal." You can't necessarily make a soft real-time system hard real-time just by speeding it up, particularly if there are unbounded waits involved.
A better and more thorough explanation.
Audio doesn't need microseconds, but 1-5 milliseconds is important for some applications. Unfortunately, Linux isn't yet there (dunno whether others are). Even with the latest RT preempt patches, I'm having problems with 1 ms latency. Also, prior to 2.6.12, you couldn't even get real-time priority (required even for 20ms latency) when running as a user (non-root).
Opus: the Swiss army knife of audio codec
It says 5 microseconds, while the clock period is around 500 ps. That's 10,000 cycles, not 10 million.
Opus: the Swiss army knife of audio codec
Getting 5us interrupt latencies is not a difficult feat for an RTOS.
Indeed. One year ago, I participated in the installation of a High Energy Physics (HEP) experiment. We were using PowerPC 604 CPUs running VxWorks (the commercial "industry standard" RTOS).
On that configuration, I was getting interrupt latencies always under 12 us (3 us average), and that was on a 300 Mhz CPU ! 5 us is hardly impressive on the kind of systems they are using.
However, what would be interesting would be to know how the system behaves under load. RTLinux has been known in the past to crumble when heavily loaded by low priority tasks. It might have gone better recently.
Note that RTLinux is not the only Real-time Linux implementation. There's also RTAI released under LGPL.
Probably mostly true, but not entirely. Consider the case when you're using a computer in a recording studio to record a bunch of musicians. In the old days, they used multitrack tape machines which could be configured so that each track could be individually set to record or to play back. Might then record a drum kit on 5 out of 16 tracks, and then come back and have someone else (the bass player, then instrumentalists, then singers) come and lay down tracks on top of that.
These days, tape is effectively dead, and people do this all with computers. And not, generally, with specialized kickass embedded computers, but with Windows machines or Macs.
So, what is the relevance of all this to latency? Well, when you are doing multitrack recording on a computer, you are doing something fairly unique: you are recording a track, but at the same time you are playing back the tracks that have already been recorded and the signal you are getting from the instrument or vocalist that you're recording right now. So, the (let's say) guitar player you're recording hears the drum and bass tracks that have been recorded as well as his own guitar, and all of this is coming through the computer.
Now, one little quirk about musicians (and I'm sure you'll understand once you think about it) is that they really hate hearing themselves in their headphones if what they're hearing is coming in 100 ms later than when they're making the sound. This tends to throw them off, because rhythm is an important part of music.
As a result, guys with workstations that are used for recording are to an extent after the lowest latency they can get their hands on. The only way to achieve this is with a minimum amount of buffering of the audio data and an OS that can handle servicing interrupts for the device driver quickly AND scheduling the audio software to run in time. Well, either that or a minimum amount of buffering and ridiculously over-specced computer so that the CPU is just so fast that it doesn't matter as much if the operating system sucks.
So, a system like this could be really great for a recording studio. That is, if Linux had pro-quality recording software and supported device drivers from pro-grade A/D and D/A converter manufacturers. (I'm not talking about your average Soundblaster Live here -- I'm talking about things that can around 16 channels of 24-bit, 192 kHz, both in and out, simultaneously. Hardware like that is fairly uncommon, and professional recording guys are not going to use a third-party driver -- they want something with support from the manufacturer of the hardware, because if the system dies in the middle of a session, they may have lost the chance to capture a performance that can never be reproduced.
My point (other than that Linux has a long way to go on the application software side of things for pro audio recording) is that latency can be a hugely important thing for audio -- if it's audio that involves live musicians or an audience in any way.
Doesn't RTLabs, claim to hold a "patent" on running the Linux Kernel as a 'task' under a 'real time' kernel?i cles/moglen.html
http://www.fsmlabs.com/openpatentlicense.html
http://www.aero.polimi.it/~rtai/documentation/art
that "real time = real fast."
Often, nothing could be further from the truth.
What's missed here is a Real Time OS is being compared to Windows XP. Windows anything is not a real time OS. It keeps getting pushed into real time use, simply because it is there, but it is not the best tool for the job. Having an OS respond in real time to interupts is like having a mouse that doesn't freeze for several seconds at a time on a busy system. (if you run Windows, you know what I'm talking about.) Using Windows in a real time environment such as video encoding, or such almost always produces the ocasional jump or glitch because Windows is not a real time OS. Interupts get ignored during such things as a disk access.
The truth shall set you free!
The AMD Opteron 265 is a dual-core processor. There was just one in the the test machine, it's not a huge mainframe/server system.
Chernobyl 'not a wildlife haven' - BBC News
With Windows it depends on your audio card and API. If you have a consumer card and use DirectSound or MME it's 30ms at least. With pro cards and ASIO or WDM KS it's often configurable. The one I have can be set anywhere between 64 and 2048 samples for it's buffer. That translates to about 1.5ms to 46ms for 44,100Hz and 0.6ms to 21ms for 96,000Hz.
As a practical matter when I set Sonar to it's lowest latency mode, with the drivers at a 64 sample buffer at 96,000Hz it reports the effective latency as 1ms in WDM KS mode. In ASIO mode, it claims 0.7ms lattency at 96,000Hz.
I don't have any way to test Sonar's claims, but I don't have any reason to doubt them either. Also not sure how well it'd handle heavy use, you might get droupouts, though it plays back a track just fine with that setting.
I believe on both Windows and Mac with ASIO compatible software and hardware, 1-2ms latencies are very attainable. I personally don't mess with it since I don't do live recording, but the software and hardware seems to claim it can handle it.
Exactly, "real time" != "real fast", real time just means "predictable".
The important thing is not to respond fast, but to ALWAYS respond within a KNOWN delay.
Windows CE 3 and later are true real time systems.
Give Ingo Molnar's RT preemption patches or the I-Pipe approach a look some day. "1-2 years" ago can be a long, long time when it comes to Linux... :)
quidquid latine dictum sit altum videtur.
Okay, I'm going to clear this up near the top of the article.
RTLinux is a *Real Time Operating System*, a tiny kernel that runs the Linux kernel. It has a rather sexy interfact that allows you to write the non-RT code that interacts with it in the regular, Linux way. You also don't need to hassle with Wind River salesmen to use it. This makes it good.
There seems to be a significant misperception here as to what an RTOS is, and the extremely misleading article summary makes it worse.
An RTOS is an extremely specific tool designed to allow someone to write code with very harsh restrictions on it with very low latency. This is almost always used for control applications (telling servos when to fire accurately). You can't "run Quake in RTLinux" and just get more accurate times -- code running as real-time in RTLinux can do very little besides memory manipulation, basic computation, and some extremely limited I/O. RTOSes are powerful tools for solving a very limited set of problems which very few people on Slashdot have.
RTLinux lets you write very simple, limited code that runs in a real-time mode, and run it on the same machine as regular Linux applications. And communicate with them. That's it. Doesn't do anything to improve the regular ol' Linux applications.
RTOSes give very low latency to the code they run -- something happens, code to handle it gets fired off very quickly. Microsecond latency (*not* millisecond) is completely overkill for the kind of general-purpose video or whatever work that people here are thinking of, unless you're trying to build some sort of specialized embedded system that does something to a real-time feed -- and your hardware's going to be very specially designed for this.
There's a good reason that we don't use RTOSes in day-to-day work. They have bad throughput, and you can't *do* very much with them in real-time. They're good if you specifically have a latency constraint from the time one sensor triggers to the time I/O goes out to another device. They aren't going to avoid audio dropouts on your GNOME desktop. Real-time is a *bad thing* from most people's standpoint -- oh, and they're really easy to accidentally hang.
If you want something to get excited about for general purpose use, look at the preempt patches for 2.6. 2.6 Linux has better latency than Windows XP (2.4 had worse). This is not RTLinux, this is regular-ol-Linux-which-can-run-Quake. My understanding is that ALSA and JACK represent improvements in the general-purpose latency area.
Unless you are designing application logic for robotic control systems, or are interfacing with PLCs, RTLinux really doesn't benefit you (okay, I'm sure there's someone out there that has a different application, but Joe Hacker with his Gentoo box doesn't benefit directly from this).
Every time realtime systems come up on Slashdot, misperceptions of 'em seem to get worse.
Any program relying on (nontrivial) preemptive multithreading will be buggy.
If you *do* want to do high priority stuff on your box, both (regular) Linux and (regular Windows, without getting RTX or any Windows RTLinux equivalents) have "realtime" scheduling options (see the "Realtime" option in Windows Task Manager, or the sched_setscheduler(2) man page under Linux). These aren't actually hard realtime, but give you all the fun of being able to lock up your box (as per realtime systems do) without losing the ability to still run general-purpose applications. You won't get single-digit microsecond latency, but you won't be worrying about audio dropout either, and you get to do fun things like use general-purpose libraries, use sockets, use virtual memory and all the other stuff that you can't do in real-time code under RTLinux.
Any program relying on (nontrivial) preemptive multithreading will be buggy.
beware: chrt'ing a badly implemented application may provoke a kernel hang
--
WebDSign: thrust the Web by trust
Yeah and it needs to be mentioned that the Amiga and older (Falcon) Ataris were some of the first truly multi-tasking machines available for the desktop. By 'truly' multi-tasking I mean they had separate chips for each part of the system. They had a video chip, an audio chip, a drive controller chip, etc. and the OS just had to lay around and send commands to each one. This was way different than the IBM PCs at that time which were horribly limited in comparison. The way they worked was beneficial to musicians, particularly MIDI musicians, because the latency tended to be very low. This makes perfect sense for arcade hardware.
It's not "funny," it's true.
± 29 dB
Howdy. I'm the author of the LinuxDevices.com article on RTLinux, and I have also played around a bit with demudi, the debian music distribution. Demudi (and the red hat remudi anodyne) use a 2.4 kernel patched to reduce application latency. You run all the sound aps as the root user in demudi, and jackd keeps track of real-time performance. Here's a summary of my experience.
Local songwriter wrote an album of songs, bought a Walmart PC for $300 powered by a Celery 1800 processor. He bootlegged a copy of cubase from a pal. Couldn't figure out how to use it. He then downloaded a cheap copy of cool edit pro. Recorded a dozen songs -- they sounded awful. He comes to me for help.
We install Demudi, stick in an M-Audio Delta 440 card, a $200 sound card with hardware mixer/monitor and 1/4-inch TRS 4-in/4-out breakout box. We record six songs in three days, each with 10 tracks or so, even though we had only a tiny amount of experience using ardour. We master everything at 24/96, and downsample to cd quality for the final. It sounds great!
The trickiest part is actually the downsampling, because jackd shuts the whole thing down when the CPU fails to deliver on real-time guarantees. This becomes a royal pain. We resort to shutting down all unneeded linux services in order to get the final mixes downsampled. It's a pain, but it finally works.
And the results are good. A professional sound engineer with 20 years experience listens to the results, and comments, "How many tracks were you doing at a time? Sixteen on this song? It's remarkable, because the sound doesn't get brittle. Usually, with that many tracks, things go brittle at a certain point."
Despite the fact that demudi has a real-time kernel, there *is* perceivable latency in the monitors when using software monitoring, though. Luckily, Alsa does support RME cards, and M-Audio cards -- pretty much. With M-audio cards, you have to use the console alsamixer to get to all the features of the card. But it can be done.
I don't think RTLinux would be used for real-time audio, though, since you need the Linux apps to be real-time, not just the hardware interrupts. But I don't know.
I do think the work Ingo and MontaVista are doing with native real-time Linux will have a huge impact on the A/V and effects industries.
To correct the last post:
1) Windows CE 3, 4, 5, were all RTOSes.
2) There are realtime extensions available for Windows XP and Windows XPe that effectively make Windows XP a realtime system, likely in the same way RTLinux could be used to make desktop "Quake running" Linux into a realtime system.
3) RTOSes are not magical. When you want to have hard realtime response to hardware, you end up sacrificing performance of something else. Likely this will be non-essential to your realtime performance, like UI response, disk transfer rate, something. CPU cycles don't come for free, they have to come from something else.
This isn't that impressive for RTLinux, which is really a scheme for loading applications as loadable kernel modules running without memory protection. RTLinux is an obsolete approach; the more recent Linux variants from Lynuxworks and Montevista have a much cleaner approach, based on the low-latency fixes in the 2.6 kernel.