Slashdot Mirror
MontaVista Rolls Out Fully Preemptable Linux
A couple people wrote with a link to an article about MontaVista. They've introduced "a fully preempetable Linux". It's based on kernel 2.4, but the timing is interesting, considering 2.4 isn't completed yet - and if this had come out earlier, perhaps some of this could have been included. The article's a bit press releasely, but has some good info.
I am currently a user of QNX OS and it's preemptive real-time capabilities are absolutely astonishing. For the sake of Linux I would hope that this company holds off on production of its distribution until the kernel is at least stable, if not fully released. If a big "oops" is made too early in the game, many people or just going to stay with their OS of choice, which isn't at the time most likely Linux, because having confidence in what OS you stick on your real-time hardware is no laughing matter at all.
This was discussed on the kernel mailing list a few months ago (before MontaVista did their implementation of it) and it seems likely that a preemptable kernel will be available (or the default) in Linux 2.5 for _uniprocessor_ systems .
The problem, though, is that the approach taken doesn't work on multi-CPU systems. The overhead of synchronization between CPUs is prohibitive when doing preemption in the manner that MontaVista describes.
This show the power of open source. These guys proposal was regeted so they decided to add in themselves. If they improvments are good enough it will be incorparted in the next version of linux. Got to love it.
Basing their code on 2.2 would mean it was outdated before it was of any use - afaik 2.4 has big extra support for SMP, which is essential for their pre-emptive solution.
It's not as though 2.4's contents are unknown, they have probably been working with 2.3 builds all the way through. they mention they have been in contact with linus torvalds, i'm sure that for something this big he would have provided them with as much information about 2.4 as soon as possible.
Integration into 2.5 makes sense, not 2.6, so this can be tested and refined within the full linux kernel development environment, and then subsequently released into 2.6 as part of the full stable kernel.
i really look forward to this, this is _good_ corporate involvement with useful goals.
fross
> Can someone explain what difficulties these people at MontaVista must have overcome? What is
> it about the linux we know that makes it not fully preemptable?
Current Linux only switches processes when the kernel explicitly decides to. (in other words, when a system call finishes all its work, or when a system call decides that work will take so long that it is time to give another process a chance to run)
This patch makes Linux able to switch between processes every time that an interrupt hits, even if it is in the middle of a system call.
The trick is to avoid deadlocks. Consider the following case: process A enters a system call, and the system call takes a lock. Then, an interrupt hits and process B is scheduled, preempting A. Now B does a different system call. This call disables interrupts and tries to take the same lock. It can't get it, though, because process A already has it. But since interrupts are disabled, process A will never get a chance to run again.
Your machine hangs hard.
The solution to this problem chosen by MontaVista is to simply disable all preemption whenever _any_ lock is taken by the kernel. The disadvantage to this approach is that:
- even when it is not possible for a set of locks to conflict, preemption will still be disabled.
- all locks become slightly slower (because every lock operation must now disable preemption too)
Well I can't claim to be an expert kernel hacker, but I have had to use Linux for what was essentially real time interactivity, relying on controlled conditions to provide me with CLOSE ENOUGH. The fact that Linux IS reliable and predictable and has been for some time, has allowed alot of folks to use it as if it was a fully multitasking system, and in many ways that sort of quality of service (having programs not freeze because of other programs) is what I think
drew me to Linux in the first place.
For a time I explored BeOS and the BeBox in particular, but the slowness of that development and the abandonment of the hardware left me cold. Still it had some interesting ability that other OS's don't have involving streaming media, and anything that brings such things (Rendering in a window continuing smoothly while window was being dragged as one example) is a good thing for the user community as well as for the embedded.
While people DO seem to throw alot of hardware at problems to make operations smooth, with my Athlon 500 and 256 ram I cannot run an IE 5 session without getting skips out of winamp. This sort of performance depresses me so much I have already begun to switch over to Linux on the machine for normal use, despite it being purchased for windows development (necessary evil).
I don't know how long it will take to get this sort of performance up, and I know that the danger of RTOS functions is that a badly programmed high priority thread can cause havoc, but if there were proper guards against such things, it would probably be enough to make Linux my OS of choice for interactive exhibits (which it is close to being as it stands).
As to why we are already talking about 2.4, what I really want to know is why 2.5 is meaningful, when it is an unstable track and thus unlikely to be seen until 2.6 in most desktops. What does this mean? If we are talking about something that is done now that is not going to be in a stable release for another year, but with 10 and 100 fold improvements, does that mean we are going to have to start supporting development kernel releases for clients because the feature set is too important to miss? It hasn't been the case for me since 1.3.. I really DON'T want to be doing kernel catchup in the modern era.
If this is important technology, then why can't we postpone 2.4 a little and move it in?
What kinds of schedules are we really talking about?
Is 2.4 expected this week? Is this RT stuff expected this year?
Unlike with Windows and Mac, I don't think that many people are sitting on the edge of their seat waiting for 2.4. I don't think it is keeping folks from choosing Linux. I am not sure I can understand the purpose of rushing forward if there is good technology that can become PART of the mainstream kernel without causing radical change in usage.
Or is this not so important after all?
Ingo did a preliminary patch for this a while ago, as part of his low-latency work. It appears that MontaVista has picked up the ball and run with it. Integrating a patch like this is going to require a lot of QA, so it should go in early in the 2.5 cycle -- this is definitely not something that one throws in at the last minute.
:-/ Anybody have a good reference to lockless data structure, algorithm, and design techniques?
It will be interesting to see how much the preemption infrastructure impacts performance (throughput) on a uniprocessor.
On a perhaps related note, I've been thinking about the growth of locks in the kernel. Many of these will have little contention, and locking is a pessimistic approach that incurs overhead. Plus, more locks means more locking discipline required.
Nothing has been "rolled out". This is not a product launch. This is the prototype release of preliminary code for a development kernel. The 2.4.x series isn't here yet, and neither is MontaVista's code.
Read the article, as there are several paragraphs in there talking about where this code will be going. They're hoping to get it incorporated into 2.5.x NOT 2.4.x.
Francis Hwang
Do domain names matter?
I like the sound of it.
:)
While I generally prefer FreeBSD, one of the reasons is exactly the desktop responsiveness spoken of. On the same machine, I find FreeBSD far more repsonsive than Linux when using X under load. On Linux, I can feel the lag in system response at a load of about 3--even though that load is from doubly niced makes and there's plenty of physical memory left. Under FreeBSD, I've gotten to loads of 10 without any noticable degradation in response.
NOw if anyone can find a way to quantify this . . .
It is fairly stable now, true. And it doesn't do any harm to try.
Oh, except when it randomizes your hard drive and suddenly all your data gets eaten because of the truncate() bugs. But apart from that, its okay to try.
Ah, theres also the random oopses. Thats the other thing I forgot, Apart from the truncate() bugs and random oopses, its okay to try.
Whoops, I forgot the memory leaks. you have to reboot quite a lot 'cos of the memory leaks. But... apart from the truncate() bugs, the random oopses, the memory leaks its okay to try.
Eek. theres the memory corruption. It scribbles on your memory. If you can put up with truncate() bugs, random oopses, memory leaks and random corruption then it's definitely stable.
Seriously folks, get linux2.4.0test8-pre6 and test it as much as you can and submit ____proper____ bug reports of any problems. But I would _not_ reccomend using it near "real" data yet - even if you do have 2.2 to srop back to like the previous poster does. remember: fsck is not magic.
(BTW, the truncate() bug is fixed now AFAIK and I made most of the other stuff up for a laugh.)
-- MartinG To mail me: echo kewyjlcxyzvjfxbqwh | tr bcefhjklqvwxyz
There are certain activities in the kernel that must be completed undisturbed. If you are running some type of real-time application that must take action in a short time period, you have a conflict. Fully preemptive simply means that you can *always* interrupt the kernel to do whatever needs to be done rather than *sometimes* or even *most of the time*.
MontaVista, recall, is an embedded linux company. Since embedded systems don't need SMP, that's not where their focus is. It'll be interesting to see if they try to add support and what actually gets into the official kernel.
Btw, I happen to work at MontaVista Software (not their main office right now, which is why I'm kinda' out of the loop and don't have more to say about this), and can attest that they're really cool folks! <grin>.
...even most people who think they do.
With CPU speeds being what they are today (and constantly increasing), the number of cases in which the regular schedular can't handle things correctly is really very low.
On the other hand, some of the improvements done for RT purposes (eg. finer-grained locking) help other people too (eg. SMP users).
> his quirks make the entire project too reliant upon him.
You could have ended that sentence four words earlier.
What makes you so sure the project would grind to a halt if Linus left? Very little development on this scale has been done in this way before. There is nothing in the past to compare this to that tells me Linus leaving would be a drastically bad thing. Some people (I'm not among them BTW) think it would even be good for Linux.
> Alan and Inigo fight over the "hot seat",
Do you even know these people? I don't, but I know enough about Alan at least to say that I would be very surprised to "fighting" over anything in that way.
I'm almost regretting replying to this. I almost think it's a proper troll.
I don't mean to be insulting, but all I see in what you have said is that you are afraid because you don't understand why the Linux development model works. It seems you would be more comfortable seeing a more traditional approach, but for no rational reason.
-- MartinG To mail me: echo kewyjlcxyzvjfxbqwh | tr bcefhjklqvwxyz
Anyway, Linux only has a few years of development left. The system we run 10 years from now will be Free Software and will contain a lot of the same tools, but might well have a different kernel. I'm surprised that people don't consider this more.
Thanks
Bruce
Bruce Perens.
> RMS's loss would cut the heart out of the free software movement.
RMS is a great inspiration to me. I now use and write free software because I believe in it, and this is due in part to RMS's work. However, if he were suddenly abducted by aliens I wouldn't suddenly stop believing in, using, and writing free software. In time another figurehead would emerge. There is always someone else.
Similarly, if Linus disappeared, I wouldn't stop reading the kernel source, trying to find and fix bugs and I doubt others would either, and in a short time the right person would assume his position, (probably Alan in the short term I have to admit,) it would ultimately be the best person for the job as decided by all Linux contributers. Such is the nature of the development model IMO.
Yes, of course there would be short term upset if Linus left, but there is upset in any model if the key people suddenly leave. The question is, in which model is the quickest recovery? I know where my money is.
-- MartinG To mail me: echo kewyjlcxyzvjfxbqwh | tr bcefhjklqvwxyz
Much of this is based on the OS design. The original RTLinux people looked at implementing realtime in plain old Linux, but decided it couldn't be done without massive work and risk to the API integrity. Of course, they were after hard realtime. Mainstream Linux would benefit greatly from "as good or better than Windows" realtime; hard realtime is only truely necessary for certain classes of embedded apps.
Another thing they don't mention, but should, is the need for a reentrant kernel and real, actual, useful, and working threads in Linux. This goes hand in hand with well written realtime/near realtime applications. Being preemptable is a whole lot more useful when the multitasking is far more fine grained than you have in today's Linux. Also, kernel-mode threads (as you have in BeOS and NT) are particularly useful for realtime-sensitive data.
-Dave Haynie
The SMP support already allows multiple processors to execute much of the kernel code already, except within locks. In principle, any code that is not in a lock and runs on SMP can be made preemptible on a single processor. So this is not a hugely revolutionary step, but rather evolutionary.
I'm glad that someone is taking the effort to actually do this.
I'll try to answer/respond to the most significant questions and points made here. 1. 2.5 is the next "development kernel", and is clearly the right place to initiate a fundamental kernel change such as preemptability. 2.6 is the right place for commercial usage, obviously. 2. Preemptibility means the systems can and will stop a process running in kernel mode and switch to a higher priority newly executable process. For more, including specifics of what our prototype does and how we want to advance it...read our white paper! Here isn't the place to repeat those details. 3. We are not dogs, but after a few beers or missed shots on the pool table we do howl alot. 4. We actually did initial work on 2.2.14 but our primary goal is proving out this technology, providing to customers who may need it, and offering back as a technology for 2.5 (->2.6). Hence, 2.4 was and is the proper base, given that we think we can have production ready software early next year, "long after" 2.4 is cooked. We feel releasing a prototype on 2.4-test software is okay. 5. A preemptable kernel is not a rate monotonic scheduler! We also offer a scheduler that sits on top of (runs before) the standard Linux scheduler that does a good job of selecting/dispatching real-time processes very quickly and with fixed overhead. It's not rate monotonic, it's a simple priority driven scheduler. 6. I won't engage in arguing the pro's and con's of "one man's control of an OS". It is the way it is. MontaVista lives with it and is trying to work with it. We think a preemptible kernel design, leveraging the SMP locks, is a good next step for the Linux kernel and benefits all "market segments" (types of users). We want to prove it with real software, and this is our starting point. We are "releasing early, releasing often", amen. 7. "these guys proposal was regeted"...I reget to tell you that you misunderstand. What was "rejected" was a set of patches that injected preemption points, which is a fundamentally different technology than what we've implemented. We have not proposed or submitted anything as of yet, beyond making our current prototype available. 8. Fross, THANK YOU. 9. Inclusion in 2.4 is absurd, it's time for it to stabilize, finalize and start getting deployed. 10. Most of the systems that Hard Hat Linux get designed into aren't serving "multiple users". They are control oriented environments. In such environments, having a design ability to assure that high priority tasks get cycles very quickly, with assurance, and all the cycles they want until they are "done", is critical. We are trying to enable that capability. Superuser privileges usually protect real-time priorities from casual mis-use on multi-user systems. 11. RTAI and RTLinux are "sub-kernels" which provide a lower level multi-tasking environment, a small set of services (mostly around synchronization and passing data to/from higher level Linux processes), and which virtualize interrupt off/on operations in Linux. Basically, they provide a multi-threaded interrupt handling environment with better worst case interrupt off characteristics. There are limitations to such an environment. A preemptible Linux kernel attempts to improve PROCESS responsiveness in Linux proper (as opposed to attached a small underlying, separate multi-tasking environment). They are quite different. They are not in conflict, and even with excellent preemption performance, a system can still benefit from improved interrupt responsiveness via use of RTLinux. 12. MontaVista did not request/support the request for preemption point inclusion in Linux. The perspective of our real-time team is that preemption points are NOT a desirable long-term technology, and we don't support their inclusion in Linus's kernel. 13. Everything MontaVista develops on our own dime is released as GPL or LGPL software. Everything to date anyway, and we don't see that changing anytime soon. Even most of what we've been paid to develop user professional services has been GPL/LGPL licensed (with our customers approval of course), as it's turned out. We are dedicated to the values of open source. We're a Linux company for gosh sakes! 14. Minimizing throughput impact on a uniprocessor is important. There are opportunities for throughput improvement under busy workloads...very simple workloads won't improve and could degrade, the questions are how much, and is the tradeoff worth it in order to get responsiveness/consistency of high priority behavior. Exploring, proving and educating in this area is an important component of our prototype work. 15. No our prototype doesn't yet support SMP. The cost of synchronizing across processors using the SMP locks shouldn't be more than synchronizing across processors using the SMP locks...don't know where you are coming from suggesting what we are doing will be prohibitively expensive on a multiprocessor! We'll share an SMP version when we get one put together (but it's not highest priority for us, our first order target for this thing is uniprocessors, which are far more prevalent in the embedded market segments.) 16. This is available for an IA32 platform. If you think that's an obscure platform...I guess you live in a parallel universe where Motorola hardware got selected by IBM back in the early 80's for their PC, or something! We'll productize on all supported MontaVista platforms (21 different boards across 4 different architectures as of our 2.1 release 2 weeks ago...). 17. I'm not aware Ingo did fully preemptable kernel work, and if Ingo did any work on such a system, we weren't aware of it and did not base this on any such work. 18. Preemptible is not interruptable. We've separately and previously done characterization work on interrupt off times in Linux. See our real-time project area on our web pages (www.mvista.com). Interrupt off times in Linux are surprisingly good, with one exception (child process exit with many children)...and of course ignoring arbitrary drivers that can misbehave. 19. Time to go howl... Sorry to be long winded. I hope this helps to clarify a few things. -Kevin