Robert Love, Preemptible Kernel Maintainer Interviewed

Tom F writes: "LinuxDevices did an interesting interview with Robert Love, the maintainer of the Linux preemptible kernel along with MontaVista. It is an exciting read and has Robert's usual wit and insight."

Comments

On Moderation and the Future of /.

[CmdrStalin]

Hi there. I'm CmdrTaco, one of the ruling despots here at slashdot. Earlier today I realized that I've been deeply troubled by recent events here on slashdot, more troubled than I've been willing to admit to myself. The plebs here have been making alot of noise in the last 24 hours, ever since this thread arose on the "Oracle Breakable After All" story. My attitude was the usual "fuck them all", and I thought the matter undeserving of even a moment of my time. But then a strange thing happened; I was stricken by impotence. I beat and I beat and I beat, and yet even with the aid of the very finest tentacle-rape anime divx selections available, I found myself unable to climax. Even having Hemos choke me with his belt till my face turned blue failed to do the trick. Clearly, something was terribly wrong.

And so, I went soul-searching. I thought back on the other night and the bitchslapping rampage I went on in that thread. I thought of some of the very low UIDs whom I'd smacked down, I thought about the many user moderations that me and the other editors had unjustly overturned with our unlimited mod points. I considered the outcry, the suggestions that we were ignoring the will of our own community, that we were evading a subject that desperately needs discussing, that we were acting autocratically and making hypocrites of ourselves. All these things I pondered, and I pondered long and hard. And this is what I have to say:

Go away.

When me and Hemos began this site so long ago now, we were starry-eyed small-town lovers; young, naive, and drunk on the possibility that a real revolution of sorts might be on the horizon, and that it might involve small furry animals. And so we opened up our labour of love to the world, never suspecting that it would become even half of what it has become today. Things were good, for a time.

But things change. None of you could ever understand just what it feels like, and what it does to a man, to have such a large userbase grow up under your shoddy perl scripts. Unless it happens to you firsthand, you can never truly know. They say that power corrupts, and I am here to tell you that it is true. Once, I cared. I really did. But you people... you went from being individuals, from people I could care about and chat with on #slashdot, to becoming a giant formless mass.. a formless mass seperated into factions; an army of idiots filling my inbox with crap and my RAID array with drivel. Every thread now, I'm lucky if I can find a single comment that doesn't appear to be stamped down by a dim-witted cookie-cutter of a mind. Endless parrotting, pompous bullshit, and people so one-dimensional that it is frightening. I have a secret for you: I too believe that the absurdist noise and distraction of the 'trolls' is the most worthwhile part of slashdot these days. I don't care about anything anymore, save bukkake vids and the occasional good troll.

So then, why do me and my editors censor with an iron fist? Why do we stamp every bit of humour and joy left in this place out and try to make it more and more soullessly grey with each passing day? Because I don't want to have a dialogue with you fucking plebs about the deterioration of this site, I don't want to save it; I want this failed experiment to implode upon the weight of its users' own hypocrisy and hyperbole. You say this site should be a microcosm for the free and open society of the future? Don't you see that it already is? And you wonder why it stinks of shit...

It will continue as it has. The majority of you will come regardless of what we do. I could piss on your mothers and assfuck your fathers and you'd still come to stroke your egos and share your reactionary politics with the world and to waste countless more hours of your miserable days away. The arms-race will continue. You think a bit of mass -1 Offtopics is bad? Brotha you ain't seen nothin yet. The editors will continue to be my hand-picked squad of the biggest fucking assholes I can manage to dredge up. We will continue to ignore your whinging about censorship, indeed we will ramp it up, because we don't fucking care. There will be countless more stories about the latest lame yuppie toys, the latest minor kernel revision, "look what I found in subsection 123.23.1c of this license!", and all manner of other predictable crap for you to hurl your canned responses at and exercise your dogmas on. Katz will continue to offer you his unique brand of insight until the end of time. And we, as always, could care less what you think or want. Droids and trolls alike; you will all lap it up.

You were expecting some great revelation or commitment to reform to become of this? hohohoho... as if I give a damn about the silly dramas you fools generate out of bit-patterns slapping into my DB. I was troubled, but only because I cloaked my true opinions in silence and continued to play the role of naive otaku-boy and open-source ass-puppet. Because I wasn't being honest with myself or with all of you. Now I can speak the truth, and my soul is unburdened. You may now go back to wallowing in the shit; I'm off to blow my load in Hemos' pretty pink face.

Remember this: we are only a mirror.

Thank you.

Re:On Moderation and the Future of /.

[Anonymous Coward]

I havent been visiting slashdot because of its low quality and groupthink, and only jumped back over after I heard about this article.

I just wanted to say that I think slashdot really sucks, and thats why I havent been here in over 8 months.

Kernel Maintainer

[Anonymous Coward]

I've got questions about kernel maintainers. When would you want to put someone on the job of maintaining a kernel version?

Obviously, you want someone that knows the kernel really well and can maintain every part of it.
BUT
Aren't you taking a good developer (who can maintain every part of the kernel) away from the newer versions of the kernel?
There are only so many developers, what happens if you run out?

Re:Kernel Maintainer

[Nothinman]

Obviously, you want someone that knows the kernel really well and can maintain every part of it.

That person doesn't exist, not even Linus knows every part of the kernel inside and out.

You have to trust the maintainers of their parts of the kernel because as good as Linus, Marcelo, Alan, etc are they can't know all the gotchas, etc of all the drivers and different kernel subsystems.

Aren't you taking a good developer (who can maintain every part of the kernel) away from the newer versions of the kernel?

They don't have to do it if they don't want to or don't have the time. But with Alan's recent want to not maintain 2.4.x so he can work on other things seems to say how much time is really required for the maintenance of a kernel tree.

There are only so many developers, what happens if you run out?

I highly doubt there will ever be that many currently maintained kernel versions.

Re:Kernel Maintainer

[Anonymous Coward]

Thanks. I'm a regular reader and contributer, but seeing that I thought I was gonna get ripped for asking a question like that, I posted anonymously. THanks for being candid and nice with your post...

Re:Kernel Maintainer

[cduffy]

There are only so many developers, what happens if you run out?

If the supply of Linux kernel developers runs out (as unlikely as that seems), then kernel developers get hired away from different kernels.

Seriously... there are folks who, for commercial reasons, need a lot of this work to be done; and a developer with real-time experience from a different UNIX kernel can familiarize with Linux within a reasonable amount of time.

For instance, quite a few of the folks MontaVista has hired to do kernel work come not from a Linux background but from working on other Unices -- one fellow we hired to work on preemptability originally did the same variety of thing for IRIX, IIRC.

Re:Kernel Maintainer

[mattdm]

Aren't you taking a good developer (who can maintain every part of the kernel) away from the newer versions of the kernel?

Stability and code maintenance are just as important (or more important, in some cases) as new and exciting features. Work on older kernels isn't wasted -- any relevant fixes can be ported up to the devel version.

Re:Kernel Maintainer

[Anonymous Coward]

You're not quite right about Marcelo's job as kernel maintainer. Yes, he is the primary maintainer but he is not the only maintainer. He probably only does direct maintenance of the code that he wrote.

But he didn't write all of the device drivers, the scheduler, the pager, the I/O subsystem, the memory manager, the whatever else. Other people wrote those and will probably continue to work on them as problems of bugs become apparent.

His primary responsibility is to oversee maintenance of the 2.4 kernel. Others submit bug fixes and patches to him for their individual components, he tests them, integrates them into the main tree, and releases a new version when appropriate.

Think coordinator, not maintainer.

SDH

Why do we need so many different kernels?

[Astral Traveller]

I fear for Linux' future now. It seems almost certain that, with Love's preemptible fork, Alan Cox's own fork which is steadily separating from Linus' core, and of course the "classic" Linux kernel, the Linux operating system will soon end up like *BSD, with several mutually incompatible, infighting factions. We can't let this happen.

Instead of making their own forks, Love, Cox and Torvalds *MUST* compromise, unless they want to doom Linux to the same marginalisation that BSD suffered due to excessive politics and infighting. Corporations are already being frightened away from Linux due to the diversity of distributions available; if this excessive diversity were to spread down to the very core of the operating system, the kernel, maintaining commercial software would exceed being just difficult and become almost impossible. We can't let this happen, or else Microsoft will eat us for lunch.

Re:Why do we need so many different kernels?

[Cyph]

I do believe that what Love is working on is in fact a patch, and not a fork.

Re:Why do we need so many different kernels?

[Nothinman]

Love doesn't maintain a complete seperate kernel tree, just a patch that applies to the main Linus tree.

Re:Why do we need so many different kernels?

[jonbelson]

>the Linux operating system will soon end up like
>*BSD, with several mutually incompatible,
> infighting factions. We can't let this happen.

Where on Earth did you get this nonsense from? There are three open-source BSDs, each with a different focus. Admittedly a few OpenBSD and NetBSD developers aren't the best of chums, but most of the developers are happy to share code between projects (eg. dirhash, smp, usb etc)

--Jon

Re:Why do we need so many different kernels?

[benedict]

You need to spend some time studying history before
shooting your mouth off.

The main reason for BSD's "marginalization"
relative to Linux is the AT&T/Novell lawsuit.

It's also worth noting that the different BSDs are
no less mutually compatible than the different Linux
distributions.

Re:Why do we need so many different kernels?

[Anonymous Coward]

I hate to take you offtopic, but can you provide a bit more info? I am really interested.

Re:Why do we need so many different kernels?

[gmack]

Complete and utter BS.. forks are a good thing Linus has said so himself. They let diffrent people try diffrent approaches to the same problem or include features not yet elegent enough to be included in the main line kernel.

And while I'm at it: Alan Cox regulairly submits parts of his patches to linus / marcello for inclusion in the main line kernel. A lot of it gets merged but some things Alan doesn't think are good enough and some things Linus disagrees with.

Re:Why do we need so many different kernels?

[mattdm]

...Alan Cox's own fork which is steadily separating from Linus' core...

Well, Alan isn't really in the business of doing this anymore. But even when he did, that's not the way things worked -- the Alan and Linus trees stayed fairly in sync with each other in many ways.

Re:Why do we need so many different kernels?

[benedict]

You mean about the lawsuit?

Linus himself has said that if BSD hadn't been
under a cloud of legal uncertainty in 1990, he
might not have embarked on the Linux project.

You can get more information about the lawsuit
here, just search for the string "the lawsuit".

Re:Why do we need so many different kernels?

[Anonymous Coward]

It seems almost certain that, with Love's preemptible fork, Alan Cox's own fork which is steadily separating from Linus' core, and of course the "classic" Linux kernel, the Linux operating system will soon end up like *BSD, with several mutually incompatible, infighting factions. We can't let this happen.

As long as the API's are the same, and they maintain binary compatabilty for applications, what's the difference? Seems to me it would actually be a Good Thing. You could then choose the kernel/distro that optimizes performance for your particular choice of applications.

Re:Why do we need so many different kernels?

[Anonymous Coward]

We need them the same way that microsnot needs different versions. For desktops, web appliances, game controllers, automobile control systems, mobile phones and other mobile devices, super high security systems... etc. One size will not fill all. With the usage of modules you can create a variety of systems for the different needs.

Re:Why do we need so many different kernels?

[snake_dad]

It's the BSD license. Who can prevent sharing? Even if they wanted to?

Re:Why do we need so many different kernels?

[Grayraven]

hubris. that's what prevents sharing.

Preemptive kernel looks good

[Marx_Mrvelous]

His comments on the preemptive kernel patch make me want to try it out. It seems like a great idea, and from what he says, it's well-implemented.

On a side note, he's my kind of geek. At least started with Slackware, kernel hakcs and even has a girlfriend. Wow!

Re:Preemptive kernel looks good

[mcramer]

It seems like a great idea, and from what he says, it's well-implemented.

Pre-emptable kernel: Bringing the multitasking of MacOS 9 to Linux since 2.3!

Re:Preemptive kernel looks good

[Anonymous Coward]

Wrong!

OS9 uses cooperative mutitasking, just like Win3.X. Win9X/NT/Linux/FreeBSD/etc. all suppored preemptive multitasking.

Re:Preemptive kernel looks good

[Toraz Chryx]

*cough*AmigaOS*cough*

preemptive multitasking since 1985 :)

Re:Preemptive kernel looks good

[Anonymous Coward]

Yes but was it the kernel that was preemptable or just the user apps? Linux has always had preemptive multitasking just the you could not preempt the kernel.

Re:Preemptive kernel looks good

[Toraz Chryx]

I'm not entirely sure actually.

finding good technical docs online for the amigaOS is something of a hurdle :/

(it wouldn't surprise me if they kernal WERE preemptable however)

Re:Preemptive kernel looks good

[Anonymous Coward]

Well, there was no true memory protection as such in the AmigaOS, so the distinction between kernel and user space was very blurry - basically, exec.library handled the functions that a typical microkernel would have, and some of its functions were preemptible. Essentially, whether a "syscall" (effectively indistinguishable from a shared library function call, not implemented via interrupts or call gates like on memory protected architectures) was preemptible or not was handled at a per-function level - some critical functions were wrapped with Forbid()-Permit() pairs, which produced non-preemptible behaviour by suspending interrupts. There was also a cooperative (i.e. you application had to be coded to use it, like the way on a cooperative multitasking system you application has to choose to yield the cpu) semaphore locking system which was the amiga's only "memory protection".

Re:Preemptive kernel looks good

[Anonymous Coward]

There was also a cooperative (i.e. you application had to be coded to use it, like the way on a cooperative multitasking system you application has to choose to yield the cpu) semaphore locking system which was the amiga's only "memory protection".

No, this was nothing like memory protection.

The Unix term for semaphores is "Mutex" - Semaphores are a mechanism for applications to share data safely - so one process doesn't write to shared data that another process is using. (because the Amiga had no memory protection, essentially all data is shared.)

Memory protection is very different.

Re:Preemptive kernel looks good

[Lozzer]

I think you'll find that Linux has premptive multitasking too. What it can't do (without the preempt patch) is prempt a task that is currently running kernel code (e.g. through a syscall). I've no idea whether the Amiga's "kernel" (exec?) was preemptible in this sense or not.

Impressive.

[VA Porware]

Anywhere from a couple hours a week to many hours a day.

Where do these college students get all this time to hack on the kernel? Don't they have to study at all?

Re:Impressive.

[Gkeeper80]

If I were good enough to do kernel hacking without taking classes on it, I wouldn't have to study either...ah who am I kidding, I don't actually study

Re:Impressive.

[wscott]

> Where do these college students get all this time to hack on the kernel? Don't they have to study at all?

I had tons of time in college. Too bad I spend it all writing borgs for netrek...

Re:Impressive.

[renehollan]

I had the most time to do serious software work when I was in school. After I started working, very little of what I did or had the time to do was significant anymore, and that which was got done only by fighting tooth and nail to do it right.

Heck, I don't even think that Teradyne has yet released all the changes I made to GPL code for them (of course, I don't know that they even released binaries so don't presume a GPL violation).

Re:Impressive.

[mattbelcher]

He goes to the University of Florida. As a recent grad (last May) with a degree in CS and Math (the same as what he's working on) I can tell you, if you need the time, its easy to find. I was busy, but only because I pushed myself - taking many more classes than I needed to. Hell, if you are intelligent and willing to accept C's, you probably don't even need to go to class or finish half the assignments.

As an aside, I'd love to see how he breezes through COP4600 (Operating Systems). Adding symbolic linking to Minix would be cake after the work he's done.

Re:Impressive.

[Anonymous Coward]

When I was in college, I normally didn't studied. Just went to class. Studying is for the inattentive.

The exceptions were the first few semesters where there I still had low-level math classes. For some reason, those guys like to assign lots of homework, so just paying attention in class isn't enough.

Re:Impressive.

[John Whitley]

Don't they have to study at all?

What in the heck do you think this IS? Many of the best students treat developing their coding and CS skills more like a musician or artist practicing and performing for many hours a day. It's a creative act, and can be very involving. Moreover, that level of involvement hones problem solving and practival skills that the "just a job" students can never hope to achieve.

I always wondered about students who didn't have any passion for the field. From what I've seen in both academia and industry, that "just a job" mentality reduces one's skills to "programming fodder", and would seem to be a pretty unenjoyable career.

Re:Impressive.

[soulsteal]

Studying comes after Kernel Hacking and right before Binge Drinking. If we're not lucky, then the Kernel Hacking would come after or during the Binge Drinking

Re:Impressive.

[arkanes]

Wouldn't any reasonable educational facility let him test out of classes like that? I mean, jeez, it's not like he doesn't know his stuff.

Re:Impressive.

[Miles]

I believe it's up to the student, usually, to take the initiative and ask to take those kinds of tests, if they're even available, which can often depend on the department, the faculty, etc...

Besides, just because you can do a lot of practical stuff, doesn't mean you'll know the theoretical stuff which may the only thing being tested.

Re:Impressive.

[arkanes]

It just occured to me that, if he's unlucky, he'll get one of those proffs that likes to test you on specific bits of obscure trivia rather than on concepts and ideas, in which case he'd have to cram even if he were every kernel maintainer wrapped into one :P

Re:Impressive.

[mattbelcher]

Sigh. Unfortunately, UF has no policy for proficiency exams. COP4600 is a required course for CS majors, so I imagine he'll be taking it like everyone else. Its not all bad though. Its possible that Robert only has experience with Linux, in which case it will be good for him to see/experiment with other OS designs (like Minix's much vaunted microkernel). Most of the class is theorectical, but it does let you get your hands dirty hacking on Minix.

One things for sure, having Robert Love in my class would have made things a lot more interesting. He'd at least put those TAs in their place.

Re:Impressive.

[Mr_Matt]

Wouldn't any reasonable educational facility let him test out of classes like that? I mean, jeez, it's not like he doesn't know his stuff.

What, and not get tuition for the semester's worth of classes the student's tested out of? Surely you jest. :) And besides, even if one is so competent that they can already take the class, that doesn't necessarily mean they won't learn anything from taking the class again - provided they take the class with the right attitude. Besides, it's an easy "A", right? :)

Re:Impressive.

[nexthec]

cs major right? Obviously not an engineer

Re:Impressive.

[Anonymous Coward]

i went to high school with rob. trust me, he definately doesnt spend any time on school work(as is illustrated by his GPA), though he is still quite possible the most intelligent person i have ever met. keep in mind that he is a 19/20 year old sophomore in college that is doing what most CS grads could barely pull off

Linux + Kernel Premption = AmigaOS 2002

[Anonymous Coward]

'bout frickin' time as well I say. But can it run in 512KB of RAM whilst playing music, drawing fractals and managing files simultaneously.

Gawd, is this an AmigaTroll (tm)? Must click on "Post Anonymously"!

Re:Linux + Kernel Premption = AmigaOS 2002

[Toraz Chryx]

It's okay, your not alone in missing the AmigaOS

damn commodore for being a bunch of mismanaging fuckwits.. bah!

If they'd kept the R&D flowing smoothly, the current Amiga's would probably resemble the Gamecube, from an architectural standpoint. and that would be really rather cool :)

Re:Linux + Kernel Premption = AmigaOS 2002

[FrostedChaos]

Commodore had some managment problems, but their intervention probably extended, rather than cut short, the Amiga's lifespan. Unfortunately, once the world switched to the (mostly) open architecture of PCs, Amiga became just another proprietary dinosaur.

One good question is why Apple survived while Commodore didn't. One possible answer is that Apple had a cult following, whereas Commodore usually tried to compete based on price. Also, game consoles like the NES and Genesis took over the low-end home entertainment business. No matter how cool a product is, if it doesn't meet consumer's needs, it might as well be put in a museum.

Re:Linux + Kernel Premption = AmigaOS 2002

[Toraz Chryx]

it was Commodore's x86 division that eventually dragged them under however, the Amiga division was profitable.

Re:Linux + Kernel Premption = AmigaOS 2002

[Anonymous Coward]

The x86 division was cut long before they went under--they weren't even carrying parts for the x86 line before they sold support to CSG.

Re:Linux + Kernel Premption = AmigaOS 2002

[Zarquon]

Ah yes.. I remember our first computer. A Commodore PC 10; 20 MB harddrive, 640k of ram, CGA graphics, and a single 5.25". 4.77 Mhz 8088, IIRC. Only thing we had for 7 years until the hard drive gave out..

Re:Linux + Kernel Premption = AmigaOS 2002

[IntlHarvester]

One good question is why Apple survived while Commodore didn't.

Mainly because Apple sold $5-10K machines with a 60% profit margin and Commodore was selling cheapo $1500 boxes to the home market.

The prices killed Apple's chances for mass market adoption, but they still have billions in the bank from the old days, which has afforded a massive number of fuckups.

low-latency patch

[S. Allen]

I'd also like to see an interview with the author of the low-latency patch as well as a comparison of the two. Bits I've read from Kernel Traffic give me the impression that people think the low-latency patch generally provides better responsiveness and scalability.

Re:low-latency patch

[steveha]

I have been following these two patches a bit, because I want my desktop to be as snappy and responsive as possible.

The current low-latency patches work by finding "hot spots" in the kernel code where something is taking a long time, and then putting in a hack to make the code yield. The good part is that you can get a really low-latency kernel; the bad part is that you have to touch the kernel code in hundreds of places, and the kernel code gets really ugly. I remember reading that Ingo Molnar, who wrote a giant low-latency patch that worked this way, agreed with Linus that his low-latency patch was just too ugly and huge and should not be included in the main source tree.

The preemption patch is comparatively small and elegant. It leverages the work that has already been done to make SMP work correctly. I'm using it on my Linux desktops, and I like it.

On one of the mailing lists, Linus said that he wants the Linux kernel to gain low latency the cleanest way: find all parts that are slow, and instead of hacking them to yield, re-write them so they are faster (but still clean code that is easy to understand). This is of course the ideal, but when will it be finished? The preemption patch is available now, and works now, and I am using it now.

steveha

Re:low-latency patch

[Catiline]

On one of the mailing lists, Linus said that he wants the Linux kernel to gain low latency the cleanest way: find all parts that are slow, and instead of hacking them to yield, re-write them so they are faster (but still clean code that is easy to understand). This is of course the ideal, but when will it be finished?

Well, I'm just going to hazard a wild guess here, but maybe that sort of change would only occur when all of the different areas' (primary) developers were interested in it.

Re:low-latency patch

[pthisis]

The preemption patch is available now, and works now, and I am using it now.

The problems with the preempt patch are several:

1. It makes kernel debugging considerably harder
2. It introduces potential livelocks which could hang the system (this is an inherent part of the design and not an implementation bug)
3. It doesn't work with SMP (yet), this is fixable
4. It doesn't improve the worst-case latency.

#4 means it isn't a viable alternative to low-latency style patches; Love has said as much on linux-kernel, though obviously he feels it's a nice complement. 1 is a large hill to climb on its own, and 2 basically dooms it unless a clean solution can be found (most proposed solutions involve some sort of priority inheritance, which is a rat's nest).

In other words, the patch "works" only in a very limited sense; it doesn't decrease worst-case latency, can potentially hang the system, and doesn't interact well with SMP. It does decrease average-case latency.

The preemption patch is comparatively small and elegant.

Uh, that's in the eye of the beholder I guess. It is small, but it's not really elegant; it essentially causes conditional reschedules to happen all over the place, and by design cannot deal with latency in noninterruptable interrupt contexts--which happen to be exactly where the worst-case latencies are. So preempt as it's currently designed simply cannot address the worst-case latency. And it causes far more runtime conditional reschedules than the LL patches.

The low-latency patch decreases both worst and average-case latencies. It also doesn't have livelocks as an inherent part of its design. And average-case latencies with LL are as good as those with LL+preempt (in other words, preempt gains you nothing over LL). LL, unfortunately, touches a lot of code and has conditional schedules sprinkled all over the place--it's not elegant either, and certainly not small. But unlike preempt, it actually decreases the worst-case latency, which is what's important to the soft realtime applications that gave birth to both patches. There's a mini-LL patch as well which has far fewer conditional rescheds, that's being discussed for potential inclusion in 2.4.x and/or 2.5.x

My gut feeling is that preempt is the wrong approach to the latency problem, but the jury is still out on it. I don't think either the current preempt or the current LL are ready to go in the kernel. If you're doing e.g sound mixing or other latency-sensitive work right now, the full LL patches seem like the best alternative available. They get latencies down extremely low, well below those of e.g. BeOS.

Sumner

Re:low-latency patch

[GooberToo]

You bring up a good point. These patches originate to satisfy soft-realtime issues. As is often the case, many users simply want a fast response in their application and GUI in general. While saying that LL may be a magic bullet, might also the O(1) scheduler place some light on this by allowing those that simply want to have a "snappy" interface to get what they want? After all, do we really care is the interface is "snappy" 90% of the time and 10% not when under a load in their worse case latency situations? I personally would take that as I feel that the lower overhead and higher scalability gained by the scheduler more than makes up for it. At that point in time, doesn't addressing the last 10% of the high latency code by optimizing it seem more of an obtainable goal?

This isn't to say that these patches don't have value, clearly they do, however, I'm trying to point out that it seems there are really two camps. Those that *need* LL for soft-realtime and those that *want* a LL-like environment so they have click a button more timely. Of course, the extra plus with the scheduler is that it should scale better than the existing schedule and is very SMP friendly. Seems like this is where the average user should be looking.

Am I missing something?

Greg

Re:low-latency patch

[pthisis]

Those that *need* LL for soft-realtime and those that *want* a LL-like environment so they have click a button more timely.

Well, I think that some sort of soft realtime is wanted pretty widely, if only to stop music and movies from skipping. LL does that fairly well, O(1) doesn't. But O(1) is definitely a win, modulo a few nits, and it's already been integrated into 2.5.x--with some work on those nits courtesy of Davide Libenzi.

So, I think O(1) is good but it's not a replacement for LL or another worst-case latency reducer, even among people who don't know what "soft realtime" means. They still don't want their mp3s to skip.

Sumner

Re:low-latency patch

[be-fan]

1. It makes kernel debugging considerably harder
>>>>>>>
Then don't compile it in while you're debugging. Most *users* don't debug their kernels.

2. It introduces potential livelocks which could hang the system (this is an inherent part of the design and not an implementation bug)
>>>>>>>>
True, but the SMP support introduced these to begin with. The preemptive patches just bring that danger to UP machines.

3. It doesn't work with SMP (yet), this is fixable
>>>>>>>>>>
Already been done. As of around 2.4.10, it experimental, but working.

4. It doesn't improve the worst-case latency.
>>>>>>>>>>>
It's not designed to. Love has already started work on a lock-breaking patch to get rid of long-held locks.

Re:low-latency patch

[steveha]

It makes kernel debugging considerably harder

But only if you actually compile it in to the kernel, right? Including the patches, which are not enabled unless you choose the option while configuring the kernel, wouldn't mess up kernel debugging, would it?

It is small, but it's not really elegant

I am not a kernel hacker, and I was not commenting on the code; I feel that the idea of leveraging the SMP locks is elegant.

I am one of the large pool of people who don't need realtime features, but do not want our movies or music playback to stutter when the system gets busy. The preemption patch seems to be doing that for me, but I might be interested in the full low latency patch.

Could you explain what "live locks" are, please? I'm not familiar with the term.

steveha

Re:low-latency patch

[pthisis]

Then don't compile it in while you're debugging. Most *users* don't debug their kernels.

If a user gets an oops and submits a bug report to linux-kernel while running preempt, the bug report is a lot harder to decipher.

True, but the SMP support introduced these to begin with. The preemptive patches just bring that danger to UP machines.

Not true. preempt introduces new hangs. Read the threads on linux-kernel, especially "Re: [2.4.17/18pre] VM and swap - it's really unusable".

Some things that are broken by preempt:
* Network drivers which disable IRQs to avoid spinlocking on uniprocs (major performance win)
* Drivers which use per-CPU data to avoid spinlocking at all
* Drivers which disable individual interrupts for long periods of time
* Drivers which depend on consecutive lines of code executing near each other in time, especially serial drivers

That thread has details on all of these.

And there's the priority inversion scenario:
SCHED_OTHER process 1 acquires a semaphore in kernel mode
SCHED_FIFO realtime process needs the semaphore, blocks on it

Now the rt process is stuck pending progress from the SCHED_OTHER process. Without preemption, the SCHED_OTHER process would have done whatever required the semaphore and released it. Now, SCHED_OTHER process 2, 3, ... n may be scheduled, leaving the realtime process twisting in the wind until the system eventually gets around to rescheduling process 1. If process 1 is e.g. SETIatHome or another nice 19 CPU hog, and process 2 is mozilla, you can get huge latencies on the realtime process, way bigger than without preempt. And even without SCHED_OTHER you run into the same problem, it's just not quite as easy to illustrate.

[Note that I said semaphore, not spinlock, so the lock-break code won't help.]

Do you not remember all the problems with priority inversion and the SCHED_IDLE patches? This is exactly the same problem, it's not like it's something new and mysterious that people are making up as FUD to stop preempt from getting in to the kernel. Any introductory OS textbook discusses it, and priority inheritance is the only robust way to eliminate it--with all the problems that come along with priority inheritance.

4. It doesn't improve the worst-case latency.
>>>>>>>>>>>
It's not designed to. Love has already started work on a lock-breaking patch to get rid of long-held locks
It won't help. The fundamental problem is that preempt in interrupts is impossible with this scheme (and AFAIK nobody has proposed a scheme that could even theoretically work), and the worst-case latencies are in interrupts. The secondary problem is that a lot of these issues are highly hardware sensitive; different hardware has different timing requirements, and the only way to find them all is to audit every driver and deal with it appropriately (as, e.g., the LL patches do).

As Alan Cox wrote,

The pre-emption patch doesn't change the average latencies. Go run some real benchmarks. Its lost in the noise after the low latency patch. A single inw from some I/O cards can cost as much as the latency target we hit.

Its not a case of the 90% of the result with 10% of the work, the pre-empt patch is firmly in the all pain no gain camp

Sumner

Re:low-latency patch

[pthisis]

Including the patches, which are not enabled unless you choose the option while configuring the kernel, wouldn't mess up kernel debugging, would it?

But if a user has preempt enabled and submits a bug report, that bug report becomes very difficult for kernel developers to decipher. Not to mention that hard-to-reproduce bugs may show up under preempt, then you have to build a non-preempt kernel and try for a potentially long time to reproduce the error.

I am not a kernel hacker, and I was not commenting on the code; I feel that the idea of leveraging the SMP locks is elegant.

It would be, if the locking requirements for SMP were sufficient to deal with the locking requirements for preempt. But they're not, these are really two different beasts--see my other message under this topic for a laundry list of reasons and linux-kernel topic to read.

Could you explain what "live locks" are, please? I'm not familiar with the term.

Definition of livelock

Sumner

Re:low-latency patch

[Zwane]

I remember reading that Ingo Molnar, who wrote a giant low-latency patch that worked this way, agreed with Linus that his low-latency patch was just too ugly and huge and should not be included in the main source tree. That would be Andrew Morton's Full low latency patch. He currently is maintaining a mini-LL patch as well. There are reports from various people (Alan Cox being one) stating that they have found no noticeable difference between running -preempt and -ll, and in some cases -ll providing better average latencies.

Re:low-latency patch

[steveha]

Thank you very much for the info. I have downloaded the low latency patches for 2.4.17, and I'll give them a try. Since I am not maintaining either the LL patches or the preempt patches, I don't actually care which one is smaller or more "elegant", I just want my low latency.

Thanks again and see you around.

steveha

True geek

[gorillasoft]

Like all true geeks, Love doesn't forget to include in his comments that, despite being a computer nerd, he does, in fact, have a girlfriend.

RL: Approximately how much time per week do you spend working on your kernel patch for Linux?

Love: My girlfriend would probably say too much. Anywhere from a couple hours a week to many hours a day.

Re:True geek

[Anonymous Coward]

Since when is being defensive about never leaving the house a good thing?

Re:True geek

[Ophidian P. Jones]

Don't worry, I'm sure she's ugly.

Re:True geek

[Anonymous Coward]

Sounds like you've never had to balance a real love life with the geek life. Unless the SO is also a geek there is always a constant tension between the SO and the computer. The wife's opinion on the matter is certainly the first thing that comes to my mind whenever someon asks me a similar question.

Anyway, a real true geek doesn't forget to include in his comments that his wife is a hot model.

Re:True geek

[Anonymous Coward]

Sounds like you didn't get the joke. I don't think that was a serious comment - note that the moderators apparently saw the humor in the post and go from there.

At the moment, yes

[Astral Traveller]

Right now it's just a patch. But Linus has made it clear that he is unwilling (for whatever foolish reasons) to accept Love's patches. If the preempt patches become popular enough (as they no doubt will amongst those using Linux for multimedia production), it won't be long before somebody decides to fork the kernel with preemptible patches simply to make it easier to use for more people. From there, you could very quickly end up with yet another completely different operating system kernel.

Re:At the moment, yes

[Nothinman]

If you read the interview you would have seen: "Linus said at ALS this year he was interested in the preempt-kernel patch. That doesn't mean anything to me until we are in, though, but it is a good sign."

And right now Linus hasn't let this and several other patches into 2.5 officially becaues he's focusing on the bio changes and it's much harder to debug if you have multiple subsystem changes going on at once. And Linus also has shown interest in Ingo's new O(1) scheduler, which the preempt patches have only recently become compatible with.

You just can't win

[wowbagger]

If you maintain different kernels, people say "OHMYGOD we are forking we will all DIE"

If you roll changes into a kernel and make it unstable, people say "OHMYGOD production kernel's not stable we will all DIE"

RTFA and lighten up. The patches are being considered for 2.5. They haven't been ruled out.

Because Linux is a Macrokernel architechture OS

[ithmus]

Remember when you had to compile your device drivers into the kernel yourself instead of using a module? The idea here is that the vital OS features are part of the kernel.

The open source movement is about modifying your software and sharing it. Anyone with the ability can modify a vital OS feature and share it. voila! Many, many kernels.

But, the problem here is that real time processing does not belong in a macrokernel architechture. Look at the commercial RTOS (Real Time Operating Systems) like QNX and you will see that a microkernel architechture -- a kernel that a provides minimal feature set is favored. This is because if you are depending on time constraints, all you want in your kernel is message passing and task syncronization.

Re:Because Linux is a Macrokernel architechture OS

[renehollan]

But, the problem here is that real time processing does not belong in a macrokernel architechture.

I'm inclined to agree, but I see no reason to exclude an attempt at "poor-man's" real-time processing in a macro kernel, by making the kernel preemptable. So long as one does not have serious hard real-time constraints (e.g. manned aircraft auto-pilot) and is concerned with snappier interactive response and media streaming this seams appropriate. This is certainly the case if it can be compiled in or out at will.

Re:Because Linux is a Macrokernel architechture OS

[Cato the Elder]

It's not that "real time processing does not belong in a macrokernel architechture" it's that "macrokernel architechture does not belong in a hard real time system"

I don't see any problem with making the linux kernel preemptible to be able to make better real time garuntees. Sure, I don't think you'll ever get hard realtime, but that doesn't mean that you won't get benefits from being able to respond to interrupts even when the system is running the kernel.

("hard realtime" -- maximum interrupt latency of xxx nanoseconds. "soft realtime" -- runs fast enough, usually)

kernel.org

[xanadu-xtroot.com]

Unfortunatly (as of 2:00 pm EST), it's seemed that kernel.org has been dead most of the day.

:-(

What's a preemptible (kernel) maintainer?

[mocm]

Or is that the maintainer of the preemptible kernel
patch ;)

I know it's not funny, but may cause a giggle.
Cordelia: "What's a rogue demon?"

Re:What's a preemptible (kernel) maintainer?

[Amazing Quantum Man]

It would have been much more fun if the title had been:

"Robert Love, Preemptible Kernel Maintainer Interview" (no -ed).

Then we would have had three different parsings:

Robert Love, (Preemptible Kernel) Maintainer Interview
Robert Love, Preemptible (Kernel Maintainer) Interview
Robert Love, Preemptible (Kernel Maintainer Interview)

OK, so I have waaaaay too much time on my hands.

Re:What's a preemptible (kernel) maintainer?

[Chirs]

A preemtible kernel maintainer is someone who can add a new and more important patch while he's in the middle of adding a less important one.

this one gets my vote

[spongman]

I've been running this patch since the early -ac days on a machine that doubles as a development desktop and a medium-load server and I've had absolutely no problems. I don't have any numbers but X/Gnome seems much more responsive even when other procs are busy. I see no reason why this isn't in the main tree, especially since it's a kbuild-time option.

a quick nice -n -5 /usr/bin/X11/X helps, too.

Re:this one gets my vote

[__aaedhn419]

Personally, I always go nasty -a * /sbin/init, and it seems to keep everybody in line. Sometimes you have to show X that you're the boss, not coddle and cuddle it.

What Kernel.org has been slashdotted ?

[CDWert]

Im joking of course but it has been down for a while. Anyone know how much this patch deals with the VM ? ,
I am asking because I am running a Rik Riel rmap-11c patch against 2.4.18pre4 and would love to try it (Rik's new VM IS SOOOOO Much sweeter than the old, I was swapping after 10 minutes with 512 megs off a standart 2.4.17, I shouldnt have been) Now I dont swap at all yet, which I shouldnt be with as littlew load,

It has improved my performance through the VM and I would love to try the preemt patch but am concerned it shares mem sctions with the VM, Anyone runnign both rmap and this ?

I could always use more speed, It is a desktop only machine.

Nice buzzwords

[yusing]

I "grew up" in the 8-bit era and recall that we used vectored interrupts to handle real-time needs. That worked extremely well with processors running at 1/1000 of today's speeds.

After reading the article, I'm left wondering why "unpreemptable kernals" were ever conceived in the first case. To lower the cost of hardware? On the "limited-access freeway" model? Or just to K.I.S.S.?

Re:Nice buzzwords

[Anonymous Coward]

It was because Linus incorrectly thought Tanenbaum didn't know what he was talking about. (Everyone seems to have the impression that Linus "won" that debate. They are wrong.) So instead of writing a modern OS with a microkernel (where there isn't much to preempt) he went with the 1970s approach. We'll all be paying for that until the day we switch to HURD.

Re:Nice buzzwords

[Anonymous Coward]

We'll all be paying for that until the day we switch to HURD.

Which shouldn't prove too difficult, if/when the time comes.

I was under the impression that Linus start writing Linux because he wanted to run software now, not when the Hurd was complete. In this respect, Linux is a success.

Re:Nice buzzwords

[Bob The Cowboy]

We'll all be paying for that until the day we switch to HURD.

Sweet Jesus! If HURD development doesn't speed up we'll be paying for a very, very long time....

;o)

Bill

Re:Nice buzzwords

[Anonymous Coward]

Except for the other free OS's right troll?

Has he never heard of Gnome 2.0?

[Jonathan Brown]

"I certainly think Linux is an excellent desktop system (both GNOME and KDE are quite mature) but I don't see huge strides here in the coming year."

In my opinion Gnome 2.0 is the coming of age of the Linux desktop.

Re:Has he never heard of Gnome 2.0?

[Anonymous Coward]

The Linux desktop depends solely on what OS comes preloaded on Joe Blow's machine. When Linux comes preloaded, then Linux is ready for the desktop. When Windows comes preloaded, then Linux is not ready for the desktop.

It has nothing to do with the merits of the software, which is why Gnome 2.0 is insignificant.

No.

[Astral Traveller]

While there are different Linux distributions, these all use the same kernel and the same applications, usually built from the same sources. A Redhat binary will generally run fine on Mandrake or Debian or Caldera or SuSE or what have you. FreeBSD, NetBSD and OpenBSD have become completely different operating systems with vastly different userspace code. Binaries are not at all portable between the operating systems. This is a far deeper and more severe fork than Linux, and if we continue down the road of having more and more Linux kernels (as we are seeing now), soon the same deep, irrevocable split could happen to Linux as well.

MS Windows?

[sitturat]

Is the MS windows kernel preemptible? Does anyone know or is it top-secret?

If a kernel is preemptible, can it be used for real time process control?

Re:MS Windows?

[gmack]

I think some parts of NT are and some parts of not.. windows 9.x/me are definatly not to prove this just watch your whole system slow to a crawl while writing to the floppy drive.

I don't think even with this the kernel will be good enough for some forms of real time process control. Real Time process control needs guarunteed sub millisecond response times and everything else including overall throughout is sacrificed for the cause.

It will however make the UI a tad more responsive and make it harder for things like web browser loading to make the mp3 player skip.

Re:MS Windows?

[Anonymous Coward]

what are you talking about mp3 player skipping? mine has never skipped under some pretty heavy loads.

Re:MS Windows?

[psavo]

Is the MS windows kernel preemptible? Does anyone know or is it top-secret?

AFAIK, it is not. I know that in Windows minimum latency is about 10ms, same as in default Linux kernel.

If a kernel is preemptible, can it be used for real time process control?

For what? Do you mean for something like.. robotics control? or shuttle devices control?
For that answer is a bit complicated, you see, even this pre-empt patch doesn't help when there are 'long' locks in kernel. There are situations when kernel disables all interrupts and does its things for long time (like 5 to 20ms). There are not many of these, but they do exist. Now, realtime kernel is one that answers any interrupt 'right away', and can be interrupted 'anywhere'. These are a LOT hairier than pre-empt patch.
If one doesn't have hard time limits, pre-empt patch delivers enough certainty on time-bounds..

Re:MS Windows?

[mark_lybarger]

ctrl+alt+delete -> kill task that's got the box all locked up. it doesn't seem like it's reemptible at all. when something's got the box locked hard, even the ole 3 finger salute won't bring up the process manager.

Re:MS Windows?

[SuiteSisterMary]

NT kernal is; at pretty much any time you can give the three finger salute and get the 'stuff to do' dialog. When you see a blue screen, that's NT having caught an exception that worries it enough to *voluntarily* halt itself, because it can no longer guarentee that it's working with valid data. It also spews out a large amount of debug data. Most such bluescreen occur due to hardware drivers. This is similar to a UNIX kernal spewing a bunch of kernal panic messages and the like to the syscon, then halting. Very rarely will you see an NT derived kernal simply 'freeze.'

Re:MS Windows?

[Anonymous Coward]

Yes and no. It is "preemptible" in the sense that you can give it the 3-finger salute and kill processes and it is a microkernel-based OS (although I've seen it called the world's biggest microkernel); however, it is generally not usable in embedded devices out of the box (I believe the reason is it doesn't have low enough latency to be useful, but I could be mistaken). MS itself plus at least a couple of other companies make "hard" RTOS wrappers for NT that basically run NT as a process. This is similar to RTLinux, for example.

Re:MS Windows?

[be-fan]

Yes, NT is fully preemptible (except when its holding some very critical locks). However, it is not a real-time kernel, since it makes no guarentees about latency. The stock kernel is fine for soft realtime stuff like audio/video things, but you can't do hard realtime with it. However, there is a product called RTX which claims to make WinNT realtime.

Re:MS Windows?

[UnknownSoldier]

> Is the MS windows kernel preemptible?
No.

> can it be used for real time process control?
You can use a non-preemptible kernal for Real Time Processes. You just need to keep the running software to a bare minimum. I did this Win NT 4.

Re:MS Windows?

[TummyX]

AFAIK, it is not. I know that in Windows minimum latency is about 10ms, same as in default Linux kernel.


Uh. NT has a preemptible kernel. Most /. nazis would dispute that cause they won't know anything about NT.

Re:MS Windows?

[WildThing]

First let me say that is has been awhile since I was writing Winblows code; however....

M$ claims Winblows to be a pre-emptive kernel. This is true and not true. It is preemtible because any task can make a 'critical system call' and get control of the system. Gee, that's great right?!? NOPE! Take a winmodem for example, try moving your mouse while it dials the phone or finalizes the handshake process. The mouse pointer will freeze on the screen! It's because the device driver make a crtical call and until the userspace program releases that lock it has total control of the system. M$'s own programmers reference material tells programers about this and recommends that programs exit that mode as quickly as possible. That makes it NOT a premptible kernel.

Re:MS Windows?

[Zwane]

NT kernal is; at pretty much any time you can give the three finger salute and get the 'stuff to do' dialog. When you see a blue screen, that's NT having caught an exception that worries it enough to *voluntarily* halt itself, because it can no longer guarentee that it's working with valid data. It also spews out a large amount of debug data. Most such bluescreen occur due to hardware drivers. This is similar to a UNIX kernal spewing a bunch of kernal panic messages and the like to the syscon, then halting. Very rarely will you see an NT derived kernal simply 'freeze.'

Exceptions don't have anything to do with a kernel being preemptive. I would hope any modern OS had exception handling. And the "3 finger salute" is actually handled by an interrupt handler, and those can preempt any kernel code regardless of wether you have a preemptive kernel or not (Hence the reason why DOS can use ctrl-alt-del too)

Re:MS Windows?

[SuiteSisterMary]

I know C-T-L is an interrupt. But DOS and Win9x will lock, often, to the point of not responding to it. I've never seen NT do so, however. As to the bluescreen stuff, the point I'm trying to make is that a BSOD isn't a 'crash,' it's a very obtuse error screen. It's a state that NT throws itself into voluntarily when something goes very very wrong, and that something is almost always a hardware driver. Where screens used to have a "C:\>" burned into the upper left corner, now it's more like "ati.sys" ;-)

too many interviews

[Anonymous Coward]

Its amazing to me how many interviews this
guy manages to pull for himself. Probably
much higher rate than Linus himself. It's what
i would call Open Source PR. I wouldnt care less
if it that patch was in there; but please cut
the open source PR^FUD. Andrew Mortons patch
is more than enough.

Preemptive Kernel?

[Anonymous Coward]

Please forgive the ignorance, but what is a preemptive kernel as opposed to a normal kernel?

Re:Preemptive Kernel?

[Nothinman]

A preemptive kernel is allowed to be stopped for another higher priority task, in the default kernel only userspace apps can be preempted. What this means is if an app is reading from the disk (the syscall for block I/O is run in kernel mode on the app's behalf) and the disk read is taking a long time (many milliseconds) that kernel path can be paused to allow other processes to run while the disk finishes it's read.

Re:Preemptive Kernel?

[maddman75]

As far as I know (and I'm no kernel expert) the normal kernel works on whatever process it is doing until its finished. This can cause delay when you are doing several processor intensive tasks at once.

A preemptive kernel allows a new process to interrupt the kernel, even if its been doing something important. Increase responsiveness.

Re:Preemptive Kernel?

[paulbd]

thats complete nonsense. the behaviour you're describing is what any multitasking kernel does. it has nothing to do with the preemptive kernel patch, which ensures that when a task becomes runnable (for example, due to a device interrupt that tells the CPU that a condition the task was waiting for is met), it can start running ASAP, rather than waiting till the next regular re-scheduling point. ASAP here means within at most a millisecond or two. There will be always be places where the kernel has to prevent this from happening; the goal is to minimize (and document) those places.

Re:Preemptive Kernel?

[Nothinman]

thats complete nonsense. the behaviour you're describing is what any multitasking kernel does.

Not entirely, without this patch any kernel code path executes until it returns or manually calls schedule() and if you have a UP box all other processes don't get to run. But, my example of a hard disk read may have been poorly chosen.

the preemptive kernel patch, which ensures that when a task becomes runnable (for example, due to a device interrupt that tells the CPU that a condition the task was waiting for is met), it can start running ASAP, rather than waiting till the next regular re-scheduling point.

The preempt patch allows processes stuck in kernel paths to be put to sleep so others can run, I don't believe it gives those processes any priority boost because they were in a kernel path but I could be wrong about that.

It sounds like you're describing a real-time scheduler more than a preemptive kernel.

Re:Preemptive Kernel?

[Anonymous Coward]

> As far as I know (and I'm no kernel expert) the
> normal kernel works on whatever process it is
> doing until its finished. This can cause delay
> when you are doing several processor intensive
> tasks at once.

I'm no kernel expert either, but at least I teach an OS course which use Linux, and read part of the kernel source, so I think I'm qualified to correct this.

Most things in Linux (and Unix in general) are done in user land. If you have a CPU intensive process, all of it should be done in user mode, so it should be completely preemptible using a stock Linux kernel. If all your jobs are like this, you don't need a preemptible kernel.

And an earlier post talks about reading a disk which takes some milliseconds. This does not introduce the need for a preemptible kernel, either. Disk reads are performed using DMA (direct memory access), i.e., the disk controller will write directly to the memory, and the CPU need not run any code for it. When a disk transfer is taking place, the process should be in the waiting state, and takes no CPU time at all.

So what makes preemptible kernel desirable? Sometimes the kernel needs non-trivial amount of time to deal with a user request or interrupt. For example, if you ask the kernel to check 200 file descriptors for readiness (using select), the kernel will have no choice but to go through each file descriptor, dereference into its device driver, and call the handler function to check whether the descriptor currently have input. That will take some thousands of clock cycles, in the order of 1ms.

Or if a device (like the old parallel port, or many high bandwidth network card) requires input and output to be done by polling, i.e., continuously reading the port until a special value is read (rather than the device sends the CPU an interrupt to wake it up). The kernel will spend time waiting there without giving up control of the CPU.

Theoretically, all these places should be fixed in the kernel itself: the kernel should not block for a long time, and when it really needs to do so, it should do it in small chunks. If I understand it correctly, that's why Linus don't like the patch so far: let's fix the cause, not the symptoms.

But as is illustrated in the select call, some of the times it is very difficult to fix the cause, because the kernel API is written in a way that at times programs can make heavy requirements on the kernel, and because at some workloads it is better not to implement everything that will make a device driver relinquish control as quickly as possible. That's the rationale for a preemptive kernel, especially because most of the work are already done because of the SMP requirements.

Use the mirror

[RadioheadKid]

US mirror of kernel.org

Resumable Pre-emtable OS calls

[leighklotz]

The ITS operating system (the world's second timesharing system, and the system for which RMS and others first developed EMACS) had a concept of state checkpoints in OS calls, called PCLSRing.

Alan Bawden wrote a paper on it, and it's quite a good read. His web site has a compressed .gz version, but I found an HTML version of the HTML PCLSR Paper and I quote from its abstract here:

Under any timesharing operating system there will be occasions when a process must access the state of another process. A process may need to start, stop, debug, load, dump, create, or destroy another process. There is also one occasion when a process must access its own state: an interrupt handler needs access to the state of the running process prior to the arrival of the interrupt, so that the process may continue after the interrupt has been dealt with.

"PCLSRing" is a mechanism the ITS operating system uses to enforce a kind of modularity when a process must access the state of another process. The modularity principle is very simple: no process ever catches another process (including itself) in the act of executing a system call. System calls thus behave as if they were directly implemented in hardware. A process can no more catch another process in the middle of deleting a file than it can catch another process in the middle of a multiply instruction.

There was also a way to put the system into a PCLSR test mode that exercised all these control points within the system calls, to help debug them. See SYSDOC TEST documentation extracted from the now decomissioned AI PDP-10 that originally served it up as ftp://ftp.ai.mit.edu/pub/alan/its/sysdoc.tgz (yes, ITS was on the Arpanet and the Internet and ran TCP/IP as well).

Re:Resumable Pre-emtable OS calls

[Zooks!]

This idea is nice but old. After reading most of the PCLSR paper at the URL you mentioned its clear to me that most of the problems addressed by PCLSR are addressed today by multithreaded kernels and fine grained locking.

I can see why PCLSR might have been necessary and feasible on a small PDP-6 operating system but it would be both a pain to implement and unecessary for today's OSes and machines.

Re:Resumable Pre-emtable OS calls

[Surak]

The ITS operating system (the world's second timesharing system, and the system for which RMS and others first developed EMACS)

Minor nitpick. RMS first developed Emacs on Unix. It was modeled after TECO, which and others hacked together on ITS.

Re:Resumable Pre-emtable OS calls

[viscous]

I was one of the very first users of ITS Emacs (in 1976 I think it was). They guy who put it together and maintained it sure looked and acted like RMS, so I'm pretty sure you are wrong!

Re:Resumable Pre-emtable OS calls

[viscous]

Actually, the idea isn't all that outdated. Other people periodically re-invent it. See, for example, this paper about the kernel API for the Fluke kernel. These guys basically reinvented PCLSRing - motivated by much the same reasoning as the ITS developers.

I suppose I should confess that I'm the author of that PCLSR paper.

Hello moron

[NDPTAL85]

Not only are binaries compatable between the various BSD's, but they are compatable to Linux, several corporate Unix's and Mac OS X/Darwin as well.

It would be of _great_ assistance to yourself it you educated yourself before opening your mouth.

Re:Hello moron

[Anonymous Coward]

no, they're not neither

Re:Hello moron

[NDPTAL85]

Yes they are

Re:Hello moron

[Anonymous Coward]

Lets see you run an app compiled for OpenBSD/386 on your FreeBSD/386 box.

You might want to take your own advice.

Re:Hello moron

[Anonymous Coward]

Wrong

A Red letter Friday!!!!

[Anonymous Coward]

Not only do we hear about yet another Linux distro (for the week) but yet another Linux Kernel Version!

Is n't He

[Anonymous Coward]

Is n't he the same bastard that had trouble with the linux licenses or that a different person

Re:So you admit Linus is a racist?

[jaredmcook]

Yes, Brazill is a third world country. A large percentage of their population does not have electricity or running water. Millions of Brazillians (i.e. black, ex-slave) live in extreme poverty. Only the rich in Brazill (like other all other third would countries) has a decent standard of living. You need to get your facts straight. Jared

Solaris

[Anonymous Coward]

I know Solaris preempts - 1ms latency in my tests - even on a low-end desktop sparc box.

Pre-Emptable Kernel & MicroKernels

[sl3xd]

I'm ignorant of this fact... but it almost sounds to me like having a pre-emptible kernel is one of the 'featues' of a MicroKernel --

Although I'm fully aware of the fact Linux is NOT a MicroKernel, what significant differences exist between Pre-Emptible Linux and a MicroKernel Linux?

(Aside from the fact that all of the Linux kernel, drivers, etc. is in 'kernel' mode, and a MicroKernel has only the message-passing and task-scheduling in 'kernel' mode, and everything else (drivers, etc.) run in 'user' mode.)

Re:Pre-Emptable Kernel & MicroKernels

[be-fan]

(Aside from the fact that all of the Linux kernel, drivers, etc. is in 'kernel' mode, and a MicroKernel has only the message-passing and task-scheduling in 'kernel' mode, and everything else (drivers, etc.) run in 'user' mode.)
>>>>>>>>>>.
That's basically everything that distinguishes a microkernel from a monolithic kernel :) The preemptive bit is orthagonal to micro/macro kernel. There are non-preemptible microkernels (like MINIX, I think) and preemptible monolithic kernels (Solaris).

so true

[pohl]

You have hit the nail on the head. I wish I could mod it up. I'm amazed that people "don't have time" to practice their chosen craft.

Alan Cox says 2.6 won't have compiled-in modules

[cpeterso]

Linux is heading even further away from the compiled-in drivers and to a more modular kernel. According to LWN [lwn.net], Alan Cox said that Linux 2.6 will not have compiled-in drivers. "Compiled-in" drivers will just be modules that are written to the "initramfs", a sort of software boot disk compiled into the kernel.


From: Alan Cox
To: babydr@baby-dragons.com (Mr. James W. Laferriere)
Subject: Re: ISA hardware discovery -- the elegant solution
Date: Mon, 14 Jan 2002 18:08:32 +0000 (GMT)
Cc: alan@lxorguk.ukuu.org.uk (Alan Cox), esr@thyrsus.com, cate@debian.org (Giacomo Catenazzi),
linux-kernel@vger.kernel.org (Linux Kernel List)
> Hello All , And what mechanism is going to be used for an -all-
> compiled in kernel ? Everyone and there brother is so enamoured
> of Modules . Not everyone uses nor will use modules .

For 2.5 if things go to plan there will be no such thing as a "compiled in"
driver. They simply are not needed with initramfs holding what were once the
"compiled in" modules.

We have already won.

[Anonymous Coward]

You just don't seem to see the forest from the trees. I'm working for a major IT company and I get to see the rapid growth of Linux usage every day. More and more customers are *demanding* Linux solutions.

Sleep 3.5 hours/day

[Anonymous Coward]

Well, you gotta restrict your sleeping time to, say 3.5 hours/day. That's how I handle the problem. I got a regular job and it seems 3.5 hours sleep is enough for me. I can still manage my job well. On the weekends I use to sleep like 12 hours/night.

Crash-course in preemtivity

[slashdot.org]

For those that don't really understand the importance of preemptive multitasking (and from reading some comments, there are a few of you out there :-O), let's explain this through an example.

Consider application (a) that wants to read 128MB of data from the disk and application (b) that wants to read 1KB of data.

Let's say that the disk transfers @ 1MB/sec and let's assume application (a) issues the read 1 second before application (b) is started.

The sequence of calls for each app will look something like this:
1) program calls read
2) read is handled by the top level file system and is handed down to the proper file system
3) the file system calls the block device
4) the block device driver breaks up the request into the maximum block size the device can handle per request (for example 256 sectors for IDE)
5) for every request, the block device sends request to physical drive
6) drive transfers data to host
7) drive indicates 'done'
8) goto 5 until done
9) file system returns
10) read returns

It's important to know that there may be limitations on the number of requests any stage can handle simultaneously. For example, an IDE drive can only handle one request at a time. Some Operating Systems however introduce even tighter restrictions, because for example the block device driver was written, assuming only one request at a time would be allowed.

Take for example an OS where the kernel assures that no more that one request is pending between step 3) and 9).

This would have the following effect on our apps: app (a) is allowed to call step 4) because it is the first request. One second later, app (b) arrives at step 3) and is blocked. It is not allowed to enter step 4) until app (a) is done and passes step 9). Effectively this means that app (b) has to wait for 127 seconds before it gets access to step 4).

Now consider an OS where the file system and drivers can handle multiple requests. It still has to assure that the physical drive receives only one request though, so it permits only access of one app between step 5) and 7). App (a) arrives at step 5) first, so it gets to start sending requests to the drive. One second later, app (b) arrives at step 5) and has to wait for app (a) to finish it's current request to the drive. As soon as this request is done though, app (a) gets to step 8). Now we have both app (a) and app (b) wanting to have access to step 5). Depending on the scheduler either one will be granted access. There's a good change that app (b) will get access, and thus it only had to wait the time it took for app (a) to finish it's outstanding request, which takes max 1/8th of a second (256 sectors = 128KB) in this example.

Btw. you may notice though that there's more likely to be an (expensive) seek introduced by allowing app (b) to interrupt the transfer of app (a)

You can see how moving the 'lock' deeper in to the OS improves responsiveness. I'm not going to start a flame war about which OS is better, all I will say is that MY OS locks steps 5) through 7) :-)

Re:Crash-course in preemtivity

[nevets]

This is fine except the way the linux kernel works (as well as most others). When app(a) calls the IDE to get the data, it goes to sleep (calls schedule) and won't wake up until it gets the received data. When schedule is called, it can go back to user land. app(b) will now do its call and it will be queued up and blocked until app(a) gets its data. Then app(b) will get a turn and this goes on and app(b) will not be affected by the big read of app(a). Your situation would happen if the kernel didn't call schedule while waiting for data. Thus that would really hurt the performance of the kernel.

The real problem is if you have a periodic program that needs to make a deadline. What a non preemptive kernel does is place in too many varients. You can't guarentee that the program will make its deadlines when you have other lower priority tasks running.

Say you have a period of 10 ms and your task needs 3 ms to do its job, and it must be done within the first 5 ms. So your app starts and gets priority, and you do what needs to be done in the first 3 ms and easily makes your 5 ms deadline. Now you may run other task for the next 7 ms. Come the next period, you start your 3 ms task and repeat.

But lets say you have a system call that takes 4 ms, and one of your non priority tasks calls it at the 9 ms time. With out being able to preempt it, your priority task will start at the 3 ms time and it won't finish until the 6 ms time and thus you missed your deadline.

The reason you have preemtive kernels is so that priority tasks are not affected by lower prioriy tasks. Not the reason you gave above.

Re:Crash-course in preemtivity

[slashdot.org]

The reason you have preemtive kernels is so that priority tasks are not affected by lower prioriy tasks. Not the reason you gave above.

Really? So what about tasks that are of equal priority?

I maybe should have explicitly said so, but I assumed the threads (apps) run at the same priority. This way it was a bit easier to explain without going into details regarding scheduling.

Preemtive kernels also improve responsiveness between equal priority tasks.

Re:Crash-course in preemtivity

[Anonymous Coward]

>>>> When schedule is called, it can go back to user land. app(b) will now do its call and it will be queued up and blocked until app(a) gets its data. Then app(b) will get a turn and this goes on and app(b) will not be affected by the big read of app(a)

So which one is it?
- app(b) will be blocked
- app(b) is not affected

The flaw is in 'Then app(b) will get a turn'. How can it if the device is locked by app(a).

Re:Crash-course in preemtivity

[nevets]

Really? So what about tasks that are of equal priority?

So what's your point, a preemptive kernel will not preempt a task for a task with the same priority. These would be treated the same as if the kernel was not preemptive. Actually, this is not quite true. The one task would be preempted when its quanta runs out. And with a preemptive kernel this would even happen if the one task was in a system call. But that's the only difference between two tasks of equal priority running on a preemptive kernel, and two tasks of same priority running on a non preemptive kernel.

Preemtive kernels also improve responsiveness between equal priority tasks

I wouldn't say responsiveness but maybe something like smoother or more consistent. But I'm not sure you'll improve responsiveness on a kernel that's preemptive. If task (a) is waiting for an interrupt and it happens while task(b) of same priority is running. Task (a) is placed on the run queue, but needs to wait for task (b) to finish its quanta or go to sleep. This doesn't even matter if task (b) is in userland. Task (a) will still wait for task (b), unless task(a) is of higher priority. So this is no more responsive than a non preemptive kernel.

Re:Crash-course in preemtivity

[nevets]

The flaw is in 'Then app(b) will get a turn'. How can it if the device is locked by app(a).

This comment has nothing to do with a kernel being preemptive or not. This comment deals with resource management.

app(a) doesn't "Lock" the device for the entire read. The read calls for one block of data and then yes the device is blocked. When the data is returned, the next request is handled which may also be the request from app(b). The device is only blocked when it is receiving one block of data of a fixed size. The system call read will only block the device while grabbing one block of data, and that will allow other apps to be interleaved with the device.

Re:Crash-course in preemtivity

[slashdot.org]

I think you misunderstand my point entirely. What I was trying to show is how the place of locks effect 'preemtiveness'. (And this is exactly what Robert is doing, by using the lower locks that now exist for SMP kernels).

It's all fine and dandy to schedule a high-prio task, but if your kernel is locked by a lo-prio task, it's useless. In other words, in order to make a kernel preemptive, you need to move locks down to the lowest possible place.

For example, the Linux IDE driver is locked by the kernel for the duration of an entire request, no matter how big. Only one request is handed to the driver at a time. So if an app issues a very large read request and finally gets a hold of the lock, the IDE driver can not be preempted to handle an other request until the entire request is fullfilled. Sure, it will call schedule so other tasks can run, as long as they don't require the IDE driver. If they do, they have to wait until the entire read request is done. The point is that by putting the lock lower, you make it possible for other tasks to obtain the lock, allowing preempting.

a preemptive kernel will not preempt a task for a task with the same priority

Maybe Roberts patches don't, but I know my OS does.

Re:Crash-course in preemtivity

[slashdot.org]

Yep, major fuck up: the current Linux IDE driver doesn't get locked for entire requests... Bad example, sorry.

The point still stands though for basically any resource that has a long (time-wise) lock on it.

Re:Crash-course in preemtivity

[nevets]

Maybe Roberts patches don't, but I know my OS does.

Really? why. Doesn't this mean higher cost of context switching. And if one preempts the other, doesn't this cause a back and forth infinit loop? What OS is yours?

I did continue to say that one would preempt the other, but only when its time ran out and not by the other one just becoming active. Now I would be curious to know of another algorithm and how it would be better.

Re:Crash-course in preemtivity

[slashdot.org]

The OS is based on the On-Time RT-Kernel32 scheduler. Most of the other stuff is written by me (like drivers/fs/tcpip etc).

Although you can configure On-Time's scheduler in many ways, we use it as a priority based, preemtive scheduler. This means that the highest priority task always runs if it wants to. When there are multiple tasks of the same priority that want to run, they are scheduled using round robin. This is largely based on the timer interrupt (which we run at 1000 Hz) which means that they preempt one an other based on time. Essentially the task that has run for the least amount of time runs.

So a task switch occurs either 'implied' by the task blocking on a semaphore/lock or 'preemptive' by the scheduler. The scheduler is called after processing each hardware interrupt (such as the timer) to assure the right task runs.

As you can see from On-Time's website, context switching is fast, even if we switched every timer interrupt, on our PIII-500 we would spend about 200 microseconds per second on the context switching.

Under the hood, it's an entire different approach than Linux and I get confused sometimes about the way things work in Linux. That and I don't really stay up-to-date. I'm happy to see Roberts work, because it addressed the main concern I've had with Linux in the past.

Anyways, I hope this makes sense...

Re:Crash-course in preemtivity

[nevets]

The point still stands though for basically any resource that has a long (time-wise) lock on it.

This is a resource management issue and not a preemptive issue. Yes if you can make the locks on a resource shorter, the better. But most of the resources in the kernel are not protected by non preemption, and those that are are mainly just the global data. Now Linux is already made to work on a SMP, and with SMP processors you don't have the protection of non-preemption on your data as you did on UP. So spin_locks were made to do a busy loop on processors that were going to modify data that was currently being worked on by another processor. For UP these smp_locks are just nulled out (do { } while(0)). So what Robert (and others) have done is use the smp_locks as the places to protect data in a preemptive kernel. Most notably link lists.

What my point is, is that its not resources that can be accessed quicker, but that higher priority tasks can be accessed quicker. The locks on resources are not shortened by the preemptive kernel, but higher priority tasks are not postponed just because a lower priority task is in kernel mode.

Although a naive kernel could use non-preemption to protect resources, but that would be just silly (it may end up being windows 95;)

Re:Crash-course in preemtivity

[nevets]

Thanks, I now understand where you are coming from. Sorry I was mainly debating the way things work in Linux. I don't use Robert's kernel, since I use my companies. We have our own version of the Linux kernel that handles preemption.

You can get it at my company's site

Re:Crash-course in preemtivity

[slashdot.org]

Yeah, fun discussion, nonetheless. I have to admit that I'm a little rusty on the Linux side. I wrote a couple of device drivers (for MPEG hardware decoders and a block dev for RIO MP3 player. Non of em actually where released for a variety of reasons...) but that's quite a while ago.

I probably should have stayed out of it, because the topic really was the Robert Love's Linux patches...

I've read up on some of the documentation on your website and it certainly cleared things up a lot. I guess I understand where you are coming from as well. :o))))

Anyways, I really go to get back to work and stop posting on /. all the time.

Cheers.

Struggle of Raster and Mandrake. A lover's Story

[Anonymous Coward]

Chapter 1: Bowel Movement: Part 1 Rectum
It has been known for sometime that Raster and Mandrake had it "in" for each other. Long before the days of E Mandrake had been smoking dope and droping a few here and there.
Raster's maddness began in the bowels of Austrila. Already Diagoned a sczho, the drugs took effect. Before long Raster learn to code. After several stabs and pokes with a pointy stick. Raster known by some as the inane blabling drunk spaindard. Also Known as Betty, but that is another story. .......

An earlier interview

[OneFix]

With Robert Love at KernelTrap

And another one with John Levon (OProfile) where he discusses the patch. (at the end of the interview)

For me, these 2 interviews offer more insight than the interview in the article. Especially the comment from Levon that this patch is really just a "band-aid"...and that the real problem lies in fixing the kernel.

So?

[UnknownSoldier]

> Like all true geeks, Love doesn't forget to include in his comments that, despite being a computer nerd, he does, in fact, have a girlfriend.

WhoTF cares if he has one or not? Are (other) geeks jealous because he has one, and they don't?

I care about the quality of his code, not about his love life.

Re:So?

[ejasons]

WhoTF cares if he has one or not? Are (other) geeks jealous because he has one, and they don't?

I care about the quality of his code, not about his love life.

I think that we need a moderating category for "No sense of humor"...

misleading article title

[potaz]

First time I read the title, I thought it said "Robot Love, Preemptible Kernel Maintainer Interviewed", which, understandably, aroused my interest.

Missed it out of the gate ....

[Zero__Kelvin]

"RL: You have been becoming well known as the maintainer of a "preemptible kernel patch" for the Linux kernel. How did that situation come about?"

"Love: Well, RL, you start with a group of people who are more than willing to brag that they know who maintains the 'preemptible kernel patch' but feel no need whatever to know what the hell that even is ... "

Any Relation... (j/k)

[BadlandZ]

to Ransom Love?

sorry, some idiot was bound to ask, so I figured I would be today's idiot!

Actually, no

[leighklotz]

Actually, no. I worked at the MIT AI lab during this period, and was in charge of EMACS on the PDP-10 series (for ITS and TOPS-20) after RMS moved on to start the GNU project.

TECO was developed for the PDP series. The PDP-6 and 10 were the systems that ran ITS. EMACS was a package of TECO macros; there were others, RMACS and TMACS for example. RMS and others maintained EMACS, and it became the lead, and eventually it took on a life of its own.

After RMS stopped working on PDP-10 software and was fully engaged on the GNU project, I was in charge of PDP-10 EMACS. There were a variety of other EMACS clones on Unix, small and large, but they all had various extensibility problems (i.e., had to be recompiled) or licensing problems (i.e., you had to buy them). RMS started with one or the other of them and fixed both the licensing and extensibility problems, and everybody knows it all from that point.

RMS did not write TECO, and EMACS was not first on Unix.

Re:Question about racism

[Zwane]

Besides that, it an Indian or African developer wanted to contribute, they could do so. That's the beauty of it all. Oh, and how many of them do? Well, that's probably why Linus hasn't appointed them to be maintainers.

You really don't keep track of anything beyond your state lines do you? There are quite a number of Indian _and_ African developers contributing (Me being one). Perhaps if you drove out further than your local grocery store you might actually meet interesting people out there.

He Isn't bragging ...

[guybarr]

He just doesn't want to be killed in the rush.

;)

Re:I will troll Slashdot from now on

[Anonymous Coward]

You find something wrong with something in your life, you discuss and then bitch about it for a little while, you find that not everyone else agrees with you and/or the people who own and run the system you're using for free don't want to change anything, and what do you decide to do?

Continue in an adult manner to keep a dialog open in the permitted channels and convince the people around you of your opinion?

Leave and go somewhere else where you can form a community with the rules you want, helping create something useful (in your opinion) for the world?

No, you choose to attempt to destroy this community. The one the rest of us are happy with and trying to enjoy.

Thanks a bunch-a-roonie!

And *then* you're all suprised and hostile when you discover that you've made some enemies?

Do you do this with everything else you disagree with? Just how the hell do you get along in real life, where the democratic process and living with the decisions made by the millions of other people around you don't always agree with your little view of the world?

Morons. Good riddance.

Ever since this blew up, I've been meta-moderating like mad. I hope lots of other people get involved and help prevent these childish idiots from getting the upper hand.

(I read through all the stories at Kuro-Shin yesterday. What a load of immature goofs and paranoid conspiracy theories! Ahhahahhhahhaha!)

Re:I will troll Slashdot from now on

[CKW]

A temper tantrum.

That's what their behaviour reminds me of.