Slashdot Mirror

← Back to Stories (view on slashdot.org)

KSE Progress On FreeBSD SMP Environment

Posted by Hemos on from the threads-for-the-world dept.

Dan writes "This is a significant milestone to be shared with everyone! Khairil Yusof reports that libkse is now running quite well on his FreeBSD 5.1+ current based SMP system. He has tested a bunch of apps on his system, taking the approach of enabling kse one app at a time. He reports a current uptime of 23hrs with these apps running with libkse.so.1, and basically a usable Gnome 2.2 desktop environment. He says that with recent updates, you can now see the threads with top(8). Kernel Scheduler Entities (KSE), is a kernel-supported threading system similar in design to Scheduler Activations [Anderson, et. al.]. It strikes a balance between user-level (1:N) and kernel-level (1:1) threading models, giving most of the advantages of both, and few of the disadvantages of either."

26 of 60 comments (clear)

  1. More Power to you by Anonymous Coward · · Score: 4, Insightful

    This is great, and advances all of the *BSDs if it works out.

  2. Testsuit and benchmark by rapiere · · Score: 5, Interesting

    KSE seems really an interesting implementation in FreeBSD as well as the process to integrate it in the kernel/userland.

    LIBMAP permits to cleanly test it on your system (5.1-RELEASE or -CURRENT) one application at a time (you redefine libraries linking per application). I've tried it with mozilla 1.4b for more than a week now without any trouble.

    However I wonder about developper's testsuit and benchmarks in order to give some clues to others about how great KSE is.

  3. Threads in top by cant_get_a_good_nick · · Score: 3, Insightful

    He says that with recent updates, you can now see the threads with top(8)

    I hope this is something you can turn off and on. One of my problems with the Linux threading model is that you see every thread as a proc, which makes it harder to administer a box. top output becomes overwhelming.

    1. Re:Threads in top by Hard_Code · · Score: 3, Informative

      Well something must have changed in recent psutils, because now 'ps' correctly collapses threads within processes when they are not active (I assume). When my site gets a hit, I see the threads expand, and once the request is served, they are collapsed back under a single process line.

      --

      It's 10 PM. Do you know if you're un-American?
    2. Re:Threads in top by shlong · · Score: 3, Insightful

      Well something must have changed in recent psutils, because now 'ps' correctly collapses threads within processes when they are not active (I assume). When my site gets a hit, I see the threads expand, and once the request is served, they are collapsed back under a single process line.

      The support that I added was to make the kernel return info on all threads unless it is told not to. I then changed ps(1) to tell the kernel to not return all threads unless the user specifies the -H option. I didn't modify top as it is vendor code and I haven't gotten around to tackling it yet.

      As for your observation, remember that in KSE/SA the kernel only knows about the threads that have blocked inside of it and the per-CPU scheduler activation (i.e. execution context) that it supplies to the userland scheduler (UTS). It has no knowledge of threads that the UTS creates that stay in userland. Sooo.... it is likely that you saw a thread that blocked in the kernel waiting for the socket, and then left the kernel once the socket woke up.

      Don't forget that there is also the libthr 1:1 library. That creates a kernel thread for every user thread. Since the kernel knows about every thread, ps and top will show each thread also.

      I'm incredibly pleased with all of this. I finally got around to reading the SA paper last month, and it's very cool to see KSE start to realize its potential. 5.2 and 5.3 will be very exciting releases.

      --
      Cat, the other, tastier white meat.
    3. Re:Threads in top by geniusj · · Score: 1

      I haven't updated 5.x in a month or so, however, this may not be how the same as what you're thinking.

      Take MacOS X's top output for example:
      PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
      14055 mozilla-bi 0.8% 7:18.05 12 185 803 43.3M 30.3M 56.2M 278M

      As you can see it's just a field in top that shows the # of threads under that process. I'd imagine that would be how it is done in FreeBSD as well. I could be wrong though, can anyone confirm?

      Cheers,
      -JD-

    4. Re:Threads in top by geniusj · · Score: 1

      Never mind, I just looked at the email. Would be nice if they did it like Darwin does though..

  4. Re:will this be eventually ported to the other BSD by merdark · · Score: 3, Informative

    Maybe.

    I know OpenBSD has a group in a totally different project working on SMP for OpenBSD. The different BSDs have fairly different kernel internals as far as I know. It certainly wouldn't be a cut and paste job.

  5. Re:bsd problems by Ricin · · Score: 1

    Skipping over the vague anecdote and pleas for "fanatics" and "addicts" to start flaming (back), here's the obvious answer:

    IT WORKS FOR ME

  6. Re:Elegy for *BSD by tarius8105 · · Score: 1

    BSD is not dying. Go look at netcraft, BSD/OS and FreeBSD are the only operating systems listed with the longest uptime. Go look at what Apache.org websites are running all FreeBSD except for two or three servers are Linux or Mac OS X (which consists like 90% FreeBSD servers).

  7. Re:bsd problems by Piquan · · Score: 4, Informative

    It would have been nice for you to mention that you're using a Mac, instead of obtusely implying it (8600, BBEdit Lite).

    The section of OSX that copying a file involves is not BSD-related, as I understand it. It's Mach. (People who know more about OSX than I do, feel free to pipe up.) The kernel of OSX is not very BSD-based, and most of what you're talking about (HDD access, scheduling) takes place in the kernel. (To be fair, the scheduler that FreeBSD uses was adapted from Mach.)

    The 8600 (which was discontinued over a half decade ago, by the way) is not even compatible with OSX. The minimum requirement for OSX is the Beige Power Mac G3. (See Apple's requirements page.) So, I'm guessing you put in a G3 daughterboard. (Find out about the 8600 and available daughterboards at lowendmac.com.) But OSX doesn't support processor upgrade cards. (First paragraph of the requirements page.) Maybe you didn't put in an upgrade card, and are using the mach_kernel for the 604 from the Darwin project like this guy did. Either way, you're still not using a supported system.

    So, you're using unsupported hardware, and a BSD-related OS in an operation that's not related to the BSD bits, and using this to say that BSD sucks. Hmmm.

    I'm surprised an 8600 w/ daughterboard (which one, by the way? The 233MHz with 512k of cache?) works at all with OSX. As for why it's slow for you, I can only hypothesize, since I'm not a Mac guy. (That's right, anybody can find this stuff out with a quick google search or two!) The G3 upgrade card is going to need an L2 cache enabler. Do you have one installed? Okay, how about this idea. I'm guessing that your old, discontinued, unsupported hardware uses a bus controller (or other critical chip) that is not being programmed optimally by OSX. Why? Because the programmer writing that code knew that the chip wasn't going to be supported!

    My advice is to stick to using 9.1 on your 8600. And don't generalize about BSD from your situation; it's an extreme.

  8. Re:will this be eventually ported to the other BSD by flynn_nrg · · Score: 2, Informative

    I'm not sure about OpenBSD, but there's not much shared code in those areas between NetBSD and FreeBSD. On NetBSD, Bill Sommerfeld did most of the i386 SMP code, and Nathan Williams is the guy who's working on scheduler activations. SA is a M:N implementation, not much different from FreeBSD's KSE, which was developed mainly by Julian Elisher. SMPng is the work of many people, specially John Baldwin. Note that KSE is only really finished on i386, Alpha and Sparc64 are still missing some bits.

    SA is not as advanced as KSE, though, but it looks very promising. AFAIK, kernel assisted threading and SMP have never been priorities for the OpenBSD people, ergo, little work has been done in those areas.

  9. Comparison by bobbozzo · · Score: 2, Interesting

    Would someone care to explain the differences/similarities between this and the new Linux (2.5) threads implementation?

    TIA

    --
    Nothing to see here; Move along.
    1. Re:Comparison by Anonymous Coward · · Score: 4, Informative

      Linux (2.5) implements a 1:1 kernel threads model. KSE is a much more complex N:M threading model. FreeBSD also has brand new 1:1 kernel threads (similar to what Linux and Solaris have). You can choose (per application!!!) which threading library you like to use :-)

    2. Re:Comparison by stripes · · Score: 1
      Would someone care to explain the differences/similarities between this and the new Linux (2.5) threads implementation?

      Linux has one kernel thread per thread-the-user-program-thinks-it-has ("apparent threads"). The old pthreads lib had one kernel level thread per process no matter how many apparent threads there are. The new KSE has some threads per process, but that can be far fewer then the number of apparent threads.

      Having one kernel thread per apparent thread means you don't need non-blocking kernel interfaces for everything and you can have one thread running per CPU (at least if locking doesn't get in the way). It also means there is a lot more kernel overhead (non swappable structores and the like), and that the kernel has to make all the scheduling choices (does hitting a blocking lock spin, or switch contexts? if it switches, does it make sure the lock holder is running? or even switch to a thread in the same processs?)

      Having one kernel thread per process has lower kernel overhead and can allow some intresting thread scheduling schemes. It also has many disadvantages: you need a non-blocking API for every single kernel resource, or you end up blocking potentally thousands of threads on the request of a single thread. If you only have one process you care about being fast addinf CPUs will never help because no matter how many apparent threads you have there is only one kernel thread per process.

      The new KSE model has "some number" of kernel threads per apparent thread. If it is like scheduler activations it is actually more or less one kernel thread per CPU or the number of apparent threads, whichever is smaller. So kernel overhead is reduced (it only tracks say 4 threads for a process with 100 apparent threads). It doesn't need a blocking version of evey API because when an apparent thread blocks the KSE does something simalar to sending the process a signal and the KSE that was running the blocked thread now runs the user-level scheduler which picks an apparent thread that was blocked and switches to it. That also lets one have all the advantages of user level thread schedulers. About the only downside is you can't track the apparent threads that are blocked from outside the process.

      There is also a 1:1 ("Linux like") threading model using the same kernel part of the KSE stuff.

    3. Re:Comparison by bobbozzo · · Score: 1

      Thank you!

      --
      Nothing to see here; Move along.
  10. Hard to do SMP securely by bee · · Score: 3, Informative

    From what I understand, SMP opens all kinds of possible race conditions and other such nasty potential security holes, so OpenBSD doesn't want to implement something like this until they can be sure they can do it securely. Which may not be for a while, hard to tell.

    --
    At least mafia-owned pizzarias make excellent pizza. Compare to Bill Gates.
    1. Re:Hard to do SMP securely by merdark · · Score: 1

      Yep. That's true. I don't imagine we'll see SMP in a production OpenBSD anytime soon. In fact, the spinlocks project, which was working on SMP for OpenBSD, seems dead. Oh well.

  11. Re:bsd problems by Dahan · · Score: 1
    It would have been nice for you to mention that you're using a Mac, instead of obtusely implying it

    My advice is to stick to using 9.1 on your 8600. And don't generalize about BSD from your situation; it's an extreme.

    It's a search-and-replace of an ancient troll--the original was about classic MacOS; it's been reposted as about MacOS X, and now as about BSD.

    In general, keep in mind that stories in the BSD and Apple sections will always have these stock trolls (e.g., "BSD is dying," "Elegy for BSD," "Developer Laments," "Dear Apple," "Dear Fr. O'Day," etc...).

    with much gayness,

    Dahan, C.S.B.

  12. Re:Well.... by Anonymous Coward · · Score: 1, Interesting

    That said, I've not had much luck with the new posix threading libraries. I was trying to build an old revision of the Wolfenstein 3D source tree at Icculus.org. It worked pretty well for a while with libc_r. The sound mixer portion was a separate thread, poorly implemented with a couple of pthread_* calls. From what I could tell, nothing was done to keep the mixer in sync with the rest of the program. Odd, it just worked, and it worked better on FreeBSD than Linux 2.4.x with just a few tweaks to the code (no flies on Linux, that's just what I happened to notice). While KSE was being implemented/debugged, I noticed that my attempts at linking wolf3d with KSE would result in the sound from the mixer thread being totally scrambled and sped up, but that the cpu usage of the entire program dropped dramatically. Linking with libthr instead is less interesting as the program just starts and exits without emitting an error. Funny, anyway.

  13. How to enable libmap by kaeru · · Score: 4, Informative

    To do this you need to recompile ld-elf.so.1 to support dynamic lib mapping:

    cd /usr/src/libexec/rtld-elf/

    make -DWITH_LIBMAP
    make install

    Then you need to have a libmap.conf in your /etc dir. Details of it can be accessed by reading the man pages libmap.conf(8).

    Here is a sample with mozilla firebird mapped to libkse:

    [/usr/X11R6/lib/firebird/lib/mozilla-1.4b/Mozill aF irebird-bin]
    libc_r.so.5 libkse.so.1
    libc_r.so libkse.so

    Just add more entries if you wish. You can test out whether an application is using libc_r or libkse by running ldd(8).

    Note the library mapping can be done as Scott mentions also for libthr also for 1:1 threading.

  14. Re:Elegy for *BSD by tarius8105 · · Score: 1

    And your words betray your point. If this were true, then BSD/OS or FreeBSD would not be the only ones with the longest uptime as the values get cycled after 497 days. Fact is if your point were correct then you would see HP-UX, Linux, or Solaris up there too.

  15. Re:bsd problems by Anonymous Coward · · Score: 1, Informative

    You know, under BSD (or any other Unix) you could have used the "sed" utility to do the search and replace on that file automatically with a single command, instead of deleting each occurance of "Mac" by hand and typing out "BSD" in its place in Notepad (even if you have discovered the search-and-replace facility in Notepad, you have to admit sed is much easier).

  16. Re:bsd problems by agent+dero · · Score: 1

    What is this guy's problem, this is the third time I have personally seen this exact same comment, most of them have been in MacOS X threads.

    This guy is one sour System 7 user.

    --
    Error 407 - No creative sig found
  17. KSE + SCHED_ULE by ByTor-2112 · · Score: 2, Interesting

    Unfortunately, the new smp-aware/optimized scheduler does not play well with KSE yet. It seems to work just fine with libthr but there are signal delivery problems with KSE that result in unkillable processes spinning endlessly. That said, it works fantastic with the old scheduler.

    FreeBSD has a number of projects that are nearing production-ready that will greatly enhance an already incredible OS!

  18. FreeBSD on SMP-steroids by Ezdaloth · · Score: 1

    Hope this becomes as stable and fast as the rest of freebsd. Way to go FreeBSD/KSE team, good job !