Understanding OS X Kernel Internals

jglidell writes "The OS X kernel has been in the news alot this past year, whether it's why its slow, Mach/micro-kernel makes it bad, it's going closed source and what not. Amit Singh has put up a new presentation on the innards of OS X. It does a pretty good job of summing up the OS X kernel architecture, and has some pretty detailed diagrams... for instance they show that there are so many process/threads layers in OS X. So if you are in the mood for doing some OS studying then head over."

Comments

huh?

[Aussie]

whether it's why its slow

What the hell does that mean ? Editors drunk ?

Re:huh?

[Ant P.]

It might make more sense in this format, and without the grammar error:

...in the news alot this past year, whether it's:

• why it's slow,
• Mach/micro-kernel makes it bad,
• it's going closed source
• and what not

Re:huh?

[jaysones]

"Alot" isn't even an actual word.

Re:huh?

[M. Baranczak]

Nope, it's still gibberish.

whether it's why it's slow

Well... why is it slow?

Mach/micro-kernel makes it bad

Debating the pros and cons of Mach is a valid topic, but a phrase like this is so vague that it's meaningless.

it's going closed source

OK, that one's intelligible. But then we come across gems like this:

for instance they show that there are so many process/threads layers in OS X.

A small request for submitters: Take a minute to actually proof-read your summary. I'm not even talking about simple typos, or the correct use of "you're/your" - those look ugly, but most of the time people can still figure out what you meant. Just ask yourself: will these words make sense to a moderately intelligent English speaker who's not on a meth bender?

Needs more editor.

[dajobi]

"alot"
"whether it's why its slow"
"they show that there are so many process/threads layers in OS X."

Do the editors even look at submissions any more? Or to put it another way, is our children learning yet?

Re:Needs more editor.

[OECD]

Do the editors even look at submissions any more?

I'm afraid they do. I think the problem is that they're not as skilled in writing english as they are in writing PERL. (That's not a slam, by the way. I suck at PERL.)

And before anyone goes on an "Off Topic" jag, it really does make a difference if the readers can understand what's being written. I stumbled over the "that there are so many" sentence a couple times trying to make sense of it. There are so many process threads layers in OS X that what? It slows it down? It's hard to program? Or is there simply a gee-whizz lot?

Yeah, I know, I'm off to R the FA. I just wish I had a better idea of what's in there.

Re:Needs more editor.

[gowen]

I think the problem is that they're not as skilled in writing english as they are in writing PERL.

I've seen slashcode.

I think the problem is that they're exactly as skilled in writing english as they are in writing PERL.

Where are good internal docs?

[thogard]

I have a small program that mmaps a bit of code and then points the program counter at it. Everything runs fun until a OS call happens. I've heard that Mach allows user land programs to install their own OS calls but I haven't seen any example code to do it and I suspect such a feature isn't in OS X. I've hunted through the source and I while I could write my own system call and compile it in, there should be an easier way. Can anyone point me in the correct direction?

Re:Where are good internal docs?

[CableModemSniper]

I don't know if this what you are looking for, but: http://rentzsch.com/mach_inject/

Re:OS X Kernel - Why?

[Fordiman]

"OS virtualization is going to kill off the native OS X software market"

Ahh, and welcome back to another thrilling episode of "doesn't know what the fuck they're on about" theatre!

Honestly. OSV is SLOW compared to native. No one wants slow.

Terrible summary.

[IANAAC]

I could barely get through the summary.

If English is a second language for the submitter, fine. But good grief, do you suppose one of the PAID editors could have done just a bit of work to make the summary more readable?

Re:Spelling

[dhasenan]

Not to mention fonts. I can never forgive anyone for using Comic Sans in a presentation.

Re:Spelling

[mangu]

"alot" isn't a word! It's "a lot."

I also get kinda pissed off because lotsa people write it that way.

"OS X is slow" claims investigated

[Logic and Reason]

Before anyone starts spouting off again about Mac OS X being "slow by design" or somesuch, read this article by an Apple engineer that investigates those claims.

Re:"OS X is slow" claims investigated

[jcr]

It shows how the author completely misses the criticism that he is trying to address and fails miserably in his arguments.

It does nothing of the kind. Read the whole article, and ALL of the comments, sparky, not just the ones that support your prejudice.

Peter's point is that the "benchmark" in question boils down to how quickly OSX will malloc and free 35KB blocks. It happens to cross the mmap() threshold for kernel versus heap allocations.

It also shows how Apple fanboys applaud the author's explaination, even though they haven't a clue about the subject matter - and even admit it in their posts.

If you had actually refuted anything that Peter had said in his article, you might have a leg to stand on, but since you didn't, and since you toss off an epithet like "fanboy" instead of doing so, I conclude that you are the clueless one.

No wonder you're an AC.

-jcr

Re:"OS X is slow" claims investigated

[gnasher719]

'' It's a good article. I wonder why OSX isn't compiled with a larger heap threshold, though? I'd do with a 2x performance increase on common operations even if it cost more memory (in the form of fragmentation). My only guess is that in real world usage this is not common, and thus it wouldn't improve real-world performance much? ''

It is _not_ a common operation.

As that article explained, the developers of the software in question could have used Shark (comes for free with every Macintosh), and within 20 seconds they would have found that they are wasting tons of time by allocating and deleting memory again and again and again and again and again. Note that for calls to malloc () to make any noticable difference in execution time, you must make an awful lot of them. So twenty seconds of observation plus two hours of tweaking their code would have made their software substantially faster _on every_ platform.

Typical code has few large allocations, which stay around for long time, and possibly a huge number of small allocations that get allocated and freed very quickly. So an implementation does the small allocations itself (because it is quicker) and leaves the large allocations to the OS (because that helps with fragmentation), just as MacOS X does.

Re:Profit!

[moro_666]

somebody must have mispelled "book commercial" as "presentation".

there's nothing really new in that presentation, most of slashdotters know this stuff already, the only thing that we didn't know as of yet is that you can mispell "book commercial" in such an interesting way.

you can "troll" or "flamebait" my post, but this is the way that it is.

Ad

[AstrumPreliator]

I looked at the "presentation" and no, it doesn't do a very good job of explaining anything. Maybe combined with an extensive lecture to explain what the hell he's talking about would make it a bit more clear. From what I saw it was basically just enumerating the different components. Then I noticed the second to last slide. It's basically an ad for a book coming out.

Maybe it's just me though. Did anyone else find it extremely enlightening?

Underpowered Little Machine

[Beefslaya]

As the purchaser of a brand new Core Duo Mini, (my first Mac, I feel "as happy as a little Gurlll!") I noticed that my system out of the box with 512 of RAM was dog slow when you start loading iPhoto, or any more then 2 apps.

Initial startup yielded a smoking fast web browser, and other single line items.

I purchased the 2GB Ram upgrade (not from Apple at 600 USD, 280USD from Crucial) and I noticed such a difference, that I couldn't understand WHY they would even consider shipping that little silver wonder with less then 1GB of RAM.

It's not the kernel, it's the apps... They just don't give enough power to the off the shelf machines to support the great apps that come with it.

Vive le Mac... Thanks for putting excitement back into computing for me.

Re:Underpowered Little Machine

[caseih]

I think much of OS X's ram-hungriness comes from the fact that outside of the system frameworks, there is very little utilization of shared libraries among the different applications. Each app bundle is largely self-contained with its own shared libraries. Granted most apps that ship with OS X (from apple) just access the shared frameworks in /Library/Frameworks and have few other dependencies in their bundles. But start adding apps like MS Word, Firefox, OpenOffice, etc, and you'll start having multiple copies of various libraries loaded. The app bundle system is very simple and reliable, but because of the shared library issue, you'll always need more ram when running these apps on OS X than Windows or Linux.

Definitely 1 GB is a minimum amount of RAM needed for OS X Tiger these days. That is quite sad when you think about it, but RAM is cheap so I'm not too concerned about it. Apple has always shipped their machines short on RAM, hoping you'll pay ridiciulous amounts of money for their official RAM upgrades.

Re:Underpowered Little Machine

[Crizp]

Not to be a total Apple fanboy but all mainstream computer suppliers sell PC's with half the amount of RAM that's usually needed. When Windows ran fine on 128 MB, you got 64. Now you usually get 512 MB, but XP SP2 is slow with less than a gig. Especially if you're playing games, but also using regular apps.

Re:Underpowered Little Machine

[ickies]

"I noticed that my system out of the box with 512 of RAM was dog slow when you start loading iPhoto, or any more then 2 apps." You know, for the first several hours of uptime after starting Tiger for the first time, depending on how much data you have, the Spotlight is indexing all your drives in the background and the system is SLOW AS MOLASSES. Like, painfully so. But it speeds back up after it finishes all that initial indexing.

Re:Underpowered Little Machine

[jocknerd]

You also have to remember that the Mac mini has integrated graphics which uses some of the memory. I've heard it uses about 80MB of your memory. But I agree, it needs at least 1GB to run smoothly.

Re:Underpowered Little Machine

[Golias]

Buying RAM for a Mac isnt' what it used to be. If I remember right, before the MacIntels, and such, you had to buy special matched/paired memory kits for upgrades, specially from Apple.

You don't remember right.

I've owned Macs for years, and never once used anything other than cheap third-party memory to upgrade them. The G3 Towers were very picky about memory that was up to spec, but even then there was no need to buy from Apple.

And how, exactly, would you suggest installing "matched/paired memory kits" into the SINGLE SLOT in the G4 version of the Mac mini?

Re:Underpowered Little Machine

[MacBastard]

Well, you're both kinda right.

Certain Power Macintosh models, usually the higher end ones, could use matched pairs so that you could interleave memory pages. If you went that route, supposedly you could squeeze out some more system performance. If I remember right, Apple claimed you could gain up to 20% more total system performance on the PowerMac 8500/9500 systems by activating memory interleaving. The only people that I remember doing this were heavy Photoshop users or serious Adobe Premiere video editors. Remember, this was about when DRAM was considered cheap at around US$12 per MB!

It didn't have to be from Apple, either. I remember setting people up with lots of Kingston SIMMs and DIMMs back then.

Otherwise, you could put any memory that fit (except ECC) in any slot order, and it would run fine that way, too.

Re:Underpowered Little Machine

[MojoStan]

Not sure why my post was rated funny, but oh well. Karma is karma.

Being modded "Funny" doesn't improve you karma anymore. Your post seems to indicate that you didn't know this, but I'm not sure. Anyhoo, here's the relevant FAQ:

What is karma?

Your karma is a reference that primarily represents how your comments have been moderated in the past. Karma is structured on the following scale "Terrible, Bad, Neutral, Positive, Good, and Excellent." If a comment you post is moderated up, your karma will rise. Consequently, if you post a comment that has been moderated down, your karma will fall.

In addition to moderation, other things factor into karma as well. You can get some karma by submitting a story that we decide to post. Also, metamoderation can cause your karma to change. This encourages good moderators, and ideally removes moderator access from bad ones.

Note that being moderated Funny doesn't help your karma. You have to be smart, not just a smart-ass.

Re:Underpowered Little Machine

[tyrione]

Perhaps if those applications actually leveraged Cocoa's AppKit Frameworks, Core Data, Core Imaging, etc., then you'd see drastically smaller app bundles and more shared use of built-in Cocoa Frameworks.

Since they are Carbon based expect them to be bloated.

When Apple replaces Finder and other critical sections of their application base with pure cocoa applications then perhaps we'll see more improvements as we should have seen.

Re:The average person

[LWATCDR]

So go read a less technical website than Slashdot.
This story was boring because the presentation was dumb. It had a little useful information but not much. The subject is interesting.
OS/X security? How to lock it down? There are many sites on the web. Look for just about any good site about UNIX security and start from that. OS/X uses the UNIX security model.

Closed?

[ShadowBottle]

Riiiight. Just because some idiot alarmists say that the kernel has gone closed when it simply just hasn't been released yet, the media and clueless bloggers start crying that it's gone closed source.
"Well... it hasn't. It's still open. IT JUST HASN'T BEEN RELEASED YET.
OSNews is reporting that Ernest Prabhakar, Apple's Open Source and Open Standards product manager, has stated in the Fed-Talk mailing that Apple has not actually closed Mac OS X's Darwin kernel for the Intel version of the OS; they simply haven't released it yet. Speculation about Apple closing the kernel arose from the fact that other non-kernel Darwin sources actually have been released, and the previous PowerPC-based kernel is still available as open source as well.Ernest wanted to make sure that tech media didn't confuse 'speculation' with 'fact'. A good lesson we all could benefit from...."

God damn alarmist idiots.

Re:Closed?

[squiggleslash]

Uh, you don't appear to be presenting an argument, you seem to have an unsupported opinion. By the way, your position (not argument) makes sense to you. Yet you have at least two people who appear to disagree with the "logic" which lead you to your position, not that that really matters.

An unsupported opinion is one where facts are not presented that back up the opinion. Disagree with me all you like, but the facts do actually agree with my position, which is why I used them in my argument.

The two people disagreeing with me haven't really given me a reason to change my mind. JCR initially reponded to my argument with responses to a completely different argument (namely that I was expecting Apple to release details about a future product), and then simply implied I was wrong without explaining why. You're doing much the same thing, even accusing me of creating "straw men" in an argument that didn't actually have anything of the sort.

He doesn't need one. He's merely presenting what he sees as relevant facts to the seemingly unreasonable.

If you want to convince someone that their argument is incorrect, then yes, you need a counter argument to it. If I say "1+1=2", then a counter argument "No it isn't" isn't going to convince me otherwise. If you, however, explain your reasoning, I might agree with you if the argument makes sense. Right now, we have a discussion that goes something like this:

Me: 1+1=2
JCR: No, you can't use an assignment operator like that
Me: Erm, I wasn't using it as an assignment operator, if you add one and one, you get two.
JCR: You may get two, that doesn't mean it's the right answer
You: One and one becomes two? I don't think that can possibly work
Me (exasperated to JCR): Can you explain your reasoning?
JCR: I've got 20 years of being a know-it-all. How dare you challenge my foolproof knowledge!
You: Why should he have to explain anything, it's you who's in the wrong, look, two of us disagree with you!

All I've said so far is that:

1. The claim that Apple has said that the source for XNU for Intel is merely delayed is false. The evidence doesn't back that up at all. Apple has made no comment. That's a fact, and the idiot OP who posted otherwise should have been modded to the floor.

2. The fact that Apple has made no comment suggests that the future release of the source code is open to question. There are multiple explanations as to why they'd not have released the source code yet, but right now the most probable are to do with Apple being in a position where they may never release it. If there was no change of policy, if they still expected to release the source code, they'd have no problems saying that. They have no problems confirming they still sell MacBooks. They have no problems confirming iTunes will continue to support iPods.

Now, you and JCR seem to think otherwise, or else you're responding to an entirely different argument (as JCR initially did.) As I've clarified that point to JCR, there's not much excuse for that.

But, whatever, here's the central point: If you disagree with me, explain why. Because I can't change my mind unless you do. If you're just masturbating, posting "No you're wrong" because it turns you on, by all means continue, but there's little point.

Re:Closed?

[Warlock7]

After re-reading your posts, and the responses, I don't disagree with you. I interpreted your statements to be supporting Yager's interpretation as fact.

Yager's speculation as to why they might close it sounds viable, but Prabhakar says that it's merely speculation. Which it is. Yager's assumption doesn't seem to address the possibility that Apple might be changing from XNU to something else entirely, which has been continually speculated for years now. Yager also ignores the possibility that they might be drastically changing it and how it operates and withholding it until they've gotten it to the point that they are comfortable with releasing it. Nobody but those inside Apple know for sure and speculation presented as fact is useless, it only serves to exacerbate matters.

Since they have never released XNU under X86 thus far it is still "business as usual" at Apple even though they've always made XNU under PPC available. They have time to make their decision and it's well within their right to do so. So, as I said before, be patient. The news will come when they are ready to release it. Until then, it doesn't matter if you or anybody else demands them to make their intentions known now or not. They've, according to Yager, stated that "...the state of an OS X-compatible open source x86 Darwin kernel is in flux." Until they release it or announce that they won't we've got unsupported speculation based on a lack of evidence.

If I called Apple because I'd waited a month for my MacBook to arrive, and asked if the MacBook had been discontinued, they'd be pretty open about the fact it's still in production.

This is what I saw as your straw-man. This has nothing at all to do with your point. While you may be attempting to make a valid correlation here, it really isn't one. Your attributing this position to Apple, your true opponent, as they haven't given you what you want, information/code. This is not as simple as your hardware shipping late and Apple needing to explain themselves. I can see the point you are getting at, but the statement doesn't really apply the same way as the situation under discussion.

The fact that Apple has made no comment suggests that the future release of the source code is open to question.

This is very true. Sorry for getting into reactionary /. mode. It was a long day and sometimes that will happen...

Re:Spelling

[Tibor the Hun]

Alot of what youre saying makes good sense.

OSX: Highly Thread Sensitive

[ElitistWhiner]

As an ex-NeXT developer, the historical speed bumps behind this architecture are directly related to code density. NeXTSEP ver 2.0 was nicely running on 25mHz. NeXTSTEP ver 3.0 suffered performance on 25mHz compared to ver 2.0 because of the additional kernel overhead resultant from code densification.

With every ver. release through 10.4.x MacOS X, mach/BSD layer exhibits funtional improvements with speed increases of the processor CPU and latent performance behaviors from the additional kernal overhead added by code complexity and densification.

Prima Facia evidence to the 4X speed improvement in performance from Apple's new Intel CPU bears witness to the limits of the kernal architecture.

Re:OS X Kernel - Why?

[MagnusDredd]

Quoth the AC:
200 bucks for a Windows license? I don't think so. Are you so clueless that you think IT people drive down to the local computer store and buy a boxed copies of Windows???

You're assuming that the only market for software is medium to large enterprise. Autocad (which is made by a well known vendor) is used by tons and tons of architects. Since I've worked in IT support for a few architectural firms, I can tell you that most of them are comprised of small business of under 50 employees. I've used Autocad under Windows, however when I needed to use something to make some plans for a shed for my church, I used Sketch-Up, because there's an OSX native version.

Furthermore, most of the people that I've run across who are excited about virtualization are primarily interested in using it to run the existing Windows apps they own, and plan on replacing those with OSX native versions where available, and comparable OSX native apps where not available. The other major segment (where I fall into this equation)interested in virtualization, is the IT people who use/support multiple platforms.

Boot Camp is for games. Noone I have spoken with that knows about virtualization is seriously intersted in using it for anything other than games.

Saith the parent:
"If you have enough Mac customers that your Mac software is a profitable product, you are going to lose money from your decision. If you didn't have that many Mac customers, you shouldn't have been making a Mac product anyway."

Quoth the AC:
Woah!

Only someone who has never worked in a real life software dev house could make such a wildly stupid statement.

+ Savings from dumping Mac developers and overhead to support them
+ Savings from dumping Mac support and packaging
- Percentage of Mac running customers who stop buying your product due to canceled OS X version

Revenue goes down, but profits go up in almost every case outside of having 75-100 percent or so of your Mac customers stop buying your product. It is pretty funny to see Mac people freaking out about everyday business economics - "But that won't happen because I wouldn't like it to!" Heh.

What?
I'm not understanding something...

This seems like Carly Fiorina Logic. If we stop making HP calculators, people will just buy handheld HP units with similar funtionality. It doesn't matter that the calculator group provides a substantial net profit for the company. If we stop spending all that money on supporting calculators and simply use the computer support people we already have... Think of the savings!!!!

When a group/division in a company that caters to a market makes the company lots of money, leave them alone. They're doing their job, pulling a PHB stunt in wall street business tactics will only hurt your company.

Re:Kernel and Kernel Theory Aside

[Guy Harris]

Its kernel is based on BSD which wasn't designed to have locks, so locking is much more coarse than it needs to be.

Few, if any, current UN*Xes were "designed to have locks", but many of them (Solaris, Linux, FreeBSD, OS X) have made their locks finer-grained anyway; it's not as if you're stuck with all the consequences of the "design" forever.

The Mach base (which as far as I can tell is completely unnecessary) adds extra indirection to system calls.

What are some examples of this extra indirection? (Note that system calls are not redirected to userland servers.)

Macs are intended to be a single-user interactive system, and most apps spend most of their time waiting for the user, so perf just isn't the high priority it has to be on Windows.

Windows personal computers (as opposed to servers) aren't intended to be single-user interactive systems?