IRC Forum with Matthew Dillon of DragonFly BSD

weebl writes "Thursday October 9th at 6:00PM PDT (9PM EDT/1AM GMT) SlashNET's #forum channel will be hosting a Q&A session with Matthew Dillon of the DragonFly BSD Project. This is your opportunity to ask about DragonFly BSD, BSD in general, or any other questions you might have for him. DragonFly BSD was first announced this past July." If you can't make it to the forum, SlashNET will have a bot running earlier in the day for question submissions, and logs available afterward.

Comments

Shame....

[NightSpots]

His contributions to FreeBSD will be missed.

I'm sure I represent a large portion of the community who greatly appreciated his work on the VM subsystem (he even pointed the Linux folks in the right direction on more than one occassion), and am disappointed to see him leaving the project.

I understand that not everyone gets along, that goals differ between members; it's just a shame to see it happen.

Re:Shame....

[CoolVibe]

Why the hell is everyone thinking that Matt left? This fork isn't an 'angry' fork. DFBSD is just another BSD based off of FreeBSD 4 that is exploring some new ideas that Matt has. It's evolution. If DragonFly beats the pants off of 5.x in the future, I'm sure lots of DragonFly code will be appearing in the "official" FreeBSD tree.

Matt already wrote a SHITLOAD of code for DragonFly. He already overhauled the threading in the kernel, and put in a totally new slab allocator. Right now he's overhauling the namecache system.

Also, the DragonFly gang are doing chores nobody in the official FreeBSD camp cares enough about, like removal of __P(), removal of the 'register' keyword and ANSIfication of old K&R code. Of course we'll be seeing those efforts back in the other BSDs.

These changes that Matt and his merry gang of hackers are making are changes that would never be accepted by the FreeBSD deities with a commit bit,because they're intrusive. Hence the reason Matt forked off DragonFly.

He never left FreeBSD, dammit. Get over it.

Re:Shame....

[m.dillon]

The more I look back upon it the more I think forking off DragonFly was the right thing to do. The timing could not be better, in fact. A year ago would have been too early (because we would have had to backport a whole lot more from 5.x, whereas in the last year the FreeBSD folks have backported a considerable amount of work from 5.x to 4.x), A year from now would be too late (4.x would have been 'dead' for too long a period of time). Now is the right time.

I still contribute to FreeBSD. If I see a bug or a security issue in DragonFly that also exists in FreeBSD they get a head's up, and some of the patch sets we've comitted to DragonFly have been organized in as legible a way as possible to promote a possible port from DragonFly to FreeBSD for those FreeBSD folks interested in the work. But my main focus these days is DragonFly.

The biggest loss to FreeBSD from my departure is that they have one less person tracking down bugs in the kernel. To be sure, I had done less and less of that over the last year as FreeBSD-5 continued to diverge from what I believed could be supported, but before anyone chimes in with a silly 'Matt hasn't done anything' comment I will note that I spent several days reviewing and commenting on Jeff's scheduler and slab allocator work, which benefited greatly from the comments, plus brainstorming sessions with Julian on KSEs, with Alan and Tor on VM issues, and so forth. I had attempted to put FreeBSD-5 on a better track several times, but the ideas were rejected in piecemeal. I don't think there are any big-picture/multi-disciplinary folk left in the project, which is a problem. Basically the FreeBSD-5 team seems to suffer from a sort of myopia. It is possible to defend a piece of work taken in isolation, such as priority stealing for mutexes or kernel preemption, but when you look at the big picture there are simply too many such pieces, each complex in its own way, in the FreeBSD-5 kernel. The result is a huge mess that only a few programmers can actually navigate through without introducing new bugs, and that is a real problem insofar as progress in FreeBSD-5 goes. Too many developers are working only in their own little corner of their world and too few developers are doing general debugging and architectural work. And, most especially, too few developers are looking out for and supporting the end-users, something that has been a significant part of my work ethic ever since I wrote DICE. There are plenty of FreeBSD developers, their numbers certainly are not decreasing, but the class of developers is far less balanced now then it was in 1998. That is my opinion, anyway.

Not having to deal with FreeBSD politics anymore as significant reduced my stress levels, and being able to work on innovative new designs rather then having to fix other people's bugs has improved my disposition dramatically :-).

-Matt

Why Dragonfly?

[Valar]

If I am a happy Linux or BSD (of some non-Dragonfly kind), then what would you say to entice me to switch? In other words, what do you think Dragonfly is particularly good at, that maybe is lacking elsewhere?

Re:Why Dragonfly?

[m.dillon]

DragonFly's first official release will be in a little less then a year. Half of that time is going to be spent rewriting the infrastructure, which is outlined on the site www.dragonflybsd.org We already have light weight threads and our slab allocator in place (which are a biggies) and the basic infrastructure for syscall messaging and DEV messaging in place. We are also steadily bringing in those elements from FreeBSD-5.x that make sense. At the moment my focus is on fixing inherited bugs and rewriting the namecache code. Other DragonFly developers are focused on syscall sepraration, networking, and userland bits. We are being very, very careful to preserve and improve upon the stability inherent in earlier 4.x releases. In fact, the namecache stage-3 work has been delayed over two weeks now while I have backed up and refocused on fixing a number of unrelated bugs.

The biggest topic on our lists right now is the discussion on the infrastructure required to support a good binary and source-based package management system. FreeBSD's package management system currently suffers greatly from bitrot and upgrade-breakage (where upgrading one package breaks half a dozen others), amoung other problems. We are discussing the use of variant symlinks and VFS-based 'environments' for deploying and validating port dependancies in a manner that allows multiple versions of anything to coexist on the same system.

Basically you can think of DragonFly as being all the things I've always wanted to do but couldn't due to politics in the FreeBSD camp. FreeBSD-5 has gone down a road that I believe will lead to a slow, cumbersome, and unmaintainable result. There are plenty of examples of this but the most aggregious is thread preemption in the kernel, preemptive thread movement between cpus (also while in the kernel), and a ridiculously complex API for various types of mutex operations. There are good things in FreeBSD-5, and we will snarf them as appropriate.

The timing is appropriate. FreeBSD-4.x is nearing its end-of-life, and it is obvious that less care is being taken in preserving its vaunted stability then in the past. DragonFly will be the logical successor to FreeBSD-4.x statring in around a year, and I also fully expect to beat the shit out of FreeBSD-5 SMP-performance-wise by the time too.

DragonFly's kernel model is basically to operate on cpus in isolation, with movement beween cpus strictly managed. For example, there is a separate light weight kernel scheduler on each cpu which handles all operations related to that cpu and requires the use of no mutexes. The slab allocator is also per-cpu and mutexless. If an operation needs to work on a data structure owned by another cpu an asynchronous IPI (message) is sent to that cpu rather then performing a locking/mutex operation. For example, if cpu #1 needs to free data owned by cpu #2's slab allocator, cpu #1 queues the work to cpu #2 (and in the case of a memory free, the work does not have to be done immediately). The term 'asynchronous' is important in this context, because it means that the IPI latency between cpus can be entirely absorbed and that sequences of messages (in more heavily loaded systems) can be processed in a tight loop. When we get into more complex entities such as the networking stack, the intention is to run the protocol stacks as threads (possibly multiple threads, one on each cpu or as needed by the situation) and assign protocol control blocks to particular threads on particular cpus. This eliminates nearly all locking and mutex operations related to any given protocol control block and guarentees cache locality of reference. If done properly the individual cpu caches on SMP systems will wind up with less duplication and far higher efficiencies then one gets using a fine-grained locking mutex model. In fact, DragonFly's model works equally well in both UP and SMP environments and we expect the SMP support will cause virtually no loss in performance on UP boxes.

-Matt

Re:It is now official

[endx7]

It is now official - Netcraft has confirmed: *BSD is dying

I really shouldn't reply to this kind of comment, but doesn't that comment seem kinda weird if you consider the fact netcraft runs FreeBSD?

Re:focus

[NightSpots]

Since it's based on FreeBSD 4, it'll be "All of good things of FreeBSD 4 with a focus on performance."

There's a LOT of work going into fine grained locking to allow faster SMP (which is a classic slowdown in FreeBSD 4) - light weight kernel threads, advanced caching, messaging APIs, and even plans for a package system that (if it works?) would completely change the way people think about installing third party software.

There are a lot of big things planned. It may be slow to start, hopefully it'll take off... and we'll see some cross development with the other BSDs.

Too many Matt Dillons!

[PHPee]

As soon as I saw Matt Dillon's name associated with BSD, I got really excited. I mean, who knew that the ol' gunslinger Marshal Matt Dillon became a programmer?

Then, I realized I must have been thinking about the wrong Matt Dillon, but I still thought it was weird that the guy from There's Something About Mary became involved in a BSD project.

Finally I remembered the other Matt Dillon who developed the DICE C compiler for the Amiga back in the good old days.

Re:Too many Matt Dillons!

[m.dillon]

You could credit me with being the primary mover for fixing the VM system going from 3.x to 4.x, and the buffer cache as well. I did some rewriting when it was called for, but most reorganized smaller sections of code in order to get rid of race conditions (bug fixing), clean up performance issues, and so forth. This created some friction because nobody else was willing to do the necessary work yet everyone still had an opinion on how the work should be done.

I also completely rewrote the swap management code from the old list-of-holes format to a fixed radix tree with dynamic size hinting, fixed issues in VN and MFS, and generally put out a lot of fires all over the tree. Most of what I did in BSDland was fix bugs, because there were a lot of bugs that needed fixing and, again, there isn't much of a point having an advanced operating system if it crashes every so often.

I probably spent far more time fixing bugs then anything else. My involvement with FreeBSD began during my BEST Internet days, in the 94 or 95 timeframe, with a commit bit coming in '98 I think, but my involvement with BSD began in 1985 at UC Berkeley, at about the same time I got involved with the Amiga.

The logs will be at slashnet.org

[Motherfucking Shit]

For some reason, nobody ever bothers to mention where the logs of the Slashdot IRC forums get posted. After the IRC interview with CmdrTaco and Hemos a few months ago, it took me some digging to figure out where the log wound up.

For those who can't make the chat, the log will eventually be at http://www.slashnet.org/forums/

Editors: After the chat is over, any chance of having the log URL linked to the story text as an update?

Re:focus

[thallgren]

>There's a LOT of work going into fine grained locking to allow faster SMP

Unless I'm totally wrong, this is exacly the opposite from what DragonFlyBSD is about. Matt has never liked the way FreeBSD and Linux are designed, a kernel sprinkeled with locks.

Instead, he's trying to do a kernel where kernel-side subsystems communicate via message-passing, not too far from how exec.library worked in AmigaOS.

So Matt and his cowboys/cowgirls are actually removing locks. :-)

Regards, Tommy