IRC Forum with Matthew Dillon of DragonFly BSD

weebl writes "Thursday October 9th at 6:00PM PDT (9PM EDT/1AM GMT) SlashNET's #forum channel will be hosting a Q&A session with Matthew Dillon of the DragonFly BSD Project. This is your opportunity to ask about DragonFly BSD, BSD in general, or any other questions you might have for him. DragonFly BSD was first announced this past July." If you can't make it to the forum, SlashNET will have a bot running earlier in the day for question submissions, and logs available afterward.

Comments

Re:focus

[NightSpots]

Since it's based on FreeBSD 4, it'll be "All of good things of FreeBSD 4 with a focus on performance."

There's a LOT of work going into fine grained locking to allow faster SMP (which is a classic slowdown in FreeBSD 4) - light weight kernel threads, advanced caching, messaging APIs, and even plans for a package system that (if it works?) would completely change the way people think about installing third party software.

There are a lot of big things planned. It may be slow to start, hopefully it'll take off... and we'll see some cross development with the other BSDs.

The logs will be at slashnet.org

[Motherfucking Shit]

For some reason, nobody ever bothers to mention where the logs of the Slashdot IRC forums get posted. After the IRC interview with CmdrTaco and Hemos a few months ago, it took me some digging to figure out where the log wound up.

For those who can't make the chat, the log will eventually be at http://www.slashnet.org/forums/

Editors: After the chat is over, any chance of having the log URL linked to the story text as an update?

Re:Why Dragonfly?

[be-fan]

Dragonfly isn't a completed project at this point, or even one that made changes substantial enough to differentiate it from other systems. Dragonfly forked from FreeBSD because they have some very different ideas about where the kernel should be headed than the FreeBSD folks. Dragonfly will be based on a model of very lightweight threads that communicate via a messaging mechanism. They think they can get highly scalable designs without all the locking that the Linux and FreeBSD designs use. If it works, it could be a very large step forward for multi-CPU and clustured systems.

Re:Too many Matt Dillons!

[MobyTurbo]

Finally I remembered the other Matt Dillon who developed the DICE C compiler for the Amiga back in the good old days.

That Matt Dillon is the FreeBSD one, he did both the DICE compiler and the FreeBSD VM (and some other FreeBSD stuff.)

Re:focus

[Leimy]

As far as I know... and I have been following the list as well as reading the website... I see no plans to rely on fine-grained locking as an SMP enabling mechanism. There is an AmigaOS based messaging system that will allow people to run things like Filesystem drivers in userland should they choose to or they can keep it in ther kernel.

But why are you reading /. comments to find out what this thing is? Go to the website and see for yourself.

Re:Too many Matt Dillons!

[m.dillon]

You could credit me with being the primary mover for fixing the VM system going from 3.x to 4.x, and the buffer cache as well. I did some rewriting when it was called for, but most reorganized smaller sections of code in order to get rid of race conditions (bug fixing), clean up performance issues, and so forth. This created some friction because nobody else was willing to do the necessary work yet everyone still had an opinion on how the work should be done.

I also completely rewrote the swap management code from the old list-of-holes format to a fixed radix tree with dynamic size hinting, fixed issues in VN and MFS, and generally put out a lot of fires all over the tree. Most of what I did in BSDland was fix bugs, because there were a lot of bugs that needed fixing and, again, there isn't much of a point having an advanced operating system if it crashes every so often.

I probably spent far more time fixing bugs then anything else. My involvement with FreeBSD began during my BEST Internet days, in the 94 or 95 timeframe, with a commit bit coming in '98 I think, but my involvement with BSD began in 1985 at UC Berkeley, at about the same time I got involved with the Amiga.

Re:focus

[m.dillon]

Some MP locking is unavoidable, but DragonFly will have much, much less of it then any other MP operating system that I know of. Having to use an MP lock is basically an admission that there will be cache contention between cpus... if not the lock itself, then the data that is being covered by the lock. If you have 4 cpus and all of them at one time or another are accessing a common data structure, then you are wasting 3/4 of your available aggregate L1 and L2 cache space storing that data structure plus you are wasting cycles obtaining the lock, wasting hardware cycles due to cache management operations, and wasting cpu.

Now, of course, some data structures are more important then others. It doesn't make sense to try to avoid all locking... there are plenty of situations where you have to eat the duplication in the cache and plenty of situations where you have to eat cache mastership changes.

But there are a ton of situations where the waste is not necessary. In DragonFly these situations translate to: The scheduler, the slab allocator, device driver interrupt management, and the networking stack, just to begin with.

Lets take the networking stack as an example. In a fine-grained mutex model if a cpu is available it might be called upon to run pending work for the network stack... say processing packets for various TCP connections. In order to do the work the kernel will pull the next packet off the queue, figure out the protocol and port, obtain a fine-grained mutex on the PCB, and then do the work.

In the DragonFly model the network interrupt will queue the packet and the queueing code will peek at the packet and direct it to a particular protocol thread. Any given connection will be directed to a single thread so, for example, if you have a 4-cpu system and you have told the kernel to create 4 TCP protocol stack threads (one on each cpu), then each new TCP connection will be assigned to a particular thread. Only that thread will process packets associated with that particular TCP connection, which means that no locks are required to process packets for that TCP connection and no L1/L2 cache duplication will occur between cpus because only one thread will ever touch it. If you have 1000 TCP connections, each of your four TCP protocol stack threads will be assigned approximately 250 of them.

In DragonFly's case the trick then becomes to balance the assignments, but even here we get a leg-up because simply distributing the connections by ^^ mod the number of protocol threads will get you most of the way to that goal. When you add in the cache-locality you have just achieved (you have 4 times the available data cache by not sharing the PCBs across multiple cpus) the result is far higher performance then a fine-grained-but-perfectly-balanced model could ever give you.

-Matt

Re:Shame....

[m.dillon]

The more I look back upon it the more I think forking off DragonFly was the right thing to do. The timing could not be better, in fact. A year ago would have been too early (because we would have had to backport a whole lot more from 5.x, whereas in the last year the FreeBSD folks have backported a considerable amount of work from 5.x to 4.x), A year from now would be too late (4.x would have been 'dead' for too long a period of time). Now is the right time.

I still contribute to FreeBSD. If I see a bug or a security issue in DragonFly that also exists in FreeBSD they get a head's up, and some of the patch sets we've comitted to DragonFly have been organized in as legible a way as possible to promote a possible port from DragonFly to FreeBSD for those FreeBSD folks interested in the work. But my main focus these days is DragonFly.

The biggest loss to FreeBSD from my departure is that they have one less person tracking down bugs in the kernel. To be sure, I had done less and less of that over the last year as FreeBSD-5 continued to diverge from what I believed could be supported, but before anyone chimes in with a silly 'Matt hasn't done anything' comment I will note that I spent several days reviewing and commenting on Jeff's scheduler and slab allocator work, which benefited greatly from the comments, plus brainstorming sessions with Julian on KSEs, with Alan and Tor on VM issues, and so forth. I had attempted to put FreeBSD-5 on a better track several times, but the ideas were rejected in piecemeal. I don't think there are any big-picture/multi-disciplinary folk left in the project, which is a problem. Basically the FreeBSD-5 team seems to suffer from a sort of myopia. It is possible to defend a piece of work taken in isolation, such as priority stealing for mutexes or kernel preemption, but when you look at the big picture there are simply too many such pieces, each complex in its own way, in the FreeBSD-5 kernel. The result is a huge mess that only a few programmers can actually navigate through without introducing new bugs, and that is a real problem insofar as progress in FreeBSD-5 goes. Too many developers are working only in their own little corner of their world and too few developers are doing general debugging and architectural work. And, most especially, too few developers are looking out for and supporting the end-users, something that has been a significant part of my work ethic ever since I wrote DICE. There are plenty of FreeBSD developers, their numbers certainly are not decreasing, but the class of developers is far less balanced now then it was in 1998. That is my opinion, anyway.

Not having to deal with FreeBSD politics anymore as significant reduced my stress levels, and being able to work on innovative new designs rather then having to fix other people's bugs has improved my disposition dramatically :-).

-Matt