Slashdot Mirror
DragonFlyBSD 3.6 Brings AMD/Intel Graphics Drivers & Better SMP Scaling
An anonymous reader writes "DragonFlyBSD 3.6 was released [Monday] with the big new features being dports, Intel and AMD Radeon KMS kernel graphics drivers, major SMP improvements, and improved language support. Dports is the new package management system based upon the FreeBSD Ports collection and replaces pkgsrc as the default; over 20k packages are available via dports. Major SMP scaling improvements come via reducing lock contention within the kernel and other multi-core enhancements. The Intel and Radeon graphics drivers on DragonFlyBSD were ported from the FreeBSD kernel, which in turn were ported from the upstream Linux kernel."
Does anyone know if a three way performance comparison between Freebsd, Dragonfly and Linux has been done lately?
Has anybody else noticed that over the last decade, almost all of the DragonFlyBSD release stories have been posted by timothy, and the majority of those were submitted anonymously?
It's not a particularly popular distro, coming in at #77 today, in an unscientific poll. I get that it's news for nerds, but I'm starting to suspect a wee bit of bias.
You do not have a moral or legal right to do absolutely anything you want.
You know, I hear a lot of folks complain about Linux fragmentation, tyrany of choice, etc. But at least we can say that, for the most part, there is one true canonical Linux kernel (Linus' tree) and all the other kernels are for the most part shallow forks tweaking a few things.
Now in BSD land we have NetBSD, FreeBSD, OpenBSD, and DragonflyBSD, each with their own true kernel.
Why?
If the project goals have diverged so widely as to take the kernel off in a completely different direction from all the other BSD's why even call it BSD anymore?
What do the four big BSD distros have in commmon besides the name and a kernel they used to use years (decades?) ago?
I am admitedly ignorant and perhaps I am underestimating the degree of cooperation between these projects.
-73, de n1ywb
www.n1ywb.com
who?
It's a DOS clone that thinks it's still relevant. These losers should just move to Windows and stop clinging to text-mode DOS clones. The worst part is it isn't even binary compatible with old DOS games making it doubly irrelevant.
If it can't run Excel, it's pretty useless to us. I have a ton of VBA code that needs to just work - I don't have time to dick around with obscure OS's.
It's a fork of FreeBSD 4.8 from 2003. I ran the most recent freebsd and pc-bsd and both are rock solid. It was easier to install flash player on bsd than opensuse 13.1(it did not work even installing from the terminal). I ran linux open source applications just fine with no issue on pc-bsd 9.2.
Check http://www.dragonflybsd.org/features/, there's a list of random links under the DEVFS bullet point.
I want to delete my account but Slashdot doesn't allow it.
This release removes almost all the remaining SMP contention from both critical paths and most common paths. The work continued past the release in the master branch (some additional things which were too complex too test in time for the release). For all intents and purposes the master branch no longer has any SMP contention for anything other than modifying filesystem operations (such as concurrent writing to the same file). And even those sorts of operations are mostly contention free due to the buffer cache and namecache layers.
Generally speaking what this means is that for smaller 8-core systems what contention there was mostly disappeared one or two releases ago, but larger (e.g. 48-core) systems still had significant contention when many cores were heavily resource-shared. This release and the work now in the master branch basically removes the remaining contention on the larger multi-core systems, greatly improving their scaling and efficiency.
A full bulk build on our 48-core opteron box took several days a year ago. Today it takes only 14.5 hours to build the almost 21000 packages in the FreeBSD ports tree. These weren't minor improvements.
Where it matters the most are with heavily shared resources, for example when one is doing a bulk build on a large multi-core system which is constantly fork/exec'ing, running dozens of the same process concurrently. /bin/sh, make, cc1[plus], and so on (a common scenario for any bulk building system), and when accessing heavily shared cached filesystem data (a very common scenario for web servers). Under these conditions there can be hundreds of thousands of path lookups per second and over a million VM faults per second. Even a single exclusive lock in these paths can destroy performance on systems with more than 8 cores. Both the simpler case where a program such as /bin/sh or cc1 is concurrently fork/exec'd thousands to tens of thousands of times per second and the more complex case where sub-shells are used for parallelism (fork without exec)... these cases no longer contend at all.
Other paths also had to be cleaned up. Process forking requires significant pid-handling interactions to allocate PIDs concurrently, and exec's pretty much require that locks be fine-grained all the way down to the page level (and then shared at the page level) to handle the concurrent VM faults. The process table, buffer cache, and several other major subsystems were rewritten to convert global tables into effectively per-cpu tables. One lock would get 'fixed' and reveal three others that still needed work. Eventually everything was fixed.
Similarly, network paths have been optimized to the point where a server configuration can process hundreds of thousands of tcp connections per second and we can get full utilization of 10GigE nics.
And filesystem paths have been optimized greatly as well, though we'll have to wait for HAMMER2 to finish that work for modifying filesystem calls to reap the real rewards from that.
There are still a few network paths, primarily related to filtering (PF) that are serialized and need to be rewritten, but that and the next gen filesystem are the only big ticket items left in the entire system insofar as SMP goes.
Well, the last problem, at least until we tackle the next big issue. There's still cache coherency bus traffic which occurs even when e.g. a shared lock is non-contended. The code-base is now at the point where we could probably drop-in the new Intel transactional instructions and prefixes and get real gains (again, only applicable to multi-chip/multi-core solutions, not simple 8-thread systems). It should be possible to bump fork/exec and VM fault performance on shared resources from their current very high levels right on through the stratosphere and into deep space. Maybe I'll make a GSOC out of it next year.
The filesystem work on HAMMER2 (the filesystem successor to HAMMER1) continues to progress but it wasn't ready for even an early alpha
Honestly, Slashdot isn't the right place to post BSD stuff anymore. It's too full of immature posts and uninformed nonsense.
-Matt
Agreed - The BSD section used to be somewhat lively. There's an awful lot of hostility towards the BSDs as they're not Linux. People feel really threatened when there is an alternative to their favorite OS. I also have to laugh at posts about BSD fragmentation. How many Linux distros are there now? Oh, but they're all related! Of course, the BSDs share ancestors, so they're related, too.
The filesystem work on HAMMER2 (the filesystem successor to HAMMER1) continues to progress but it wasn't ready for even an early alpha release this release. The core media formats are stable but the freemap and the higher level abstraction layers still have a lot of work ahead of them.
Have you considered space maps for tracking free space? I thought that was one of the more interesting ideas in ZFS.
Anyway, great work on the SMP scalability. It is refreshing to see a concerted effort in reworking the system to be more SMP friendly, rather than the profuse and convoluted locking that most others have adopted.
And now they're gone, I guess someone over there read my post.
I want to delete my account but Slashdot doesn't allow it.
I've read the space map work but there are several issues that make them impractical for H2. The main one is that H2 snapshots are read-write (ZFS snapshots are read-only). Secondarily, my experience from H1 is that any complicated ref-counting or space accounting can lead to hidden corruption. Even if we assumed that the media data validation mechanism (a CRC or cryptographic hash) detects all media issues, there is still the problem of software complexity introducing bugs at a higher level.
So H2 is going to use a relatively simple freemap mechanism. In fact, it is using the same dynamic radix tree for the freemap as it does for the file and directory topology. It will still roll-up tiny allocations into a somewhat chunkier granularity to reduce freemap overhead (similar to what H1 did but at a finer 1KB grain vs H1's 2MB granularity), and it will roll-up space availability hints and propagate them up the tree. And it is type-aware so it can do things like collect inode allocations together for burstability. It isn't trivialized. But it does not attempt to implement ref-counting or any complex sub-topology to keep track of who and how many own which blocks.
The actual block freeing is going to be done in the background rather than in the 'rm' path (for a COW filesystem with snapshots which are actually used regularly, there's no point trying to free blocks in real time). This will allow H2 to implement several major features such as de-duplication, writable snapshots, and so forth with far less software complexity. So the complexity of the block allocation and any block duplication becomes trivialized while the complexity of the block freeing code increases. But it's an incrementally solvable problem. I can start with a brute-force memory-bounded scan and slowly add optimizations to recognize duplication in the topology, add topology hints to improve the efficiency of the block freeing scan, and so forth.
-Matt
...hundreds of thousands of tcp connections per second...
Hold on there, I'm still using DNet!
So are the vast majority of Linux distributions, but it sure is nice to have options, isn't it?
/* No Comment */
Slashdot really isn't the place to post anything technical anymore.
Thanks for that post Matt, there's still a few old skool folks about that like this stuph. /. has a lot of noise, but there's still some signal in there now and again.
I have plenty of common sense, I just choose to ignore it. -- Calvin
Seconded.
How your opinion correlates with Windows application and server OS?
I understand why you don't post this stuff yourself but as a hardcore Linux user I like to follow the developments of Dragonfly here. It's always pleasing to see the kind of serious OS design work that you do. Linux doesn't have everything good and nor should it.