Slashdot Mirror


User: m.dillon

m.dillon's activity in the archive.

Stories
0
Comments
771
First seen
Last seen
Profile
(view on slashdot.org)

Comments · 771

  1. Re:Shame.... on IRC Forum with Matthew Dillon of DragonFly BSD · · Score: 4, Informative
    The more I look back upon it the more I think forking off DragonFly was the right thing to do. The timing could not be better, in fact. A year ago would have been too early (because we would have had to backport a whole lot more from 5.x, whereas in the last year the FreeBSD folks have backported a considerable amount of work from 5.x to 4.x), A year from now would be too late (4.x would have been 'dead' for too long a period of time). Now is the right time.

    I still contribute to FreeBSD. If I see a bug or a security issue in DragonFly that also exists in FreeBSD they get a head's up, and some of the patch sets we've comitted to DragonFly have been organized in as legible a way as possible to promote a possible port from DragonFly to FreeBSD for those FreeBSD folks interested in the work. But my main focus these days is DragonFly.

    The biggest loss to FreeBSD from my departure is that they have one less person tracking down bugs in the kernel. To be sure, I had done less and less of that over the last year as FreeBSD-5 continued to diverge from what I believed could be supported, but before anyone chimes in with a silly 'Matt hasn't done anything' comment I will note that I spent several days reviewing and commenting on Jeff's scheduler and slab allocator work, which benefited greatly from the comments, plus brainstorming sessions with Julian on KSEs, with Alan and Tor on VM issues, and so forth. I had attempted to put FreeBSD-5 on a better track several times, but the ideas were rejected in piecemeal. I don't think there are any big-picture/multi-disciplinary folk left in the project, which is a problem. Basically the FreeBSD-5 team seems to suffer from a sort of myopia. It is possible to defend a piece of work taken in isolation, such as priority stealing for mutexes or kernel preemption, but when you look at the big picture there are simply too many such pieces, each complex in its own way, in the FreeBSD-5 kernel. The result is a huge mess that only a few programmers can actually navigate through without introducing new bugs, and that is a real problem insofar as progress in FreeBSD-5 goes. Too many developers are working only in their own little corner of their world and too few developers are doing general debugging and architectural work. And, most especially, too few developers are looking out for and supporting the end-users, something that has been a significant part of my work ethic ever since I wrote DICE. There are plenty of FreeBSD developers, their numbers certainly are not decreasing, but the class of developers is far less balanced now then it was in 1998. That is my opinion, anyway.

    Not having to deal with FreeBSD politics anymore as significant reduced my stress levels, and being able to work on innovative new designs rather then having to fix other people's bugs has improved my disposition dramatically :-).

    -Matt

  2. Re:focus on IRC Forum with Matthew Dillon of DragonFly BSD · · Score: 3, Informative
    Some MP locking is unavoidable, but DragonFly will have much, much less of it then any other MP operating system that I know of. Having to use an MP lock is basically an admission that there will be cache contention between cpus... if not the lock itself, then the data that is being covered by the lock. If you have 4 cpus and all of them at one time or another are accessing a common data structure, then you are wasting 3/4 of your available aggregate L1 and L2 cache space storing that data structure plus you are wasting cycles obtaining the lock, wasting hardware cycles due to cache management operations, and wasting cpu.

    Now, of course, some data structures are more important then others. It doesn't make sense to try to avoid all locking... there are plenty of situations where you have to eat the duplication in the cache and plenty of situations where you have to eat cache mastership changes.

    But there are a ton of situations where the waste is not necessary. In DragonFly these situations translate to: The scheduler, the slab allocator, device driver interrupt management, and the networking stack, just to begin with.

    Lets take the networking stack as an example. In a fine-grained mutex model if a cpu is available it might be called upon to run pending work for the network stack... say processing packets for various TCP connections. In order to do the work the kernel will pull the next packet off the queue, figure out the protocol and port, obtain a fine-grained mutex on the PCB, and then do the work.

    In the DragonFly model the network interrupt will queue the packet and the queueing code will peek at the packet and direct it to a particular protocol thread. Any given connection will be directed to a single thread so, for example, if you have a 4-cpu system and you have told the kernel to create 4 TCP protocol stack threads (one on each cpu), then each new TCP connection will be assigned to a particular thread. Only that thread will process packets associated with that particular TCP connection, which means that no locks are required to process packets for that TCP connection and no L1/L2 cache duplication will occur between cpus because only one thread will ever touch it. If you have 1000 TCP connections, each of your four TCP protocol stack threads will be assigned approximately 250 of them.

    In DragonFly's case the trick then becomes to balance the assignments, but even here we get a leg-up because simply distributing the connections by ^^ mod the number of protocol threads will get you most of the way to that goal. When you add in the cache-locality you have just achieved (you have 4 times the available data cache by not sharing the PCBs across multiple cpus) the result is far higher performance then a fine-grained-but-perfectly-balanced model could ever give you.

    -Matt

  3. Re:The real shame on IRC Forum with Matthew Dillon of DragonFly BSD · · Score: 2

    Gee I hope not *exactly* like Theo. But watch out who you lambast. Theo is responsible for or as managed some fairly major improvements in the open-source world, and we are far better off with that work.

  4. Re:Is this the guy that wrote dcron? on IRC Forum with Matthew Dillon of DragonFly BSD · · Score: 3, Interesting
    Yah. I still get an occassional email regarding dcron. DCron was written because, at the time anyway, the VixieCron that Linux was using had a huge number of bugs in it. For example, just changing the time could cause VixieCron to suddenly execute hundreds of jobs all at once. This was a long time ago and I expect VixieCron has been fixed since then, but that was the original motivation for writing DCron. I just wanted something small, simple, and dependable, with per-user capabilities.

    It is gratifying to see that DCron is still being used regularly, but I am a bit miffed that nobody has gone and made any truely significant improvements to it. That is part of the nature of open-source, I guess.

    -Matt

  5. Re:Why Dragonfly? on IRC Forum with Matthew Dillon of DragonFly BSD · · Score: 4, Interesting
    DragonFly's first official release will be in a little less then a year. Half of that time is going to be spent rewriting the infrastructure, which is outlined on the site www.dragonflybsd.org We already have light weight threads and our slab allocator in place (which are a biggies) and the basic infrastructure for syscall messaging and DEV messaging in place. We are also steadily bringing in those elements from FreeBSD-5.x that make sense. At the moment my focus is on fixing inherited bugs and rewriting the namecache code. Other DragonFly developers are focused on syscall sepraration, networking, and userland bits. We are being very, very careful to preserve and improve upon the stability inherent in earlier 4.x releases. In fact, the namecache stage-3 work has been delayed over two weeks now while I have backed up and refocused on fixing a number of unrelated bugs.

    The biggest topic on our lists right now is the discussion on the infrastructure required to support a good binary and source-based package management system. FreeBSD's package management system currently suffers greatly from bitrot and upgrade-breakage (where upgrading one package breaks half a dozen others), amoung other problems. We are discussing the use of variant symlinks and VFS-based 'environments' for deploying and validating port dependancies in a manner that allows multiple versions of anything to coexist on the same system.

    Basically you can think of DragonFly as being all the things I've always wanted to do but couldn't due to politics in the FreeBSD camp. FreeBSD-5 has gone down a road that I believe will lead to a slow, cumbersome, and unmaintainable result. There are plenty of examples of this but the most aggregious is thread preemption in the kernel, preemptive thread movement between cpus (also while in the kernel), and a ridiculously complex API for various types of mutex operations. There are good things in FreeBSD-5, and we will snarf them as appropriate.

    The timing is appropriate. FreeBSD-4.x is nearing its end-of-life, and it is obvious that less care is being taken in preserving its vaunted stability then in the past. DragonFly will be the logical successor to FreeBSD-4.x statring in around a year, and I also fully expect to beat the shit out of FreeBSD-5 SMP-performance-wise by the time too.

    DragonFly's kernel model is basically to operate on cpus in isolation, with movement beween cpus strictly managed. For example, there is a separate light weight kernel scheduler on each cpu which handles all operations related to that cpu and requires the use of no mutexes. The slab allocator is also per-cpu and mutexless. If an operation needs to work on a data structure owned by another cpu an asynchronous IPI (message) is sent to that cpu rather then performing a locking/mutex operation. For example, if cpu #1 needs to free data owned by cpu #2's slab allocator, cpu #1 queues the work to cpu #2 (and in the case of a memory free, the work does not have to be done immediately). The term 'asynchronous' is important in this context, because it means that the IPI latency between cpus can be entirely absorbed and that sequences of messages (in more heavily loaded systems) can be processed in a tight loop. When we get into more complex entities such as the networking stack, the intention is to run the protocol stacks as threads (possibly multiple threads, one on each cpu or as needed by the situation) and assign protocol control blocks to particular threads on particular cpus. This eliminates nearly all locking and mutex operations related to any given protocol control block and guarentees cache locality of reference. If done properly the individual cpu caches on SMP systems will wind up with less duplication and far higher efficiencies then one gets using a fine-grained locking mutex model. In fact, DragonFly's model works equally well in both UP and SMP environments and we expect the SMP support will cause virtually no loss in performance on UP boxes.

    -Matt

  6. Re:Too many Matt Dillons! on IRC Forum with Matthew Dillon of DragonFly BSD · · Score: 4, Informative
    You could credit me with being the primary mover for fixing the VM system going from 3.x to 4.x, and the buffer cache as well. I did some rewriting when it was called for, but most reorganized smaller sections of code in order to get rid of race conditions (bug fixing), clean up performance issues, and so forth. This created some friction because nobody else was willing to do the necessary work yet everyone still had an opinion on how the work should be done.

    I also completely rewrote the swap management code from the old list-of-holes format to a fixed radix tree with dynamic size hinting, fixed issues in VN and MFS, and generally put out a lot of fires all over the tree. Most of what I did in BSDland was fix bugs, because there were a lot of bugs that needed fixing and, again, there isn't much of a point having an advanced operating system if it crashes every so often.

    I probably spent far more time fixing bugs then anything else. My involvement with FreeBSD began during my BEST Internet days, in the 94 or 95 timeframe, with a commit bit coming in '98 I think, but my involvement with BSD began in 1985 at UC Berkeley, at about the same time I got involved with the Amiga.

  7. Worm Load on Microsoft "Swen" Worm Squiggles Into Sight · · Score: 4, Interesting
    There were over 4500 attempted deliveries of this 150K+ worm through my mail server overnight, and they are still coming. Easy to filter, but this is by far the worst worm load I've seen to date on my little server.

    On the bright side, deliveries of unrelated spam seem to have fallen due to the worm's load on the internet :-)

  8. DragonFly at BSDCon on BSDCon '03 Nearly Here (OpenBSD 3.4, Too) · · Score: 5, Informative
    I will be attending BSDCon this coming week and will be setting up a BoF session on DragonFly as well as doing a quicky summary WIP (Work In Progress) talk. Some extremely interesting work is being done in DragonFly as well as in other BSD's (FreeBSD-5, OpenBSD, and so forth). But the conference is not for the weak of heart. If all you care about is the next K00l music UI don't bother to come {EVIL GRIN}.

  9. DragonFly on DragonFly BSD: Daily Snapshots Available · · Score: 2, Informative
    It should be noted that DragonFly, being a fairly new project, is not meant to be production-ready for some time. Despite the short time since we forked from FreeBSD-4 we have made considerable progress, including getting a fully optimized light weight kernel thread scheduler in place, asynchronous IPI messaging (between cpus), a high performance slab allocator, and other items.

    DragonFly will be doing things that the other BSDs simply cannot, primarily due to having to support a large existing user bases. For example, right now I am in the midst of completely rewriting the VFS file path lookup code (namei, lookup, vfs_cache_lookup, VOP_LOOKUP, VOP_CACHEDLOOKUP). This is not something the other BSDs would be able to easily do though it might just be possible to port it to FreeBSD-5 once it is done. But the results are going to be phenominal... an almost complete removal of the vnode locking requirements for path lookups, and at least a 3x improvement in path lookup performance. We are also converting all system calls and both the file descriptor and DEV interfaces to messaging interfaces and asynchronizing the path all the way through using Amiga-style semi-synchronous I/O messaging (and it would be a serious mistake to compare the methodology to, say, Mach messaging), so it will be possible to support userland threads without eating a kerneland stack context for each running operation. There are many goals and this is going to be a multi-year project.

    It is quite possible that DragonFly will be an alternative upgrade path for FreeBSD-4.x users rather then going to FreeBSD-5. FreeBSD-4 is nearing its end-of-life and DragonFly is really going to give FreeBSD-5 (and linux for that matter) a run for its money. We are taking an entirely different approach to SMP, one that involves asynchronous inter-cpu messaging to resolve conflicts and requires far fewer mutex operations in the critical path. Those interested in understanding the new approach should read DragonFly's light weight kernel threading code (kern/lwkt_thread.c and friends).

    The first user release will probably not happen for a year. In the mean time, only serious developers and knowledgeable programmers should really be using DragonFly.

  10. Google pulled a fast on on the RIAA on Google Removes Kazaa Links, Keeps Sponsored Links · · Score: 4, Insightful
    You guys are idiots. Google pulled the links, but they also put a link up to the original DMCA notice that was sent to them which... guess what? LISTS THE BANNED SITES!

    So now instead of these sites being deep in the list of results, they are now front and center, and probably getting far more hits then they were before the notice was sent.

  11. Re:The real question is: How do I short SCOX? on SCO Says It Has No Plan To Sue Linux Companies · · Score: 1
    The lack of shares makes the stock highly volatile and basically only speculators are buying and selling it, which is why it often seems to go counter to what you would expect. But the low volume also means that insiders cannot sell huge chunks of stock without destroying the stock price. For example, if you look at the graph for today a single 7500 share transaction dropped the stock price by $0.50, which is huge. Basically they are playing a push-pull game with speculators. Eventually the house of cards will collapse but it is impossible to say when or to predict how the stock will behave through to that point. Also, every insider share sold throws more shares onto the open market.

    Shorting can be dangerous if you don't have enough reserve to handle the volatility. Buying long term PUT options is more expensive, but a lot safer. Unfortunately no options are available on SCO.

  12. Re:SCO doesn't want to expose its trade secrets on "Stolen" SCO Linux Code Snippets Leaked · · Score: 1
    It can't be a trade secret if it is published on dozens, if not hundreds of web servers, and when hundreds of thousands of sites have a copy of the linux source. The only 'secret' here is that SCO is refusing to point out which parts of the Linux kernel contain their so-called trade secrets, and that itself cannot be a trade secret.

    SCO has used a wide range of terminology freely, everything from 'IP' to 'Copyright' to 'Trade Secret'. Not one of their public arguments makes any sense and they are fast running out of ammunition to spend on their public persona.

  13. Re:Why does IBM take it? Here's why... on "Stolen" SCO Linux Code Snippets Leaked · · Score: 3, Interesting

    Part of IBM's countersuit is an indictment of SCO's use of press releases, stated exaggerations and falsehoods in interviews, and so forth. By not engaging in these tactics itself IBM is adding weight to its countersuit. Besides, IBM has all of us to beat the drums for it.

  14. Re:Doesn't affect SCOX stock price at all on "Stolen" SCO Linux Code Snippets Leaked · · Score: 4, Insightful
    The price of a stock traded on the NYSE or NASDAQ (not counting over-the-counter (OTC) trading) is determined by the open market. Holding a high percentage of stock in a company does not really have any effect on its price, only on transaction volume. Transaction volume can have an indirect effect on price but the volatility you get from low transaction volume works in both directions. The same low trading volume makes it easier for speculative actions to move the price up or down also makes it fairly difficult for insiders to unload massive amounts of stock all at once, because doing so will kill the price. They are basically playing a game with the speculators in order to try to maintain the price of the stock and yet still be able to slowly sell their own, IMHO.

    Also don't forget the shorts. At some point shorts have to buy the stock back, which can boost the price of the stock. I don't think short covering is a big part of the current holding value of SCOX but it does tend to mitigate the downramp a bit when downramps happen.

    In short, SCOX is a highly speculative and volatile stock and any simplistic view of cause and effect in the matter is no more accurate then rolling the dice.

  15. Re:It's from the BSD and PDP11 sources on "Stolen" SCO Linux Code Snippets Leaked · · Score: 1
    Caldara (aka SCO) gave the linux people this code under the BSD license. See my reference (reprinted below). Note that there is a link to a PDF in this reference which clearly states that the PDP code was given to the linux folks.

    Reference to Linux-Kernel posting

    SCO is screwed.

  16. This code was apparently donated by Caldara (SCO) on "Stolen" SCO Linux Code Snippets Leaked · · Score: 4, Informative
    From the linux-kernel list, the code was apparently donated by Caldara under the BSD license in 2002. Here are the references.

    Start of Thread

    Conclusion

  17. SCO is really getting desperate on SCO Announces Final Termination of IBM's Licence · · Score: 3, Interesting
    SCO is really getting desperate. They've cried wolf so many times now that even the stock speculators have finally caught on. Their stock price hardly blipped at the latest announcement. SCO is fast running out of curve balls to throw.

    In fact, IBM addresses both the issue of the stock price manipulation and SCO's continued misrepresentation of the AIX license in their counterclaim. For SCO to continue to make these sorts of public statements is insane.

  18. Not too hard on Surviving Slashdotting with a Small Server · · Score: 1
    I just turned on tcp inflight limiting on my FreeBSD box, which works kinda like TCPVegas, and even though the T1 bandwidth was pegged all day long it survived being slashdotted just fine. Of course the site only had one or two carefully resized graphics on it in anticipation. Still, it worked. Even the slowest consumer box these days can web serve far more bandwidth then a T1, DSL, or cablemodem can support so it really comes down to bandwidth management. If the client is downloading a large picture and its going slowly you want to be sure that when he cancels your server isn't going to push out another 64KB+ of data before it figures it out, nor do you want to allow too many packets to backup on your interface (which destroys the latency for everything going out of the machine), The inflight limiting solves that problem.

    -Matt

  19. Re:Too big on 4Gb CF Card Announced · · Score: 1

    Unbelievable. I am glad his wife and friends were able to recover his legacy!

  20. Re:How about Manufacturers on 4Gb CF Card Announced · · Score: 1

    Hmm. Well, remember that zip/gzip basically use the LZH algorithm, which is good at finding and compressing duplicate substrings. These algorithms will *NEVER* do a good job on high resolution images due to pixel flicker in the lower bits, even from a high quality camera. It is certainly possible to compress these images far more then you get with the RAW output, with little or no loss in quality, but you aren't going to get it zip/gzip.

  21. Re:How about Manufacturers on 4Gb CF Card Announced · · Score: 1
    True, which is why RAW images are more then twice as large as a high quality JPEG but still not as large as a TIFF. But RAW images don't compress either. If I try to gzip a JPEG I get a file that is basically the same size as the original JPEG. If I try to gzip a (canon) RAW file from my EOS-10D, I get a file that is basically the same size as the original as well. In otherwords, standard compression will not work. JPEG compression on the otherhand works wonderfully but of course it is lossy.

    Canon at least is very conservative when you tell it to produce a high quality JPEG. The only thing you lose is contrast from the 12bpp->8bpp conversion. This is important, of course, but it depends on the type of shot being taken.

    If I take the high quality JPEG output from the Canon EOS-10D and run it through a JPEG->JPEG converter with the quality set to 80% the original 2.5MB JPEG turns into a 654MB JPEG and I am hard pressed to see any difference in quality between the two.

    What this tells me is that we need a new JPEG standard which operates on 16bpp instead of 8bpp, and once we have it there won't be much of a reason for even professionals to use RAW any more.

  22. Re:You'd pay $1,500 for that? on 4Gb CF Card Announced · · Score: 1

    I can find 1GB cards in the $200-$250 range, but not under a $100. 18 months for the price to drop that much is possible, but not very likely because the demand curve isn't going to be there for several years. Vendors have been playing around with 11Mpix cameras but 11Mpix is massive overkill for 95% of the photographers out there. Even with a 6Mpix camera the lenses you use and the quality of the CCD/CMOS sensor are far more important then the number of pixels. In fact, I would go as far as to say that the next big improvement is going to be a move towards 16 bits per pixel, which when coupled with further improvements on digital noise issues will yield film-like contrast and exposure capabilities.

  23. Re:Too big on 4Gb CF Card Announced · · Score: 1
    I suppose it's always a worry but compact flash is far more robust then a microdrive. Even if the filesystem gets corrupted you can still stick the CF card into a computer and probably recover most of the data. I don't even consider microdrives any more because of the horror stories I've read on their fragility, but I've never lost data from a (flash based) CF card.

    Still, the entry point price of the 4GB card is far too expensive relative to the price of 1GB cards. People with 11Mpix+ cameras might buy them, but the price performance curve is still squarely in the 512M-1GB range.

    -Matt

  24. Re:How about Manufacturers on 4Gb CF Card Announced · · Score: 1
    Well, zip and gzip don't do a very good job on JPEGs, since JPEGs are already highly compressed. You can already 'save' CF space simply by telling the camera to take a lower quality or lower resolution shot. I can put 300-500 high resolution JPGs on a 1GB CF card with my EOS-10D (6Mpix) but if I drop the resolution and/or quality down that number increases beyond 2000. Of course, nobody in their right mind actually takes pictures at less then full quality when one has that kind of storage, but it's still an option.

    In a few years all high grade consumer cameras are going to have automatic WIFI upload capabilities anyway, which will limit how much in-camera CF capacity you actually need in a professional setting. At the moment the new Nikon has it but it isn't worth the price premium. Or, alternatively, there will be CF format cards with WIFI. NFS anyone?

  25. Re:Rechargable Alcalines on Rechargeable Batteries - Yes or No? · · Score: 4, Informative
    I would recommend you throw your rechargeable Alkalines away. 'Rechargeable' Alkalines are really just normal alkalines with a beefy casing to reduce leakage, but the full charge drops off significantly even after the first recharge cycle and gets worse from there. Alkalines are fragile beasts compared to NiMh. With an Alkaline the charger has to be very careful not to damage the battery. With NiMh the charger pretty much just pushes in current and limits by temperature, and the battery is very forgiving to chargers that overdo it.

    -Matt