OpenBSD 4.9 Released

An anonymous reader writes "The release of OpenBSD 4.9 has been announced. New highlights included since 4.8: enabled NTFS by default (read-only), the vmt(4) driver by default for VMWare tools, SMP kernels can now boot on machines with up to 64 cores, support for AES-NI instructions found in recent Intel processors, improvements in suspend and resume, OpenSSH 5.8, MySQL 5.1.54, LibreOffice 3.3.0.4, and bug fixes." Also in BSD news, an anonymous reader writes "DragonFly BSD 2.10 has been released! The latest release brings data deduplication (online and at garbage-collection time) to the HAMMER file system. Capping off years of work, the MP lock is no longer the main point of contention in multiprocessor systems. It also brings a new version of the pf packet filter, support for 63 CPUs and 512 GB of RAM and switches the system compiler to gcc 4.4."

Comments

Why is NTFS read only.

[jack2000]

Why is NTFS always read only. It shouldn't be so hard to make a proper file system driver what the hell?

Re:Why is NTFS read only.

[Phibz]

You do realize that NTFS is completely closed source right? All the work on it has been done through reverse engineering.

Re:Why is NTFS read only.

[DarkOx]

Add to that a few other fun things

1.Multiple versions of NTFS with subtle changes
2.Its a complex file system with lots of features, some of which are not even used by windows but you still have to take care of the on disk data correctly.
3.The security scheme does not cleanly map onto UNIX style rules even with ACL support and such.

NTFS is by no means avant guard but its by any means simple and without documentation figuring out its internals completely and correctly is a BIG job. Now why they can't gleen allot of that from the Linux source I don't know. I know they can't use the Linux source because of the GPL being incompatible with BSD maybe there is a contamination concern.

Re:Why is NTFS read only.

[hedwards]

Contamination isn't normally an issue for kernel code, they can always cram it in its own corner of the code and not include it in binaries by default.

Without being involved with the discussions its hard to say, but I've personally found Linux filesystem code to be less than reliable. But there's also the issue of that it would have to pass their auditing to be included in the base install, there's a reason why they have so few base exploirts.

Re:Why is NTFS read only.

[Hognoxious]

NTFS is by no means avant guard

Just like your knowledge of French, it would seem.

Re:Why is NTFS read only.

[phantomcircuit]

NTFS is only writable on linux though NTFS-3G, the write support in the kernel only works if the file size doesn't change.

Re:Why is NTFS read only.

[mlts]

Those are important items, especially #1. There are a lot more which make life hell for someone trying to get NTFS to work fully as a supported filesystem for a UNIX based OS. A few more:

4: Alternate data streams. It is common for malware to add an ADS onto a file, a directory, a junction point, or even the C: drive object itself. Without a dedicated utility that snorts out these, they are essentially invisible.

5: Like #1 above, NTFS changes in undocumented [1] ways. For example, EFS changed to add different encryption algorithms between Windows XP and Windows XP Service Pack 3. So, not knowing that may bring someone a world of hurt.

6: Similar to #3, NTFS's ACLs are hard to reimplement in the UNIX world. U/G/O permissions can be mapped (Cygwin does this).

7: For a filesystem to be usable as a production one, it needs a filesystem checking utility that can go through the whole filesystem and check/repair integrity on every part of it, be it mostly unimplemented/unused items (transactional-NTFS), features off the filesystem (NTFS compressed files, EFS), and many other items.. Yes, there are ways to run Windows's chkdsk.exe utility, but that is a hack at best.

One of the biggest problems with operating systems today is that there are no compatible filesystems beyond FAT and FAT32. Perhaps UFS. Either one filesystem has too much patent encumbrance to be used, or its license.

I wonder how easy life would be if we had a standard filesystem that could replace the LVM (similar to ZFS), offer modern features (deduplication, encryption, 64-bit checksumming [2], encryption, compression (various levels), snapshotting [3]. On an LVM level, it would be nice to have mountable disk images similar to OS X's sparse bundles. If something changes on the encrypted drive, only a few bands change, as opposed to having to back up the whole file.

Life would be easier if every OS out there had a common filesystem with modern features. A good example about how useful this would be would be antivirus scanning. Unpresent a LUN from a Windows server, scan it on a Solaris box for malware, then re-present it, for example.

[1]: Undocumented unless you are elite enough to have the MS source code handed to you, all work on the filesystem is all reverse engineering.

[2]: Backup programs would have it easy and not rely on dates or archive bits... just look for files where the checksum has changed and back those up just like the -c option in rsync.

Re:Why is NTFS read only.

[fuzzyfuzzyfungus]

The specifications for NTFS are completely closed. If it's what Windows produces when told to format a volume as NTFS, it is NTFS. There are reverse-engineered attempts(NTFS-3G being the most practical, if rather slow); but they aren't entirely to the point where you'd want to trust vital data to them.

In the specific case of OpenBSD, I suspect that the read-only support is because the OpenBSD team has very low tolerance for what they see as crap. If they can't support something the way that they want to, they can and will just toss it(see the Adaptec RAID driver case, or some wireless chipsets). They don't do binaries, they don't do NDAs, they don't do blobs. They also don't like software they consider to be of inadequate quality. Thus, since the state of full NTFS support is a bit dodgy, it is entirely in character for them to drop it.

More broadly, NTFS read/write isn't really something that there is a strong incentive in the OSS world to polish to a high sheen. NTFS-3G is pretty much good enough for dual booters and rescue disks. NTFS doesn't have any points of superiority strong enough that building top-notch reverse-engineered support would be competitive with spending the same effort implementing a non-secret design. Also, for the sorts of purposes that pay the bills for a lot of Linux development, NTFS support is largely irrelevant. You don't dual-boot servers, and any halfway serious network setup is going to either use SMB/NFS(which makes the local filesystem irrelevant to all other hosts), or some filesystem with concurrent access support or other esoteric features that isn't NTFS.

NTFS R/W is really just a convenience feature for sneakernet and dual-boot scenarios. Neither of those really pay for enough development to get a fully baked reverse engineering of a (quite complex) filesystem.

Re:Why is NTFS read only.

[the_B0fh]

eh? The last wireless fiasco I remembered was one of the linux wireless guys stealing openbsd's reversed engineered code, and re-releasing it as their own. I guess you can say they threw away encumbered code as they reverse engineered and re-wrote it.

Re:missing some key features...

[snowgirl]

wake me when they have:

1) start/stop scripts, so I don't have to ps|grep|kill|...crap, what were those flags for the daemon again... to manage running processes or daemons

Well, for this one:

New rc.d(8) for starting, stopping and reconfiguring package daemons:
The rc.subr(8) framework allows for easy creation of rc scripts. This framework is still evolving.
Only a handful of packages have migrated for now.
rc.local can still be used instead of or in addition to rc.d(8).

Re:missing some key features...

[rubycodez]

why, they're not necessary. The flags for starting a daemon are in /etc/rc.conf and /etc/rc.conf.local, and the pid of running daemons are in /var/run or use ps ea for them. Simple and clean with no cruft is why I like OpenBSD for applicances and routers so much.

Re:At the risk of being modded flamebait, etc

[Impeesa]

Netcraft confirms it, BSD jokes are dead.

Re:63 CPUs?

[m.dillon]

The basic mobo support for large N-way configurations has gotten cheap. Power management still has a long ways to go on these beasts, though. Our monster.dragonflybsd.org box is using the quad-socket supermicro mobo with four 12-core opterons (48 cores) and 64G of ram, and I think all told cost around $8000 or so.

The limitation for for these sorts of boxes is basically just power now. The 12-core opterons are effectively limited to 2GHz due to power issues, and these big beasts are really only high performers in environments where all the cores can be used concurrently with very little conflict.

By comparison, a PhenomII x 6 or an Intel I7 runs 6 cores for the PhenomII and 4 x 2 cores for the I7 but automatically boosts the base ~3.2GHz clock to almost 4 GHz when some of the cores are idle. These single chip solutions also have a MUCH faster path to memory than multi-chip solutions, particularly the Intel Sandybridge cpus, and much faster bus locked instructions. So if your application is only effectively using ~4-6 cores concurrently it will tend to run at least twice as fast as it would on a high-core-count monster.

That means that for most general server use a single-chip multi-core solution is what you want. The latest single-chip mobos for Intel and AMD support 16G-32G of ram and 5 or more SATA-III (6GHz) ports. Throw in a small SSD and you are suddenly able to push 400MBytes/sec+ in random-accessed file bandwidth out your network using just ONE of those SATA-III ports. That's in a desktop configuration! So today's modern desktop mobos is equivalent to last year's server mobos at 30-50% the power cost.

A modern high-end configuration as above eats ~60W idle where as the absolute minimum power draw on a 48-core Supermicro box w/ 64G of ram (the ram eating most of the power) is ~250-300W. Big difference.

So lots of cores is not necessarily going to be the best solution. In fact, probably the only really good fit for a 48+ core box is going to be for virtualization purposes.

-Matt

Re:At the risk of being modded flamebait, etc

[m.dillon]

ZFS has a large team of people behind it and resources that I don't have. That said HAMMER wasn't really designed to try to compete against it. HAMMER was designed to solve similar problems, but it wasn't designed to replace RAID as ZFS was. But ZFS is no panacea, and anyone who uses it can tell you that. The IP is now owned by Oracle, the license isn't truly open-source. ZFS itself is an extremely heavy-weight filesystem and essentially requires its ARC cache and relatively compatible workloads to work efficiently... and a veritable ton of memory.

HAMMER has a tiny footprint by comparison, gives you fine-grained automatic snapshots, and most importantly gives you near real-time queueless mirroring streams that makes creating backup topologies painless. Among many other features. Frankly ZFS might be the filesystem of choice if you are running dozens of disks but HAMMER is a much better fit otherwise.

People scream the RAID mantra all the time but the vast majority of people in the open-source world don't actually need massive RAID arrays to put together a reliable service. Often it takes just one 2TB HD and one 80G SSD x a few servers and in DragonFly HAMMER + swapcache fits that bill extremely well.

Our ultimate goal is real-time multi-master clustering. HAMMER doesn't get us quite there, primarily owing to the topology mismatch between HAMMER's B-Tree and OS filesystem cache topologies (mostly the namecache), but as the work progresses it will eventually achieve that.

In anycase, there's a huge difference between the people who do the actual design and implementation of these filesystems and the people who merely use them. Our goals as designers and programmers are not necessarily going to match the goals of the typical end-user who wants a magical black box that does everything under the sun with maximal performance in all respects and works without having to life a finger. ZFS can't even achieve that!

-Matt