Slashdot Mirror
Mac OS X 10.3 Defrags Automatically
EverLurking writes "There is a very interesting discussion over at Ars' Mac Forum about how Mac OS X 10.3 has implemented an on-the-fly defragmentation scheme for files on the hard drive. Apparently it uses a method known as 'Hot-File-Adaptive-Clustering' to consolidate fragmented files that are under 20 MB in size as they are accessed. Source code from the Davwin 7.0 Kernel is cited as proof that this is happening."
I can only speak for ext2/ext3. Linux tries to preallocate large blocks when writing files to prevent fragmentation. If you disk is mostly full (or even was once mostly full) or you have heavy concurrent disk activity going on you can still get fragmentation.
The end goal of the disk subsystem is to get your data to you as soon as you need it. In general that goal would be achieved if the data you want to read next happened to be under the read head. If you're reading sequentially through a single file then this will happen when the file is in a single contiguous region (i.e. unfragmented). For any other access pattern fragmentation doesn't matter as much, since you'll be skipping around the disk regardless of how the files are arranged.
Prefetching heuristics and caching can hide a lot of these problems from the user as well.
Yes it would be a one-time hit, but we hard disk intensive audio and video people don't want to be streaming multiple tracks off our hard disks while they are defragging themselves!
...which is why there's the 20 MB limit.
But instead of defraging FFS just makes better decisions as to where to put files....Seriously, if Apple got rid of HFS, none of this technology would be necessary.
You speak as though HFS+ has trouble with file fragmention. It's easily already one of the best filesystems for avoiding fragmentation - I've worked on Macs that have been run for years without attention and were better than 90% unfragmented. This is considerably better than any of the Microsoft filesystems, for instance. This tweak is an improvement, to get from 90% to 99%.
HFS+ doesn't just put the files down randomly, either, it has some smarts.
This also explains why the hard drive on my iBook seems alot hotter since upgrading.
The only way this feature can do that is if you're writing small files continuously. That's very strange software behavior, and perhaps a worst case for this optimizer. Why would you be doing that?
Don't get me wrong, HFS+ isn't the best filesystem ever created, but it's very featureful (multiple forks, file ID's, case-preserving, case-insensitive-possible, UNICODE, attributes, 64-bit file sizes, POSIX compliance, etc.) and the MacOS relies on it heavily. Anything to replace it would be a superset of HFS+. Fortunately, Apple hired the guy behind the Be Filesystem a few years back. I doubt he's working on iMovie 3.1.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
According to the ARS writeup, this feature is on only when journalling is on. This makes total sense, since journalling prevents an incomplete or unverified write from being used.
Some drink at the fountain of knowledge. Others just gargle.
If you refer to a file by an inode you are basically creating a hard link so if the file is deleted the file still 'exists'.
Also you cannot get a file path from an inode thus if the file path is changed (moving a file for example) the application cannot know what the new path is.
A FileID is really more equivalent to a path, or rather used in place of a path with the advantage that the path can change and the fileID remains the same. Thus referring to a FileID is less fragile.
Also FileIDs are smaller so searching for files using a FolderID or FileID is faster and uses less memory.
They're not equivalent.
>80 column hard wrapped e-mail is not a sign of intelligent
>life
A more technical explanation: http://developer.apple.com/technotes/tn/tn1150.htm l
>80 column hard wrapped e-mail is not a sign of intelligent
>life
Its a just recently added feature!
See the -s option for newfs_hfs:
(from man newfs_hfs)
-s
Creates a case-sensitive HFS Plus filesystem. By default a case-insensitive filesystem is created. Case-sensitive HFS Plus file systems require a Mac OS X version of 10.3 (Darwin 7.0) or later
"I'm a Genius!"*
*Not an actual Genius
The source code is posted to that thread; the only conditions are (1) 3 minutes after the system time starts (i.e. avoid doing so when booting up), (2) less than 20 MB of size, (3) file isn't already opened.
The only negative consequence is a possible speed hit, though. There's no danger.
I'm pretty impressed by this. Sure, it's been done before. Sure, there are more elaborate methods. But this is just a simple little lump of code that'll defragment the worst files most of the time.
this should defrag all of the 20M or less files on your hard drive.
it locates every file, opens it and reads every bite then closes it.
This should force the defragger to run on all files under 20M. Not that technically the defragger only activates when the file is broken into more that 8 extent regions. So this does not actually defrag everything.
but its also possible that having the file broken into less extents is harmless. first because the the first 8 extents are the fasttes to access in HFS+ and second its theoretically possible that on a multi-headed disk drive that having the file slightly fragmented might be good. Larger numbers of frags than read head would be bad of course
Some drink at the fountain of knowledge. Others just gargle.
POSIX compliance
Doesn't being case insensitive violate POSIX? Or has that been fixed?
yes
Don't call me back. Give me a call back. Bye. So yeah. But bye our, well, but alright we are on a shirt this chill.
Because the rest of the computing world is more interested in successfully interacting with itself and has realised that filesystem metadata is practically impossible to successfully move between systems using "common" tools.
Apple has finally figured this out, too, which is why they're moving away from it.
Filesystem metadata is another one of those cool ideas that is (or, rather, was) kneecapped because of lowest-common-denominator restrictions.
The summary appears to not be quite right.
To clarify, there are 2 separate file optimizations going on here.
The first is automatic file defragmentation. When a file is opened, if it is highly fragmented (8+ fragments) and under 20MB in size, it is defragmented. This works by just moving the file to a new, arbitrary, location. This only happens on Journaled HFS+ volumes.
The second is the "Adaptive Hot File Clustering". Over a period of days, the OS keeps track of files that are read frequently - these are files under 10MB, and which are never written to. At the end of each tracking cycle, the "hottest" files (the files that have been read the most times) are moved to a "hotband" on the disk - this is a part of the disk which is particularly fast given the physical disk characteristics (currently sized at 5MB per GB). "Cold" files are evicted to make room. As a side effect of being moved into the hotband, files are defragmented. Currently, AHFC only works on the boot volume, and only for Journaled HFS+ volumes over 10GB.
Nope:
Link