Measuring Fragmentation in HFS+

keyblob8K writes "Amit Singh takes a look at fragmentation in HFS+. The author provides numbers from his experiments on several HFS+ disks, and more interestingly he also provides the program he developed for this purpose. From his own limited testing, Apple's filesystem seems pretty solid in the fragmentation avoidance department. I gave hfsdebug a whirl on my 8-month-old iMac and the disk seems to be in good shape. I don't have much idea about ext2/3 or reiser, but I know that my NTFS disks are way more fragmented than this after similar amount of use."

Comments

Huh?

[Anonymous Coward]

but I know that my NTFS disks are way more fragmented than this after similar amount of use

Is this based off of instinct, actual data, or what?

Re:Huh?

[lpangelrob2]

Maybe he ran defrag in windows and measured how many bright blue blocks were next to the medium blue blocks and the dark blue blocks. :-)

HFS+ defrag source

[revscat]

As mentioned in the article, HFS+ does defragging on the fly when files are opened if they are less than 20MB. The source code for this is available here, as is a discussion about it that contains input from some Darwin developers.

Re:HFS+ defrag source

[Exitthree]

You've only defeated the purpose if you re-fragment the file again after opening it. If this isn't the case, the amortized cost (the initial cost of de-fragmentation when opening the first time minus the speed benefits from a file in a single chunk) over the many times the file is read yields a speed bonus, not a speed loss.

A good example is me, installing a program from disk onto my computer. I run the program and it accesses a group of files that have been fragmented when copied to my hard drive. The first time it opens the files it spends a little extra time de-fragmenting them. However, subsequent times that I open the program, these files will load faster.

Re:HFS+ defrag source

[MattHaffner]

Are you talking about the "Optimizing System" phase? As far as I know, that updates binary-library prebindings--not fragmentation. You can read more about it here:

http://developer.apple.com/documentation/Perform an ce/Conceptual/LaunchTime/Tasks/Prebinding.html

In theory, when you install anything (on any system) and have a reasonable amount of contiguous free space on your disk, the installed files should always be unfragmented since I believe that's what most file systems look for first to allocate: a large chunk of contiguous space.

Fragmentation typically occurs more when you open a file, increase its size, and write it back out. But operations that write large files to disk that do not know beforehand what the final size may also do this to some files that were only written once to your disk. For example, some of the largest fragmented files on my HFS+ volume are things snagged with BitTorrent. The fragments in these files are very regular chunks of blocks, which could be the typical 'buffer' size BT grabs when writing.

Re:HFS+ defrag source

[Daniel_Staal]

I believe the actual sequence is this:

1. Get request for file
2. Open File
3. Buffer file to memory
4. Answer request for file
5. If needed, defragment file

In other words, it defrangments after the file has been returned to the program needing it, as a background process. The buffer to memory is a pre-existing optimization, so the only real trade off is the background processor usage goes up. If you aren't doing major work at the time, you'll never notice. (And if you are doing major work, you probably are using files larger than 20MB in size anyway.)

Files larger than 20MB just aren't defragmented, unless you have another tool to do it.

Re:HFS+ defrag source

[shamino0]

HFS+ was one of the major features of the OS 8.1 update. OS 8.0 and earlier can't "see" HFS+ volumes- they see a tiny disk with a simpletext file titled "where have all my files gone?" which, if I remember correctly, gives a brief explanation that the disk is HFS+ and requires 8.1 or higher to view. :)

And the person who came up with this idea was a genius. This is far far better than what most other operating systems do (refuse to mount the volume.)

If I boot MS-DOS on a machine that has FAT-32 or NTFS volumes, I simply don't find any volume. I can't tell the difference between an unsupported file system and an unformatted partition. If the file system would create a FAT-compatible read-only stub (like HFS+ does), it would be much better for the user. Instead of thinking you have a corrupt drive, you'd know that there is a file system that your OS can't read.

NTFS is not so bad

[W2k]

It must be pretty damn good if it can outdo NTFS. I have three computers with WinXP (NTFS 5.1) that I run quite a bit of data through on a daily basis, and neither needs to be defragmented very often at all (two of them have never needed defragmentation in more than a year of use). Mind you, I might fall into some special category of people who don't fall victim to fragmentation for some reason. Anyway, my point is, before you make remarks regarding how well this compares to NTFS, and/or how much "Microsoft sucks", consider how well NTFS still holds up considering its age. Another bonus is, I don't risk losing file system integrity if there's a power failure. ;)

Re:NTFS is not so bad

[MemoryDragon]

Ntfs does not fragment that strongly as long as you dont hit the 90% full mark of your disk, once you reach that, see the files becoming fragmented in no time. NTFS uses the open space for write access and then probably relocates the files in time, once it hits 90% the open space usage algorithm does not seem to work anymore.

that's not a good measurement...

[Chuck Bucket]

it's not how fragmented your disk is, it's what you can do with your fragmented disk that counts.

CVB

Panther Defrag

[stang7423]

I'm sure someone else will point this out as well but its worth noting. In 10.3 there is kernel level defragmentation. When a file is accessed the kernel checks to see if its fragmented, then moves it to a area of the disk where it can exist unfragmented. I think there is a limitation to file size under 20MB but it may be higher. This still gets rid of a great deal of fragmenation. Just food for thought.

Centrifugal Force

[Prince Vegeta SSJ4]

I just put my hard drive in my drier when it is fragmented. Since the group of unfragmented bits weighs more than the fragmented ones, The spinning action causes all of those stray bits to attach to the greater mass.

Disk Fragmentation

[List of FAILURES]

A lot of people simply equate disk fragmentation with slow application execution and opening of data files. While this is the most visible effect that fragmentation has on a system, it's not the only one. If you are dealing with large files (multi track audio, video, databases) then you will get a different kind of performance hit due to the non-contiguous nature of the free space you are writing to. If you want to capture video with no dropouts, you really want a drive that has all of it's free space basically in one location. This allows you to write those large files with no physical disruption in location. Please do not think that the only benefit to unfragmented space is just "my programs launch faster". If you do any real kind of work on your system with large data files, you should know that a defragmented drive is a godsend.

How to determine fragmentation...

[Calibax]

This is a very arcane procedure in XP. I shall try to explain, but only a professional should attempt this.

1. Right click on drive icon, select properties
2. Select Tools tab and click on "Defragment Now"
3. Click on "Analyze"
4. When analysis finishes, click on "View Report"

This shows two list windows, one containing general properties of the disk such as volume size, free space, total fragmentation, file fragmentation and free space fragmentation. The second list shows all fragmented files and how badly they are fragmented.

Re:Defrag = placebo?

[Greyfox]

It shouldn't really be an issue post-FAT. I think most people's obsesison with fragementation are a remnant of having to defragment FAT drives regularly. One did it superstitiously in those days because an overly fragmented filesystem did slow down considerably. No modern filesystem has an excuse for not handling fragmentation with no interference from the user.

As a cute side note, I remember having to explain fragmentation to my high school FORTRAN class and teacher back in the '80's. I'd changed schools in my senior year and the new school had just started using the Apple II FORTRAN environment, which happened to be the same as the Aple II Pascal environment that I'd used at the previous school. The file system was incapable of slapping files into whatever blocks happened to be available (I'm not even sure it used blocks. Probably not...) so you would not be able to save your files if it was too fragmented, even if there was enough space available to do so. Ah those were the days...