Slashdot Mirror


Measuring Fragmentation in HFS+

keyblob8K writes "Amit Singh takes a look at fragmentation in HFS+. The author provides numbers from his experiments on several HFS+ disks, and more interestingly he also provides the program he developed for this purpose. From his own limited testing, Apple's filesystem seems pretty solid in the fragmentation avoidance department. I gave hfsdebug a whirl on my 8-month-old iMac and the disk seems to be in good shape. I don't have much idea about ext2/3 or reiser, but I know that my NTFS disks are way more fragmented than this after similar amount of use."

104 of 417 comments (clear)

  1. Huh? by Anonymous Coward · · Score: 5, Insightful

    but I know that my NTFS disks are way more fragmented than this after similar amount of use

    Is this based off of instinct, actual data, or what?

    1. Re:Huh? by lpangelrob2 · · Score: 5, Funny

      Maybe he ran defrag in windows and measured how many bright blue blocks were next to the medium blue blocks and the dark blue blocks. :-)

    2. Re:Huh? by Ann+Elk · · Score: 4, Informative

      My own experience, using a small tool I wrote to analyze NTFS fragmentation:

      NTFS is pretty good at avoiding fragmentation when creating new files if the size of the file is set before it is written. In other words, if the file is created, the EOF set, and then the file data is written, NTFS does a good job of finding a set of contiguous clusters for the file data.

      NTFS does a poor job of avoiding fragmentation for files written sequentially. Consider a file retrieved with wget. An empty file is created, then the contents are written sequentially as it is read from the net. Odds are, the file data will be scattered all over the disk.

      Here's a concrete example. Today, I downloaded Andrew Morton's 2.6.6-mm4.tar.bz2 patch set. (Yes, I run WinXP on my Toshiba laptop -- deal with it.) Anyway, the file is less than 2.5MB, but it is allocated in 19 separate fragments. I copied it to another file, and that file is unfragmented. Since the copy command sets EOF before writing the data, NTFS can try ot allocate a contiguous run of clusters.

      Note - This was done on uncompressed NTFS. My feeling is that compressed NTFS is even worse about fragmentation, but I don't have any numbers to back that up.

    3. Re:Huh? by bfg9000 · · Score: 4, Insightful

      (Yes, I run WinXP on my Toshiba laptop -- deal with it.)

      Why would anybody have a problem with you running Windows XP on your laptop? I'm a card-carrying Linux Zealot, and I don't have a problem with it.

      --

      I'm not normally an irrational zealous dickhead, but I figure "When in Rome..."

    4. Re:Huh? by EvilAlien · · Score: 4, Funny
      "'m a card-carrying Linux Zealot, and I don't have a problem with it."

      Apparently you are actually a closet Rational Linux Advocate. I'm sure there are a few people in the drooling horde reading these comments that will have a problem with someone being foolish enough to actually choose to run Windows on anything ;)

      I run Gentoo on my laptop, but the specs on the crusty old thing are so low that my only other "choice" would be the run Windows 95, and I'd sooner eat my usb key than do that.

      --
      perl -e 'print $i=pack(c5, (41*2), sqrt(7056), (unpack(c,H)-2), oct(115), 10)'
    5. Re:Huh? by bogie · · Score: 4, Interesting

      Actually the number of Windows users dwarfs the numbers of Linux users here these days. Sure Widows gets beatup on more because of the constant worms etc but have a look at the average "is linux ready for the desktop" thread. You get post after post of people critical of Linux on the desktop. At best some people will agree that linux is fine in some very specific situations. As I've said many times there is a reason why Slashdot won't show recent web browser statistics. My guess is it over 80% IE and not just because people are at work.

      For the record I also use XP on my laptop. Until everything works perfectly out of the box, ACPI and all, I'm not installing any nix on it.

      --
      If you wanna get rich, you know that payback is a bitch
    6. Re:Huh? by Rakefighter · · Score: 2, Informative

      Those are stange numbers...

      When you download a file with internet explorer, it downloads to a temporary directory, and then copies it to the location you selected in the "Save" dialog (using Windows copy facilities). According to your logic, the file that you downloaded should not be fragmented, at all.

      Care to explain yourself?

      --

      --Life may have no meaning, or, even worse, it may have a meaning of which you disapprove.

  2. HFS+ defrag source by revscat · · Score: 5, Informative

    As mentioned in the article, HFS+ does defragging on the fly when files are opened if they are less than 20MB. The source code for this is available here, as is a discussion about it that contains input from some Darwin developers.

    1. Re:HFS+ defrag source by rharder · · Score: 3, Interesting

      I still wish there was a reliable, preferrably included, defrag program. I do a lot of video work, and I not only want defragmented files, I want large contiguous sections of free space on my hard drive. I've not had much luck with 3rd party degrag programs (though I've yet to try SpeedDisk).

    2. Re:HFS+ defrag source by Joe5678 · · Score: 2, Insightful

      I guess it may take slightly longer to open a file, but that seems like it would be worth it in my opinion.

      That would seem to defeat the purpose to me. The main reason you want to avoid fragmentation of the data is that fragmented data takes longer to pull from the disk. So if by preventing fragmentation you slow down pulling data from the disk, you have just defeated your purpose.

    3. Re:HFS+ defrag source by shotfeel · · Score: 4, Interesting

      I thought this was a feature of Panther, not HFS+.

      HFS+ has been around since OS 8.5 (?? somewhere in OS 8). So either this is a feature of HFS+ that hasn't been implemented until now, or its a bit of code added to Panther. Or has HFS+ been updated?

    4. Re:HFS+ defrag source by Exitthree · · Score: 5, Interesting

      You've only defeated the purpose if you re-fragment the file again after opening it. If this isn't the case, the amortized cost (the initial cost of de-fragmentation when opening the first time minus the speed benefits from a file in a single chunk) over the many times the file is read yields a speed bonus, not a speed loss.

      A good example is me, installing a program from disk onto my computer. I run the program and it accesses a group of files that have been fragmented when copied to my hard drive. The first time it opens the files it spends a little extra time de-fragmenting them. However, subsequent times that I open the program, these files will load faster.

    5. Re:HFS+ defrag source by ericdano · · Score: 4, Informative
      Intech's Speedtools is a good set of utilities and includes a good defragmenter. For a complete defrag, something like Drive 10 or TechTool 4 work better.


      Good luck

      --
      It's either on the beat or off the beat, it's that easy.
      I moderate therefore I rule!
      --
    6. Re:HFS+ defrag source by ahknight · · Score: 4, Informative

      As stated in the article, this is a feature of the HFS+ code in Panther. The filesystem cannot have a defrag feature as the filesystem is just a specification. The implementation of that specification, however, can do most anything to it. :)

    7. Re:HFS+ defrag source by MattHaffner · · Score: 5, Insightful

      Are you talking about the "Optimizing System" phase? As far as I know, that updates binary-library prebindings--not fragmentation. You can read more about it here:

      http://developer.apple.com/documentation/Perform an ce/Conceptual/LaunchTime/Tasks/Prebinding.html

      In theory, when you install anything (on any system) and have a reasonable amount of contiguous free space on your disk, the installed files should always be unfragmented since I believe that's what most file systems look for first to allocate: a large chunk of contiguous space.

      Fragmentation typically occurs more when you open a file, increase its size, and write it back out. But operations that write large files to disk that do not know beforehand what the final size may also do this to some files that were only written once to your disk. For example, some of the largest fragmented files on my HFS+ volume are things snagged with BitTorrent. The fragments in these files are very regular chunks of blocks, which could be the typical 'buffer' size BT grabs when writing.

    8. Re:HFS+ defrag source by ahknight · · Score: 4, Insightful

      Last time I checked filesystems were also operatining system components. Often these components might be referred to as drivers.

      Then you didn't check hard. Again, HFS+ is a specification of how to write data to media in order to organize another collection of data. The implementation is what handles the defragging. There are no drivers involved as drivers are the software component of a hardware/software union and there is no hardware involved at this level (just logical organization).

    9. Re:HFS+ defrag source by Anonymous Coward · · Score: 4, Informative

      You just made his point. The DRIVER does the defragging. The HFS+ is a specification for how the files are laid out and written to the disk, such that a driver that understands this specification can read it. Linux has HFS+ drivers, but I doubt they defrag on the fly. Supposedly (though I don't know), Mac OS versions prior to 10.3 didn't defrag either.

      So therefore it might be a part of the operating system's filesystem. That's the system that deals with files. But that's not what was asked. What was asked was whether it was an inherent feature of HFS+, and that's not possible, since HFS+ doesn't tell the OS what to do when a file is opened, only how the stuff is stored on the disk.

      Perhaps you didn't understand the dual nature of the word filesystem: it can be the subsystem of the OS that handles files, or it can be the physical representation of the data on to the hard drive. If you assume it's only the first, your explanation makes sense. If you assume the second one (which would be the usage intended and understood by most people given the fact that the question and response were about HFS+ (physical filesystem) compared to Panther (OS filesystem)), then you'd be wrong.

      And I've been trolled, but who cares.

    10. Re:HFS+ defrag source by Daniel_Staal · · Score: 5, Informative

      I believe the actual sequence is this:

      1. Get request for file
      2. Open File
      3. Buffer file to memory
      4. Answer request for file
      5. If needed, defragment file

      In other words, it defrangments after the file has been returned to the program needing it, as a background process. The buffer to memory is a pre-existing optimization, so the only real trade off is the background processor usage goes up. If you aren't doing major work at the time, you'll never notice. (And if you are doing major work, you probably are using files larger than 20MB in size anyway.)

      Files larger than 20MB just aren't defragmented, unless you have another tool to do it.

      --
      'Sensible' is a curse word.
    11. Re:HFS+ defrag source by javax · · Score: 2, Informative

      hmmm... the advantage would be that your system wouldn't need to get defragmented every night you're asleep.

      The real advantage is that it will speed up things immediately for the 2nd time the file is read. Though you can still defrag your disk overnight if you like that.

    12. Re:HFS+ defrag source by solios · · Score: 2, Informative

      HFS+ was one of the major features of the OS 8.1 update. OS 8.0 and earlier can't "see" HFS+ volumes- they see a tiny disk with a simpletext file titled "where have all my files gone?" which, if I remember correctly, gives a brief explanation that the disk is HFS+ and requires 8.1 or higher to view. :)

      Journalling didn't show up until one of the Jaguar updates, where it could be enabled via the command line on clients and via disk utility on Server.

    13. Re:HFS+ defrag source by jimfrost · · Score: 3, Informative
      No, FFS does not do after-the-fact defragmentation. It attempts to allocate blocks that have low seek latency as files are extended. For the most part this avoids the problem entirely.

      If you ever wondered why there is a "soft limit" on FFS filesystems, the reason why is that its allocator's effectiveness breaks down at about the point where the filesystem is 90% full. So they sacrifice 10% of the filesystem space so that they can avoid fragmentation problems. It's not a bad tradeoff, particularly these days.

      I didn't know that HFS+ used an after-the-fact defragmentation system, but they've been around for awhile too. Significant research was done into such things as part of log-based filesystem research in the early 1990s (reference BSF LFS and Sprite). You had to have a "cleaner" process with those filesystems anyway (to pick up abandoned fragments of the log and return them to the free pool) so it made sense to have it also perform some optimization features.

      --
      jim frost
      jimf@frostbytes.com
    14. Re:HFS+ defrag source by shamino0 · · Score: 5, Informative
      HFS+ was one of the major features of the OS 8.1 update. OS 8.0 and earlier can't "see" HFS+ volumes- they see a tiny disk with a simpletext file titled "where have all my files gone?" which, if I remember correctly, gives a brief explanation that the disk is HFS+ and requires 8.1 or higher to view. :)

      And the person who came up with this idea was a genius. This is far far better than what most other operating systems do (refuse to mount the volume.)

      If I boot MS-DOS on a machine that has FAT-32 or NTFS volumes, I simply don't find any volume. I can't tell the difference between an unsupported file system and an unformatted partition. If the file system would create a FAT-compatible read-only stub (like HFS+ does), it would be much better for the user. Instead of thinking you have a corrupt drive, you'd know that there is a file system that your OS can't read.

    15. Re:HFS+ defrag source by moongha · · Score: 2, Interesting

      Case sensitivity is available on HFS+, it's just not the default.

      You can also use UFS.

    16. Re:HFS+ defrag source by dasmegabyte · · Score: 2, Insightful

      Which is why Apple is such a great company.

      At some companies, a developer would go to his project manager, propose this feature, and get a head shake. Too much work to test and spec, not worth the gains. Let's devote our time to our core competencies.

      Apple on the other hand was built on details like this. In fact, one of my favorite things about OS 10.3 is Expose...a feature nobody really asked for, and now I can't live without it (fuck virtual desktops...I want one desktop I can use!)

      --
      Hey freaks: now you're ju
  3. File allocation Table by SirChris · · Score: 4, Interesting

    what type of file system is there where there is no main allocation table just a header then the file then a header then the file so you could theoretically break a disk and still read the half that was good because all pertinent information relating to a file was in one place?

    1. Re:File allocation Table by jdunn14 · · Score: 2, Interesting

      In general, such a file system would have *horrible* performance, especially for writes, since it could require rearranging data on disk in order to allocate more space for a growing file. However, I would bet that some simple embedded filesystems do this sort of thing, especially if they are meant to be mainly read-only. Sorry, I can't just tell you the name of one, but you won't find such a filesystem made for regular desktop usage. Gonna be a special-purpose thing.

    2. Re:File allocation Table by SideshowBob · · Score: 4, Insightful

      That isn't a filesystem that is a tape. Any number of tape systems exist, pick whichever one you like.

    3. Re:File allocation Table by AKAImBatman · · Score: 4, Informative

      There are a couple things that you have to consider. For one, if part of the disk corrupts, how will you identify a header? Or for that matter, how would you identify the header space vs. file space in a non-corrupted file system?

      You're probably thinking "just store the size of the file", This is perfectly valid, but it does have certain implications. You see, in Comp-Sci, we refer to a list like this as a "linked list". The concept basically being that each item in the list has information (i.e. a "link") that helps identify the next item in the list. Such a data structure has a worst case access time of O(n). Or in other words, if your item is at the end of the list,and you have you have 2000 files, you'll have to check through all two thousand headers before finding your file.

      Popular file systems circumvent this by using what's called a Tree structure. A tree is similar to a linked list, but allows for multiple links that point to children of the node. A node that has no children is referred to as a "leaf node". In a file system the directories and files are nodes of a tree, with files being leaf nodes. This configuration gives us two performance characteristics that we must calculate for:

      1. The maximum number of children in a node.
      2. The maximum depth of the tree.

      Let's call them "c" for children and "d" for depth. Our performance formula is now O(c*d) and is irrespective of the number of items in the data structure. Let's make up and example to run this calculation against:

      Path: /usr/local/bin/mybinary

      Nodes:
      / (34) /usr (10) /usr/local (9) /usr/local/bin (72)

      Longest path: /usr/X11R6/include/X11

      Plugging the above numbers (72 for c, 4 for d) we get a worst case of 72*4 = 288 operations. Thus our worst case is much better than the linked list. And if we calculate the real case to access /usr/local/bin/mybinary, we get 34+10+9+72 = 134 operations.

      Hope this helps. :-)

  4. Re:Anonymous by o0zi · · Score: 3, Informative

    Ext2/3 and reiserfs both have inbuilt defragmentation capabilities. This can be seen, for instance, when you boot an ext2 system after an unclean shutdown and it checks the integrity of the filesystem. Ext3 and reiserfs are both journaling filesystems, which also helps with this problem. This is often strange for new Linux users, as they're used to worrying about their Windows NTFS/FAT32 filesystems. In Linux, it's just not necessary (nor in any Unix derivative such as AIX or BSD that uses those filesystems).

  5. Measuring fragmentation in NTFS by Anonymous Coward · · Score: 2, Informative

    Goto My Computer. Right click the drive to be analyzed. Select tools/defragment now.../Analyze.

    This was my PhD Thesis.

    1. Re:Measuring fragmentation in NTFS by moonbender · · Score: 4, Interesting

      I wrote a script some time ago to more easily let me check how badly my partitions are fragmented, here's it's current output:
      C: 5,72 GB Total, 1,97 GB (34%) Free, 4% Fragmented (8% file fragmentation)
      D: 40,00 GB Total, 1,00 GB (2%) Free, 41% Fragmented (82% file fragmentation)
      E: 66,69 GB Total, 105 MB (0%) Free, 10% Fragmented (21% file fragmentation)
      F: 30,00 GB Total, 1,21 GB (4%) Free, 3% Fragmented (7% file fragmentation)
      G: 10,00 GB Total, 1,54 GB (15%) Free, 5% Fragmented (9% file fragmentation)
      H: 35,03 GB Total, 551 MB (1%) Free, 39% Fragmented (79% file fragmentation)

      D ("Dump") and H ("Online") get a lot of throughput, by personal computing standards anyway, E ("Games") doesn't get changed that much, but if it does, a lot of data leaves and comes. Seems like whenever I defrag D or H, they're back to the values above within days. I guess Win XP has a hard time doing the internal on-the-fly defragging of the hard drives that rarely have moer than 1% free space... Guess I should just get a new HD and have some more free space that way - but I bet I'd have that filled up with junk after some weeks, anyway.

      That said, I'm not sure how relevant this is for NTFS partitions, anyway. I recall hearing that they aren't affected by fragmentation as much as FAT partitions (which were a nightmare), however I'm not sure if that means they don't fragment that easily (heh) or whether accessing data isn't slowed down as much by any existing fragmentation.

      I've also rarely heard anyone talking about fragmentation in the popular Linux file systems, a Unix partisan I know actually thought they didn't fragment full stop, which I don't believe is possible, at least not if you consider situations which might not occur in practice. But then again, I suppose Linux might solve it the same way Apple seems to - I guess I'll know more after a couple of hundred comments on this article. :)

      --
      Switch back to Slashdot's D1 system.
    2. Re:Measuring fragmentation in NTFS by Unregistered · · Score: 2, Insightful

      You need at least 10% free space for windows to keep the fs reasonably defragmented. put some larger hdds in there and it'll work much better

  6. NTFS is not so bad by W2k · · Score: 5, Interesting

    It must be pretty damn good if it can outdo NTFS. I have three computers with WinXP (NTFS 5.1) that I run quite a bit of data through on a daily basis, and neither needs to be defragmented very often at all (two of them have never needed defragmentation in more than a year of use). Mind you, I might fall into some special category of people who don't fall victim to fragmentation for some reason. Anyway, my point is, before you make remarks regarding how well this compares to NTFS, and/or how much "Microsoft sucks", consider how well NTFS still holds up considering its age. Another bonus is, I don't risk losing file system integrity if there's a power failure. ;)

    --
    Quality, performance, value; you get only two, and you don't always get to pick.
    1. Re:NTFS is not so bad by pjt33 · · Score: 2, Informative

      HFS+ is also journalled by default.

    2. Re:NTFS is not so bad by MemoryDragon · · Score: 5, Informative

      Ntfs does not fragment that strongly as long as you dont hit the 90% full mark of your disk, once you reach that, see the files becoming fragmented in no time. NTFS uses the open space for write access and then probably relocates the files in time, once it hits 90% the open space usage algorithm does not seem to work anymore.

    3. Re:NTFS is not so bad by Malc · · Score: 4, Interesting

      You must be too young to remember FAT-based systems. Lost clusters and cross-linked chains... oooh, didn't I enjoy them! NTFS has proven rock-solid and I haven't looked back since I jumped Win95 to WinNT4.

    4. Re:NTFS is not so bad by Daytona955i · · Score: 2, Insightful

      Do you actually check your drives? I just got a laptop with XP and I've been using it for less than a month now. After reading this thread I though well my computer is fairly new but I'll see how it looks anyway. After running Disk Defragmenter and clicking analyze I get:
      Analysis is complete for: (C:)
      You should defragment this volume.

      I then looked at the report and found the following:
      Total fragmentation = 21%
      File fragmentation = 42%
      Free space fragmentation = 1%

      Pretty bad especially considering I've only had the laptop for less than a month and that I still have over half the 40 gig drive free. Oh and HFS+ in panther is journaled so no loss of file system integrity there. So compared to HFS+ in panther, I'd say that NTFS in XP sucks! It would be interesting if someone has done similar research on ext3.

    5. Re:NTFS is not so bad by 13Echo · · Score: 2, Insightful

      NTFS isn't technically "as old" as you might think. Each version of NT over the past few years has added several upgrades to NTFS.

      http://en.wikipedia.org/wiki/NTFS

      NTFS has its strong points. It is reliable and has several extensions that make it quite flexible. On the other hand, it's not hard to "outdo NTFS" in some respects. There are many things that HFS+ and ReiserFS do better than NTFS. There are many things that NTFS does better.

      I think that NTFS is pretty good when it comes to cataloging changes to the drive. NTFS' speed leaves something to be disired though, and the nature of its fragmentation (though better than FAT32) still presents even more performance problems.

    6. Re:NTFS is not so bad by 222 · · Score: 4, Informative

      For proof, check out this. This drive was defragged about a week ago, and although it does go through heavy use, the current low disk space causes massive fragmentation.

    7. Re:NTFS is not so bad by Atomic+Frog · · Score: 3, Informative

      No, it doesn't take much to outdo NTFS.

      NTFS fragments _very_ fast on me, after a few months of use, it is in the 20% or more range.

      Same user (i.e. me), so same usage pattern, on my HPFS disks (yes, HPFS, that would be OS/2, not OS X), the fragmentation after 3 _years_ is less than 2% on ALL of my HPFS disks.

    8. Re:NTFS is not so bad by michael_cain · · Score: 2, Insightful
      You must be too young to remember FAT-based systems.

      Youngster. Go back far enough in UNIX and it required PERFECT disk packs to function -- no handling of bad sectors. Of course, those were the days when disk "drives" were the size of a small washing machine, the top opened, and you loaded/unloaded the multi-platter disk pack that was the size of a hat box. Was always interesting to see one of the gurus arrive to troubleshoot your system carrying their own disk pack with their specialized utilities... :^)

    9. Re:NTFS is not so bad by itwerx · · Score: 2, Insightful

      Pretty bad especially considering I've only had the laptop for less than a month...

      Any new machine will have an image dumped onto the hard-drive by the manufacturer.
      Most imaging apps don't bother with defragmenting so you probably started out with it fairly fragmented from the initial build of the image.

  7. HFS Filesystem vs. ReiserFS by dTaylorSingletary · · Score: 2, Interesting

    I've had a continued problem on my iBook for the past year or so.

    Under HFS+ in Mac OS X Jaguar or Panther, after about a day of having a clean install, fresh partition and format my hard drive starts making clunking noises and the system locks up (without actually freezing) -- then when reboot attempts are made they take aeons.

    Under ReiserFS in Gentoo Linux for PPC: never have the problem. Same hard drive. Months of use, never once hear the hard drive being funky. No lockups.

    Do I put the blame on HFS? OS X? I just can't figure out this strange problem.

    --
    d. Taylor Singletary,
    reality technician techra.el
    1. Re:HFS Filesystem vs. ReiserFS by sith · · Score: 2, Interesting

      Clunk clunk from a drive is the drive saying "I cry out in pain! Replace me!"

      Seriously - it's likely that gentoo just isn't using the particular sector on the drive that OSX is - perhaps there is a file there that doesn't get accessed regularly or something. In any case, Clunk-Clunk is never ok.

    2. Re:HFS Filesystem vs. ReiserFS by Daniel+Dvorkin · · Score: 2, Insightful

      What everybody else said. Back up your data and replace that drive NOW.

      --
      The correlation between ignorance of statistics and using "correlation is not causation" as an argument is close to 1.
    3. Re:HFS Filesystem vs. ReiserFS by sterwill · · Score: 2, Informative

      Grab smartmontools and run them on your drive (like "smartctl -a /dev/hda" or similar). Most SCSI and most newer ATA drives will maintain a SMART error log of any defects/problems. smartmontools will also print drive attributes (for most drives) that can tell you when a drive is about to fail, before it actually does.

  8. My stats by Twirlip+of+the+Mists · · Score: 4, Informative
    I throw these out there for no real reason but the common interest.

    I've got a G4 with an 80 GB root drive which I use all day, every day. Well, almost. It's never had anything done to it, filesystem-maintenance-wise, since I last did an OS upgrade last fall, about eight months ago.
    Out of 319507 non-zero data forks total, 317386 (99.34 %) have no fragmentation.
    Not too shabby, methinks.
    --

    I write in my journal
  9. Re:Give it a rest by chasingporsches · · Score: 3, Informative

    i believe the topic at hand is fragmentation, not how well it works as a filesystem. in that regard, FAT32 and NTFS do have horrible problems with fragmentation, while HFS+ defragments on the fly.

  10. Re:Anonymous by Malc · · Score: 4, Interesting

    To be honest, I don't worry too much about NTFS. It seems reliable and for my needs performs satisfactorily. I wonder if fragmentation was an issue overly-promoted by companies like Symantec. What does it mean to have a fragmented disk at logical level when the physical layout is hidden by the drive electronics, which could in fact be translating sectors? Do these defragmentation tools take things like the number of platters in to consideration, and can neighbouring sectors be assumed to be in the same cylinder across platters?

  11. Re:Give it a rest by marktoml · · Score: 3, Insightful

    Agreed and the fragmentation on NTFS can have subtle effects (such as fragmenting the MFT) that are NOT easily fixed by simply running a defragmentation tool.

  12. Re:Anonymous by operagost · · Score: 2, Interesting

    NTFS is a modern journaling filesystem. There is no reason for concern while using it over other journaling file systems.

    --

    Gamingmuseum.com: Give your 3D accelerator a rest.
  13. that's not a good measurement... by Chuck+Bucket · · Score: 5, Funny

    it's not how fragmented your disk is, it's what you can do with your fragmented disk that counts.

    CVB

  14. Panther Defrag by stang7423 · · Score: 5, Interesting

    I'm sure someone else will point this out as well but its worth noting. In 10.3 there is kernel level defragmentation. When a file is accessed the kernel checks to see if its fragmented, then moves it to a area of the disk where it can exist unfragmented. I think there is a limitation to file size under 20MB but it may be higher. This still gets rid of a great deal of fragmenation. Just food for thought.

    1. Re:Panther Defrag by aristotle-dude · · Score: 2, Informative

      Have you heard of kernel extensions aka kernel modules? Drivers on OSX end with a .kext which denotes a kernel extension.

      --
      Jesus was a compassionate social conservative who called individuals to sin no more.
  15. HPFS by gmuslera · · Score: 2, Interesting
    When i had OS/2, i enjoyed the low hpfs fragmentation. When you copy a file it gives to it the next free block that fit in that size, as long you have a big enough free chunk of the disk, the file were not fragmentated. But also it unfragmented when more operations where done with the directory or the file system. I remember that a basic "unfragment" script was to go thru all directories and just copy or even rename the files to unfragment them.

    But not sure how this are managed in linux filesystems, not just ext2/3 and reiserfs, but also in xfs and jfs.

  16. Who even cares about Fragmentation anymore? by greymond · · Score: 4, Interesting

    Seriously, with NTFS and HFS+ I see very little fragmentation on both my Wintel and Apple machines.

    Both have 40gig HD's and both have applications installed/uninstalled quite often. My PC feels the worst of this as he gets games installed and uninstalled in addition to the apps.

    For example the last time I reinstalled either of these machines was back in january(new year fresh install) and since then my pc has felt the install/uninstal of various games usually ranging from 2-5 gigs each. The Apple has been installed and with the exception of updates, plugins, video codecs and basic small apps that get added/upgraded often has done alright.

    Right now Norton System Works on my PC is saying the drive is 4% fragmented. Disk Warrior on my Apple is saying the drive is 2% fragmented.

    Conclusion: Fragmentation is no longer an issue for the HOME USER(note how i'm not saying your companies network doesn't need to be concerned), unless there still running a FAT32 partition >. which well they deserve to have there computer explode at that point anyway.

  17. Bzzt! Nope. Close, though! by danaris · · Score: 3, Informative

    That's not quite correct. In Panther (Mac OS X 10.3, for the uninitiated), journaling is enabled by default: that is, when you first install Panther, it will add journaling to your existing HFS+ disk, and if you're reformatting, it will default to HFS+ (Journaled). However, prior to Panther, there was no journaling support in HFS+, to my knowledge.

    Dan Aris

    --
    Fun. Free. Online. RPG. BattleMaster.
  18. Re:Anonymous by EsbenMoseHansen · · Score: 3, Insightful

    The main problem with fragmentation is cache-faults. The disk drives assume that you will be reading the following sector;: when you don't, you'll have to wait for the sector you requested to be brought in from disk. This applies even in the face of the tricks you mention.

    --
    Religion is regarded by the common people as true, by the wise as false, and by rulers as useful.
  19. Centrifugal Force by Prince+Vegeta+SSJ4 · · Score: 5, Funny

    I just put my hard drive in my drier when it is fragmented. Since the group of unfragmented bits weighs more than the fragmented ones, The spinning action causes all of those stray bits to attach to the greater mass.

  20. Defrag = placebo? by justforaday · · Score: 3, Interesting

    I've often wondered if defragging and defrag utils are more of a placebo for people concernced with system performance. In my experience I've never noticed any perceivable difference after using a defrag util, on either OS8/9, OSX, 95, 98SE or XP. Then again, I've always made sure to have plenty of free space on my disks [and made sure they were fast enough] whenever I've done anything seriously disk intensive like multitrack audio...

    --
    I'll turn into a supernova and burn up everything. Well I'll turn into a black little hole and you'll turn into string.
    1. Re:Defrag = placebo? by Greyfox · · Score: 5, Interesting
      It shouldn't really be an issue post-FAT. I think most people's obsesison with fragementation are a remnant of having to defragment FAT drives regularly. One did it superstitiously in those days because an overly fragmented filesystem did slow down considerably. No modern filesystem has an excuse for not handling fragmentation with no interference from the user.

      As a cute side note, I remember having to explain fragmentation to my high school FORTRAN class and teacher back in the '80's. I'd changed schools in my senior year and the new school had just started using the Apple II FORTRAN environment, which happened to be the same as the Aple II Pascal environment that I'd used at the previous school. The file system was incapable of slapping files into whatever blocks happened to be available (I'm not even sure it used blocks. Probably not...) so you would not be able to save your files if it was too fragmented, even if there was enough space available to do so. Ah those were the days...

      --

      I'm trying to teach myself to set people on fire with my mind... Is it hot in here?

    2. Re:Defrag = placebo? by Woody77 · · Score: 2, Insightful

      I'm a software developer (C++, mostly msdev). As such, I recompile a LOT. With a drive dedicated to mostly source and intermediate files, about 6GB of a 9GB drive, it regularly fragments itself into molasses. It normally takes a while, but it happens. The continual replacement of files just dries NTFS up a wall. This is using both win2k and xp. I end up needed to defrag about once a month.

      The problem really rears it's head once the space between the files isn't big enough for the new files, and then things start getting fragmented in a hurry.

      And does it make a difference? In disk-intensive things like compiling (lots of small modules that get compiled into big binaries/debug symbol files), it really has an impact. The more the heads have to move around, the slower it all gets.

      I also use lots of partitions to isolate things, so that even as it fragments, the fragments don't get too far apart. However, if you do a lot of swapping to disk, the partitions will kill you, unless the swap is on the same partition as where you're working. So I tend to foist the swap off on another drive entirely (all SCSI) so that the seeking is reduced as much as possible.

      Placebo? probably is for most people that aren't continually writing/rewriting to disk. But if you're constantly reading/writing/erasing files, it is useful.

      Now if I could just figure out a way to safely split up my root filesystem on linux to keep the heavily used trees separate from the not-so-heavily used ones (performance experiment). Mainly my /var/tmp/*, which gets used heavily for compiling (gentoo and portage).

    3. Re:Defrag = placebo? by jimfrost · · Score: 2, Informative
      Consider an average eide hard drive: 10ms seek rate, ata100 interface.

      If you need to save a 100kb file, it will take 10ms (1/100th of a second) to seek to the first block, and then, assuming everything is perfect, it will take 100MB/sec / 100kb = 1/1000th of a second to write the file... so, seeking to the start of the file took 10 times as long as writing it!

      This gross simplification actually trivializes the real effect. The 10ms seek figure is an average track-to-track seek delay between adjacent tracks. The farther apart the tracks are the longer a seek takes (it's more or less linear although there is a per-seek overhead). You also don't deal with the fact that you're going to have to perform seeks on larger files no matter what.

      I note that there is a similar latency issue with head switches.

      There is a big difference between the delay necessary to pull a sector off of a track adjacent to where the heads currently are and one 1000 tracks away -- the delay can be an appreciable fraction of a second just for that single seek.

      The problem with simplistic filesystem block allocators is they do not weight their block allocations according to seek time. Usually they just pick whatever's first in the free list. This results, over time, to random block placement and therefore seek times that will on average approach 50% of worst case. I'd have to look closely to give a good figure with today's drives, but order-of-magnitude degredation is certainly possible. What you would prefer is blocks placed such that they're within a few percent of best case if at all possible.

      This is not hard to do, and to my knowledge BSD FFS was the first to attempt it -- and it was wildly successful at it.

      As an aside, I've never seen a fragmentation analysis program that took noncontiguous-but-well-placed into account. It's entirely possible to create a block layout that those programs think is awful that is within 90% of optimum. I think, actually, that BSD FFS would typically show up that way although I never investigated.

      I also note that smart block allocation makes a defragmenter's job a heck of a lot easier.

      --
      jim frost
      jimf@frostbytes.com
    4. Re:Defrag = placebo? by MyHair · · Score: 2, Informative
      It shouldn't really be an issue post-FAT. I think most people's obsesison with fragementation are a remnant of having to defragment FAT drives regularly. One did it superstitiously in those days because an overly fragmented filesystem did slow down considerably. No modern filesystem has an excuse for not handling fragmentation with no interference from the user.

      Head seek and rotational latency is still much slower than contiguous blocks. True, modern systems deal with it better, partially due to b-tree and other file indexing strategies and partially due to having plenty of RAM for metadata caching and predictive caching. But fragmentation is still a major issue for me on multiuser Windows PCs and periodic disk cleanup and defragmentation is necessary for reasonable operation speed.

      <MS gripe>
      In particular, the hidden "Content.IE5" cache of IE on 20-100 user PCs fills up hard drives in a big hurry, and I haven't found a way of controlling this except for periodically deleting with the following batch file I made for Win2k. (Limiting the cache size doesn't seem to affect these files.)
      @echo off
      echo.
      echo ***** WARNING!!!!! This will wipe out
      echo ***** C:\Documents and Settings\*\Local Settings\Temporary Internet Files contents.
      echo ***** Press Control-C to abort or any other key to delete all temp files.
      echo.
      pause
      for /D %%x in (c:\"Documents and Settings"\*) do rd /s /q "%%x\Local Settings\Temporary Internet Files\Content.ie5"
      This needs to be done preventatively, though. In addition to fragmenting data, 20-100 user PCs with large numbes of files (and Content.IE5 is my killer in my situation) fill up the MFT and then fragment it, and once you get the MFT fragmented you're basically screwed.
      </MS gripe>

      Admittedly the biggest problem PCs have disks less than 12G and I don't have as much of a problem with 20G+ systems. But have you ever run defrag after a clean install, even with an enormous hard drive? You'd think it could at least install itself without severe fragmentation. Oh well.
  21. Disk Fragmentation by List+of+FAILURES · · Score: 5, Insightful

    A lot of people simply equate disk fragmentation with slow application execution and opening of data files. While this is the most visible effect that fragmentation has on a system, it's not the only one. If you are dealing with large files (multi track audio, video, databases) then you will get a different kind of performance hit due to the non-contiguous nature of the free space you are writing to. If you want to capture video with no dropouts, you really want a drive that has all of it's free space basically in one location. This allows you to write those large files with no physical disruption in location. Please do not think that the only benefit to unfragmented space is just "my programs launch faster". If you do any real kind of work on your system with large data files, you should know that a defragmented drive is a godsend.

  22. sysinternals.com by FatSean · · Score: 2, Insightful

    I have a program from there that at startup will check the MFT, swapfiles and other important files and will make each one contiguous collection of disk blocks. Gotta be done then, as you can't lock them once Windows is completely up.

    --
    Blar.
  23. On-the-fly defragmentation for NTFS disks in WinNT by exwhyze · · Score: 3, Informative

    Buzzsaw and Dirms -- I admit, the site looks a little seedy, but I've used both of these programs on several machines for upwards of a year and they've done a superb job of keeping my NTFS disks defragmented.

  24. How to determine fragmentation... by Calibax · · Score: 5, Funny

    This is a very arcane procedure in XP. I shall try to explain, but only a professional should attempt this.

    1. Right click on drive icon, select properties
    2. Select Tools tab and click on "Defragment Now"
    3. Click on "Analyze"
    4. When analysis finishes, click on "View Report"

    This shows two list windows, one containing general properties of the disk such as volume size, free space, total fragmentation, file fragmentation and free space fragmentation. The second list shows all fragmented files and how badly they are fragmented.

    1. Re:How to determine fragmentation... by spectecjr · · Score: 4, Insightful

      This is a very arcane procedure in XP. I shall try to explain, but only a professional should attempt this.

      1. Right click on drive icon, select properties
      2. Select Tools tab and click on "Defragment Now"
      3. Click on "Analyze"
      4. When analysis finishes, click on "View Report"

      This shows two list windows, one containing general properties of the disk such as volume size, free space, total fragmentation, file fragmentation and free space fragmentation. The second list shows all fragmented files and how badly they are fragmented.


      If you're not using the same tool to measure fragmentation on each OS, how do you know that they're using the same semantics to decide what a fragmented file is?

      IIRC, the Linux tools use a different metric to calculate fragmentation than the NT ones.

      --
      Coming soon - pyrogyra
    2. Re:How to determine fragmentation... by pantherace · · Score: 4, Informative
      Well, all modern operating systems handle it so that any program, except certain tools such as the defragmenter, which either look at it directly, or use a lower level call.

      NTFS is horrible. on a system installed less than a week ago, and a few programs (nwn, firefox, avg, itunes, aa, nvdvd, windows updates, and a couple more programs, it has 9.3GB used, and it is reported that it has "Total Fragmentation: 22%, File Fragmentation: 45%"

      So yes there are various methods of calculating file fragmentation. (2 I can think of: (# of files with fragments)/(total number of files) = 0 for a totally defragemented hd (& gives nice percentages) & (# of file fragments)/(total number of files) = 1 for a perfectly defragmented hd. or variations on those, and I haven't been able to find what calculations Windows, & e2fstools use, so I can't tell.

    3. Re:How to determine fragmentation... by ProfessionalCookie · · Score: 3, Funny

      What's a "Right Click"?????

      -Faithful Macuser
      (ok I have a 3 button logitech)

    4. Re:How to determine fragmentation... by theguitarizt · · Score: 2, Funny

      the opposite of a wrong click.

    5. Re:How to determine fragmentation... by spectecjr · · Score: 3, Informative

      How many ways are there do define fragmented files? If I can read the file starting from the byte address of the first byte of the file sequentially all the way to the EOF, it isn't fragmented. Otherwise, every time I have to jump to a non-sequential byte address, that's another fragment. Am I missing something?

      As an example, look up the docs on ext2. Note that file fragments are not necessarily the same as fragmented files. Also note that people use the "file fragment" number as an indicator of how fragmented their ext2 partition is - which is wrong.

      --
      Coming soon - pyrogyra
    6. Re:How to determine fragmentation... by swilver · · Score: 3, Interesting
      I've managed to get my download drive (NTFS) so fragmented that the defrag tool in Win2k/XP is unable to defragment it, no matter how often you run it.

      The files on the drive had an average size of 200 MB, were downloaded in 1kB increments several files at a time over a period of a week on average per file.

      The reason for it failing on defraging (it doesn't say it fails, it just doesn't do much and stops after a while) is because the free space was also so badly fragmented that it couldn't even defragment a single file (it expects there is free space in a single chunk large enough to defragment one file... which is hard since the files are all quite large). The worst fragmented files had all well over 5000 fragments, and none less than 100 orso.

      Even after making more than 50% space free on the drive, it was still suffering from fragmented free space and refused to defrag. After a while I decided it was far easier to just copy the whole drive (30 GB) and copy it back... writing the 30 GB back was about 20-30 times faster compared to reading the drive for the initial copy...

      On a side note, I've put the exact same stress on a ext3fs disc, and it also fragmented very badly, despite claims that ext3fs doesn't need defragmentation "because it won't fragment your files much". If free space was also very fragmented, I cannot say, but I assume it was as deleting a very fragmented file would result in a very fragmented area of free space.

      --Swilver

  25. Re:Bzzt! Nope. Close, though! by Steveftoth · · Score: 4, Informative

    Jaguar (10.2) has journaled support as well, but you had to enable it as it was not a default option.

    Even in 10.3 it's optional, not required, but it's the new default for new disks. Probably because Apple decided that their code was solid enough to put into production. After testing it on 10.2 I agree with them.

  26. Re:Big frag issues under EXT2 too by 42forty-two42 · · Score: 4, Informative
    Manually run e2fsck it'll tell you how fragmented it is, as in:
    $ e2fsck -f -n knoppix.img
    knoppix.img: 453/7680 files (3.1% non-contiguous), 12180/30720 blocks
  27. Apple updated their stand recently by djupedal · · Score: 4, Informative

    http://docs.info.apple.com/article.html?artnum=256 68

    Mac OS X: About Disk Optimization

    Do I need to optimize?

    You probably won't need to optimize at all if you use Mac OS X. Here's why:

  28. Re:Anonymous by muck1969 · · Score: 3, Informative

    The most significant display I've ever seen for the benefits of defragmentation was on a 386 box that had Win 3.11 in 1992. The boot time was cut from two minutes down to 40 seconds and response was very noticeable. I didn't defrag due to any outside encouragement; I happened to find the utility in some drawer on a job site and gave it a try.

    Fragmentation is a performance killer for Win 9x on older machines ... presuming that Win 9x actually performs.

    --
    m.mmm..myyy ... sssissxxxtthh bbboottle offf mmmmmoouunnnttain ddeeewww.. in thhe pppassst ffffif
  29. Defragging XP now... by PhilHibbs · · Score: 2, Interesting

    My new laptop, with a 60GB 7200RPM disk, is under two months old, and I'm defragmenting it now. It's been running for 5 minutes, and is 3% complete, on a disk that is 62% full.

    20 minutes later, and it's on 17%. That's pretty damn fragmented, in my opinion.

    1. Re:Defragging XP now... by ewhac · · Score: 3, Insightful

      No, it's just that the defragger built-in to Win2K/XP is shite. Its runs like molasses in liquid helium, and it almost never does a complete job in a single run. You have to run it several times in a row before it's even close to doing a reasonable job. And if it's your system drive, then there are some files (including the swap file) that it simply won't touch no matter how badly the blocks are scattered. This can be a real pain in the posterior if you're trying to defrag a drive in preparation for a Linux install.

      Schwab

    2. Re:Defragging XP now... by ewhac · · Score: 3, Informative
      How safe is [resizing an NTFS partition] anyways?

      With the latest versions of ntfsresize, fairly safe. I did it on a machine at work with very important data on it (yes, I backed it up first), and had no trouble at all. However, all ntfsresize can do is truncate an NTFS partition's free space. In other words, it won't relocate blocks to other free areas of the disk. So the most you can shrink it is by however much free space you have at the end of the partition. ((After Googling around a bit, I've learned that the most recent versions of ntfsresize will now move datablocks around, so apparently that restriction is now gone. I have not personally tested this, however.))

      Incidentally, ntfsresize is part of Knoppix, and gets run through QTPartEd, a partition editing tool. It is an older, non-relocating version, however.

      Schwab

  30. my stats by MyDixieWrecked · · Score: 2, Interesting
    Out of 232167 non-zero data forks total, 231181 (99.58 %) have no fragmentation.
    Out of 6528 non-zero resource forks total, 6478 (99.23 %) have no fragmentation.

    Not bad. That's 8 months of heavy use since my last format.

    I gotta bring this to work today and see what that machine's like. My co-worker has been complaining that he doesn't have a defrag utility since he got OSX. I've been telling him that I don't think it matters. Now I can prove it to him.

    I remember back in the days of my Powermac 8100/80av, we would leave the 2 800mb drives defragging over the weekend because they had like 75% fragmentation.

    --



    ...spike
    Ewwwwww, coconut...
  31. Re:Anonymous by flynn_nrg · · Score: 3, Informative

    What are you talking about?

    Ext2/3 and reiserfs both have inbuilt defragmentation capabilities.

    No, they don't. But since they borrow their design from BSD's FFS they don't need it either.

    This can be seen, for instance, when you boot an ext2 system after an unclean shutdown and it checks the integrity of the filesystem. On journaled filesystems, the log is replayed. IBM's jfs also runs a modified fsck.

    Erm, that's fsck. fsck doesn't do defragmentation.

    In Linux, it's just not necessary (nor in any Unix derivative such as AIX or BSD that uses those filesystems).

    It's true, however performance is severely degraded when disk usage reaches around 90% for classic FFS-like filesystems. While the BSDs can mount ext2 partitions none of them uses ext[23] as default. AIX uses a JFS version that's a bit different from the one you see in Linux, which was based on OS/2's code. I think you're mixing up filesystem integrity with fragmentation. In classic BSD UFS/FFS data is stored in datablocks, which are partitioned in fragments, usually 1/4th of the datablock size. A fragmented file is a file that's stored in non-contiguous fragments. Just that. The performance impact of fragmented files vs the time needed to reorganize the data shows that it's not worth running a defrag program on FFS filesystems.

    This paper has some more info on the subject.

  32. Oh please. by warrax_666 · · Score: 2, Insightful
    Which means this isn't a valid test.

    It's a perfectly valid test -- it measures how much fragmentation can be observed after a certain amount of use. According to your logic we couldn't compare any properties of NTFS/ReiserFS/FAT32/HFS+ because they work differently.
    --
    HAND.
  33. Re:Defrag = placebo? - yes and no by NeedleSurfer · · Score: 4, Interesting

    Yes and no, it won't have any long time effect on your performance but there is a short time effect that can be usefull when dealing with audio. On a Mac, using a drive with block sizes of 64K to 256K (ideal when dealing with digital audio, as long as you set the buffer per track size of your daw to the same size as the blocks on your drive) you can gain up to 8 tracks by defraging your drive. Sometimes on large projects I have to record a file or playback the entire session in edit mode (no tracks frozen, everything real-time and not bounced), after editing for a while the daw refuses to play the project, lags, stutter or present some serious drop-outs, I defrag and this is where I get this 6-8 tracks headroom, but that will last only for a day of work and even then (Pro-Tools, Nuendo, Cubase, MOTU DP all present this caracteristic, as for the other I haven't tested them enough to provide meaningfull data).

    however, defraging is not the same for every defrag utility. For example, I was working with Avid Audiovision about 5-6 years ago on a TV show, it seems that defraging a drive hosting files created or edited with Audiovision with Speed Disk by Symantec would actually corrupt the entire projects contained on the drive (the biggest mistake and the only serious one I had in my career, I didn't loose my job but my boss did loose his temper, live and learn!), audio file were not readable at all after, it was actually a documented bug of Audiovision and I even think it was affecting every OMF files not just the ones used by Audiovision (not sure about this though), thats what happens when your boss won't let you RTFM. Only Disk Express, some Avid defrager or, later, Techtool could defrag those drives.

    On a side note, in the Classic mac (7-9.2), defragmenting your drive was also a way to prevent data corruption, actually its the other way around, not defraging would lead to data corruption. I don't know if its also the case with NTFS, EXT2 et al.

  34. I rarely see XP drives w/ bad fragmentation probs by Larry_Dillon · · Score: 4, Funny

    Mostly because they end up re-installing the OS every year or so!

    --
    Competition Good, Monopoly Bad.
  35. Re:Anonymous by EsbenMoseHansen · · Score: 2, Informative

    The cache on the disk. And while we do not REALLY know, the described behaviour is a common and proven strategy. Remember that the disk does not have a lot of information to go by; it basically just sees request to read individual sectors. More or less.

    --
    Religion is regarded by the common people as true, by the wise as false, and by rulers as useful.
  36. portable fragmenter by harlows_monkeys · · Score: 2, Interesting
    Here's how you can write a portable fragmenter, if you need to get a disk very fragmented for testing.
    1. Create small files until you run out of disk space.
    2. Pick a few thousand of the small files at random, and delete them.
    3. Create a large file, large enough to fill the free space.
    4. Go back to #2, unless you are out of small files.
    5. Pick one of the large files, and delete it.

    Result: you have a bunch of large files, all very fragmented, and the free space is very fragmented.

  37. Write your own defragmentation tool? by multiOSfreak · · Score: 2, Funny
    from the article:

    If you are really bent upon defragmenting your HFS+ volume, a more appropriate way would be to write your own defragmentation tool.


    Does anyone else think that statement is a bit odd? Maybe it's just me, but I think he's being a little bit presumptuous about the programming skills of the average geek site browser.

  38. HPFS fragments could be good by wardk · · Score: 2, Informative

    My reccollection of the OS/2 HPFS file system from IBM was that in many cases it would purposely fragment to take advantage of the disk spin, thus using fragmentation to increase performance.

    Defrag utils for OS/2 had options to only defrag if there were more than 3 extents, to avoid nullifying this effect.

    funny, years after the death of OS/2, it still kicks ass on much what we use now.

    1. Re:HPFS fragments could be good by gerardrj · · Score: 2, Informative

      There used to be several disk access optimizations

      Vendors used to do interleaving with the format/fdisk commands I recall. The idea was that writing the sectors in a continuous stream was not very efficient as the drives of the time could not move data to or from the disk so quickly. You'd read sector 1, and by the time you were ready to read sector two, sector 3 was under the head, so you had to wait almost an entire disk revolution to find sector 2 again.
      The interleave told the OS to skip X physical disk sectors for each 1 logical sector.

      For example, assume a disk with 12 sectors on a track such that when stationary the disk's sectors align with the hours on a clock face. With interleave of 3 the OS would put sector 1 at 1:00, sector 2 at 4:00, sector 3 at 7:00, sector 4 at 10:00, sector 5 at 2:00, and so on. The OS would occasionally skip more than the "interleave" number of sectors in order to not overwrite previous sectors. This meant that by the time that logical sector 1 was read and transferred to the computer, logical sector 2 would just about be under the heads for reading, thus eliminating or at least minimizing the rotational latency.

      Another big advantage was placing the directory structures in the middle tracks of the drive. This minimized the longest seek that would have to be performed. Unless a single file was very large or in just the wrong spot, it would usually be positioned completely on the inside or outside half of the tracks. After reading, the head only had to move at most half way across the disk to locate the next file or cluster/fragment to read or write; the again at most 1/2 the disk to perform the next operation.
      Most of today's file systems start placing directory/catalog information at the start of the disk, this effectively doubling average seek times to the data stored on the disk.

      As others mentioned, on some "faster" drives, there were filesystems that essentially treated the platters in a drive as individual units and managed then like a RAID 0, a RAIP so to speak (Redundant Array of Independent Platters).

      File fragmentation in today's fast, large buffer drives is, I think, the least of our worries. Fragmented or not we need more optimization of data structures on the drive. I'd rather have related files fragmented and near-by each other than contiguous and spread evenly across the drive.

      --
      Article X: The powers not delegated... by the Constitution...are reserved...to the people
  39. Microsoft really pisses me off by jimfrost · · Score: 4, Interesting
    Ok, we have this filesystem fragmentation buggaboo that's been plaguing MS-DOS and Windows for more than twenty years. We've got a whole industry built around building tools to fix it.

    That would be well and good if the problem were otherwise insurmountable. But, it turns out, we've known how to minimize, if not entirely eliminate, filesystem fragmentation for twenty years now - since the introduction of the BSD Fast File System.

    It doesn't take expensive (in time, if not in money) tools. All it takes is a moderately clever block allocation algorithm - one that tries to allocate a block close in seek time to the previous one, rather than just picking one at random.

    The fundamental insight that the authors of FFS had was that while there may only be one "optimal" block to pick for the next one in a file, there are tens of blocks that are "almost optimal" and hundreds that are "pretty darn good." This is because a filesystem is not a long linear row of storage bins, one after another, as it is treated by many simplistic filesystems. The bins are stacked on top of each other, and beside each other. While the bin right next to you might be "best", the one right next to that, or in another row beside the one you're on, or in another row above or below, is almost as good.

    The BSD folk decided to group nearby bins into collections and try to allocate from within collections. This organization is known as "cylinder groups" because of the appearance of the group on the disk as a cylinder. Free blocks are managed within cylinder groups rather than across the whole disk.

    It's a trivial concept, but very effective; fragmentation related delays on FFS systems are typically within 10% of optimum.

    This kind of effectiveness is, unfortunately, difficult to achieve when the geometry of the disk is unknown -- and with many modern disk systems the actual disk geometry is falsely reported (usually to work around limits or bugs in older controller software). There has been some research into auto-detecting geometry but an acceptable alternative is to simply group some number of adjacent blocks into an allocation cluster. In any case, many modern filesystems do something like this to minimize fragmentation-related latency.

    The gist of this is that Microsoft could have dramatically reduced the tendency towards fragmentation of any or all of their filesystems by doing nothing else but dropping in an improved block allocator, and done so with 100% backward compatibility (since there is no change to the on-disk format).

    Maybe it was reasonable for them to not bother to so extravagantly waste a few days of their developers' time with MS-DOS and FAT, seeing as they only milked that without significant improvement for eight or nine years, but it's hard to explain the omission when it came to Windows NT. NTFS is a derivative of HPFS which is a derivative of FFS. They had to have known about cylinder group optimizations.

    So the fact that, in 2004, we're still seeing problems with filesystem fragmentation absolutely pisses me off. There's no reason for it, and Microsoft in particular ought to be ashamed of themselves. It's ridiculous that I have to go and degragment my WinXP box every few months (which takes like 18 hours) when the FreeBSD box in the basement continues to run like a well-oiled machine despite the fact that it works with small files 24 hours a day, 365 days a year.

    Hey Microsoft: You guys have like fifty billion bucks in the bank (well, ok, 46 or 47 billion after all the antitrust suits) and yet you can't even duplicate the efforts of some hippy Berkeleyite some twenty years after the fact? What's up with that?

    (I mean "hippy Berkeleyite" in an affectionate way, Kirk. :-)

    --
    jim frost
    jimf@frostbytes.com
  40. Re:ReiserFS and fragmentation by Bombcar · · Score: 2, Funny

    parted? I thought that was only used to fiddle with FAT...

    Parted

  41. Re:fragmentation and dimension by jimfrost · · Score: 2, Informative

    A drive is not one dimensional, it is three dimensional: Rotation, platter, track. It is this geometry that BSD FFS takes advantage of to avoid large fragmentation-related delays, since while there may only be one "optimal" sector to use there are quite a few that are "nearby" in terms of rotation, seek, or platter.

    --
    jim frost
    jimf@frostbytes.com
  42. File types and fragnentation by Artifakt · · Score: 4, Informative

    There are so many comments already posted to this topic that seem to not grasp the following point, that I think the best way to deal with it is to start a completely new thread. I'm sorry if it seems more than a little obvious to some of you:

    There are fundamentally only a few types of files when it comes to fragmentation.

    1. There are files that simply never change size, and once written don't get overwritten. (Type 1). Most programs are actually type 1, if you use sufficiently small values of never :-), such as until you would need to perform disk maintenace anyway for lots of other reasons in any 'reasonable' file system. A typical media file is probably Type 1 in 99%+ of cases.

    2. There are files that will often shorten or lengthen in use, for example a word processor document in .txt format, while it is stll being edited by its creator. (type 2). (That same document may behave as effectively Type 1 once it is finished, only to revert to type 2 when a second edition is created from it.)

    Of type 2, there are files of type 2a. Files that may get either longer or shorter with use, on a (relatively) random basis. (as a relatively simple case, a .doc file, that may become longer for obvious reasons like more text, but may also become longer for less obvious reasons (such as the hidden characters created when you make some text italic or underlined). (These are reasons that are not obvious to most end users, and often not predictable in detail even to people who understand them better). The default configuration for a Windows swap file is type 2a. It is likely to be hard for an automated system to predict the final size of Type 2a files, as that would imply a software system of near human level intelligence to detect patterns that are not obvious and invariant to a normal human mind. It may be possible to predict in some cases only because many users are unlikely to make certain mistakes, (i.e. cutting and pasting an entire second copy of a text file into itself is unusual, while duplicating a single sentence or word isn't).

    Then there are files of type 2b. Files that get longer or shorter only for predictable reasons, (such as a Windows .bmp, which will only get larger or smaller if the user changes the color depth or size of the image, and not if he just draws something else on the existing one.). A good portion of users (not all by any means) will learn
    what to expect for these files, which suggests a well-written defragger could theoretically also auto-predict the consequences of the changes a user is making).

    3. Then there are type 3 files, which only get longer. These too have predictable and unpredictable subtypes. Most log files for example, are set up to keep getting longer on a predictable basis when their associated program is run (type 3b). Anything that has been compressed (i.e. .zip) is hopefully a 3b, but only until it is run, then the contents may be of any type. A typical Microsoft patch is a 3a (it will somehow always end up longer overall, but you never know just what parts will vary or why).

    4. Type 4 would be files that always get smaller, but there are no known examples of this type :-).

    These types are basic in any system, as they are implied by fundamental physical constraints. However, many defrag programs use other types instead of starting from this model, often with poor results.

    In analyizing what happens with various defrag methods, such as reserving space for predicted expansion or defragging in the background/on the fly methods, the reader should try these various types (at least 1 through 3), and see what will happen when that method is used on each type. Then consider how many of those type files will be involved in the overall process, and how often.

    For example, Some versions of Microsoft Windows (tm) FAT32 defragger move files that have been accessed more than a certain number of times (typically f

    --
    Who is John Cabal?
  43. Re:Not Exciting by greck · · Score: 2, Informative

    Ideally you wouldn't see your harddrive thrash when booting...

    actually, Darwin/OS X has a really nifty feature called BootCache that collects information at boot time and primes the read-ahead on subsequent boots to smooth things out... everyone found out the hard way when it was mildly broken in an update to 10.2 exactly how much difference it makes (it knocks about 2/3 off the boot time of my PowerBook).

    see Amit Singh's excellent article for more info, there's a chunk on BootCache at the bottom of this page.

  44. Re:Bzzt! Nope. Close, though! by tenton · · Score: 2, Informative

    Journalling has existed since 10.2.2 (at least on the Server end; I believe the consumer end too, except you had to enable it via a terminal command), so... ^_^

  45. Re:Unsupported diss, unsupported support by cft_128 · · Score: 2, Insightful
    How exactly is FAT not a "real" file system? It's still very widely in use, particularly on devices smaller than 2 GB (digital cameras come to mind). It's still useful because it's so simple, well-known, and easy to implement. That makes it real enough to be "real" to most people with a clue.

    OK, I was being a bit snobbish in saying it is not a 'real filesystem', it does have its uses - small devices, floppies, etc. BUT, even when it was originally designed it was considered primitive it had many known flaws, among those: it is very easily fragmented (what we are all talking about, no redundancy to help recover from failure and wastes quite a bit of disk space.

    My comparison between NTFS and FAT is valid because if you are running Windows, those are the only two filesystems you have to choose between. Comparing NTFS with, for instance, ReiserFS is not really interesting because they're not really alternatives to each other. Unless you choose your operating system based on what filesystems it supports...

    The article was comparing HFS+ to NTFS, not windows filesystems. You your self said NTFS deals with fragmentation far better than many other file systems, most notably FAT (emphasis mine) which implies you were not only comparing NTFS to other windows file systems but to many other filesystems. It was the apparent straw man argument that I was pointing out. NTFS is leaps and bounds better than FAT, I 100% agree with you on that. It could be better, and I wish it was open source (last I checked it was not) but is still the best option window users have.

    --

    Underloved Movies and Pub Quiz: donotquestionme.org

  46. Bring on the browser stats! by WgT2 · · Score: 3, Insightful

    A word about browsers (and any thing else that requires change):
    People, in general (more than 50% of them), prefer to resist change, and for that matter, extra work and/or thinking. It's just the way they are. It's what explains product loyalty. In this case, the product loyalty is browser based.

    In my job, as a web server support admin, I find that 95%, or more, of the people I speak with in support situations are not even aware of the alternatives available to them. In fact, just last Sunday, a friend of mine was showing off his new Power Book to me (by the way, even though I am a complete Linux advocate, you have to give credit where credit is due: Mac has a great GUI). I had to laugh during his enthusiastic demo of Mac OS X's features when my friend opens up Safari and goes, "Check this out. It's a feature called 'tabbed browsing.'" He was a kid in a candy store and had just found new, profound flavor of buble-gum or something. But, how could I not laugh at this previously 100% Windows user's intron to me of something that I began using in Opera, back around 5.x-6.x (I really don't remember if 5.x had tabs or not. I really don't care since that browser drives me crazy. But that's just me.) Translation: it's be around for years. In my work day I begin with 12-13 of them opening in FireFOx (NT 2000 doesn't like that, even with 512MB RAM, but it gets by well enough). The number of tabs only increase from there, unless there's an accident of closing a tab. But no big deal there either, I just open another one and then drag it back between where I normally would have it in my list of tabs. You won't find any thing like that in a browser direct from MS.

    Another example: my co-workers, particularly the NT techs. Most, certainly not all (thank God), of our NT techs still use IE for their work. I don't really know what they need for their work, but I've seen their desktops and their taskbars; WHAT A MESS! It's beyond me why they would waste their time with a browser (read: IE) that doesn't organize their open web pages into one taskbar entity, because they DO use other programs on the NT 2000 desktop, which we all must use at my job, regardless of the servers we admin for. (If you haven't guessed yet, I don't admin for NT servers, I get the please and ease of admining for Linux boxes. And a big THANK GOD for that!)

    Back to my point: most people are not aware of features in other browsers AND if they are aware of new inovations (read: tab browsing, which is one reason I will never go back to IE) they are not in any hurry to change and think and evaluate something that, however troubling it can be at times: pop-ups, vulnerbilities, "________________" [fill in the blank], lack of inovation, etc.

    So what if most of /. visitors are Windows based? There are plenty of better choices to MS products, even on their own OS platform. But, people the world over resist change; they get stuck in a rut, good or bad in it's results, and they either don't like to change, don't "need" to change, or cannot change. Thus, the end result is resistance to change; for the better or for the worst.

    1. Re:Bring on the browser stats! by Nexx · · Score: 2, Informative

      FWIW, WinXP will, when you have a tonne of browsers open, place them under one entry on your taskbar. Same goes with Excel windows, vi windows, etc. I think it goes by the process names.

      What's not too obvious is that a lot of Windows administration tools are instances of one executable (though I imagine it executing different things), so they all get lumped under one entry on my taskbar too.

  47. Re:Offtopic by bnenning · · Score: 2, Insightful

    Tell ya what: FOAD.

    Ah, liberal tolerance rears its head again.

    Bush lied, Bush continues to lie, and our country is far, FAR more in danger now than when we started this stupid fucking war.

    I'd be interested in the metric you use to compute danger, seeing as how there have been exactly zero terrorist attacks on US soil since 9/11. (By the way, were you out protesting the "stupid fucking war" in Serbia, or are Democrats allowed to invade sovereign nations who pose no external threat?)

    Bush said he was 100% certain that Saddam had massive stores of WMD and a nuclear program.

    Is it just barely possible that maybe he really did believe that, and he was mistaken? Intelligence agencies have been known to make mistakes before. Never mind, I forgot he's from Texas and worked in the oil industry so he's obviously made a pact with Satan.

    Oh, but of course, he never ACTUALLY said it. He just IMPLIED it, which makes it all ok, doesn't it? It was a failure of intelligence, which means it ain't his fault! Nothing is his fault! And I mean shit, who needs morals when you're having to deal with them dirty hippie commie faggot libruhls and that libruhl media?

    It's amusing watching you guys get progressively more unhinged. Kerry should be leading Bush by a healthy margin given the Iraq situation and that the economic recovery isn't completely visible yet, but when your talking points are all variations on "Bush is a fascist", you can't expect much from the middle. I'm not a huge fan of Bush, but I'll be enjoying his victory on election night just envisioning the enragement of the left.

    --
    How to solve most of our problems: 1.Lots of nuclear plants. 2.Cure aging.
  48. Re:Offtopic by amRadioHed · · Score: 2, Interesting
    I'd be interested in the metric you use to compute danger, seeing as how there have been exactly zero terrorist attacks on US soil since 9/11.
    And I'd be interested in the metric you use. Tell me, how often did we have terrorist attacks on US soil before 9/11? Well let's see, there was Oklahoma city, but that was an American so it doesn't really apply does it. So that leaves The first WTC bombing as the most recent attack preceding 9/11. 8 years lie between those two attacks.

    If we take the extremely generous assumption that foreign terrorists attack the US every 8 years, for you to make an even remotely reality based assessment of our relative security since 9/11 I'd say you need to wait at least 12 years without any attacks.

    Granted, the grandparent would have difficulty proving his assertion that we are in much greater danger now, but I'm sure it could be proven that we have many more enemies. That's not exactly comforting.
    --
    We hope your rules and wisdom choke you / Now we are one in everlasting peace
  49. Re:Offtopic by toupsie · · Score: 2, Informative
    And I'd be interested in the metric you use. Tell me, how often did we have terrorist attacks on US soil before 9/11? Well let's see, there was Oklahoma city, but that was an American so it doesn't really apply does it. So that leaves The first WTC bombing as the most recent attack preceding 9/11. 8 years lie between those two attacks.

    You are forgetting two embassies in Africa and an American Warship. All of those are American soil. So it is not an attack every eight years.

    --
    Strange women lying in ponds distributing swords is no basis for a system of government.
  50. Fast! by rixstep · · Score: 2, Informative

    One thing people rarely talk about is how fast HFS+ is. Or perhaps how slow UFS on the Mac with OS X is. But the difference is more than dramatic: a clean install of OS X using HFS+ can take less than half an hour - including the developers tools. The same procedure using UFS seems to never end.

    It might be the way they've 'frobbed' UFS for use with OS Server, but UFS really gives high priority to disk ops with GUI ops taking the back seat, and yet HFS+ is in comparison blazingly fast.

    I believe in a good clean machine like anyone, and I do see the probability DiskWarrior will be needed now and again, but the speed alone is quite a pedigree for HFS+ IMHO.