Slashdot Mirror

← Back to Stories (view on slashdot.org)

Automated Tiered Storage Coming to Desktops?

Posted by ryuzaki0 on from the not-the-droid-jpgs-you're-looking-for dept.

roj3 writes "Tiered storage has been the scourge of administrators because the vendors tell us to hold meetings with all departments and then classify data to storage tier based on its type or relative importance. eWeek has a story about a new approach to tiered storage — sorting it all by usage patterns. Regularly used data goes on high-performance storage, idle data goes on slower/cheaper storage. Volumes and files even span several types of drives or RAID levels. Is automated tiered storage headed to desktops?"

40 of 110 comments (clear)

  1. Networks, sure. by celardore · · Score: 5, Insightful

    I can see the usefulness of this technology over a busy network with multiple users and masses of files and storage... I just can't see needing anything more than a mirror&stripe RAID array on a PC with only one user. Even that could be considered excessive.

    1. Re:Networks, sure. by dsginter · · Score: 4, Interesting

      I think we'll actually see the opposite:

      With multiple PCs per household, it makes sense to get rid of the hard drives at the PC level and put them in a RAID enclose that is secured into a wall.

      This, however, is a threat to Microsoft because you'll be able to PXE-boot any image of your choice (just think that perhaps your employer or bank supplies their own secure image in order to connect to their resources). Someone needs to get Windows to PXE boot at the hardware level (emulate IDE or something).

      This will be huge but we've got to squeeze Microsoft into it, first. Then, everyone will be free to try linux and see what we've all been jabbering about.

      --
      More
    2. Re:Networks, sure. by 0racle · · Score: 4, Insightful

      That only makes sense if the people in the household wish to learn how to use what you've mentioned. Since current evidence points to the fact that most people look at computers as a magical box that can not be understood, the chances of them learning how to do a fraction of what you suggest is about as likely as you winning the lottery.

      The XP file sharing wizard is too much for a lot of people and you think a raid array sharing up OS images over a network via PXE makes sense?

      --
      "I use a Mac because I'm just better than you are."
    3. Re:Networks, sure. by jwjcmw · · Score: 4, Insightful

      "Life is changing to the digital a bit more evey day. And just as we have cardboard boxes in our attic holding the things we dont use, file cabinets in our office alphabetized, firesafes for important documents, and Safe Deposit boxes for wills. The average home user will need to know and use the digital equivalents."

      Or, if you are like many people, you have documents on your desk and in piles on the floor that you will never use, your kids birth certificate is in a stack of papers from when you had to take it to school for registration, your file cabinets have partially labeled folders that are in chronological order...as in the order that you stuffed them in the filing cabinet, your will is in the "to be filed" folder in the bottom of said filing cabinet and you could fill the bathtub with your old phone and electric bills.

      Hopefully the digital equivalents will be better for the organizationally challenged.

  2. Great Idea by Jazz-Masta · · Score: 5, Insightful

    This is exactly what everyone is looking for. People defrag their hard drives in the hopes to increase performance. There is no reason why storage that is accessed more shouldn't be on the high performance drives. Or at least some sort of class rating that defines what storage may need high performance. For example, automatically installing and saving 3D Max to a RAID 0 media, and saving word documents to the lesser-performing drives.

    I try to follow this idea all the time with my system. Fast stuff goes on RAID 0, slow stuff, and backup stuff goes on the ole' 200 GB backup drive.

    1. Re:Great Idea by mollog · · Score: 4, Informative

      Hewlett-Packard Company developed a product that did this automagically. It was an external RAID system that connected via one or two SCSI busses to a host. All incoming data was stored in RAID 0/1; striped and mirrored. (aka RAID 6 and RAID 10). As the storage filled up, unused data was automagically migrated to more space-efficient RAID 5. Data that had been accessed recently remained in RAID 0/1. You could add disk drives and it would automagically include the drives (but you would have to use LVM or other utilities in the OS to increase its file system.) You could mix two drive sizes, say, 18GB and 36GB, without trouble. If a drive failed, the array would rebuild reduncancy. If another drive failed, ditto. It was fast, it was fully redundant.

      But it was a lot smarter than the admins who had to use it so it wasn't very popular.

      --
      Best regards.
    2. Re:Great Idea by pla · · Score: 3, Insightful

      This is exactly what everyone is looking for.

      No.

      You (and a number of other posters on this topic) have described what we look for - Geeks who want to get the most out of their systems with the least expense. If I could get killer performance with a RAID0 of tiny but fast drives (think Raptors, or even Cheetahs if you don't mind dealing with SCSI), while still having the capacity of a cheap 400GB IDE drive - Of course I'd have such a setup (and in fact, many of us already do, we just manually transfer things to/from the big-n'-slow).

      Most people, however, do not want this. For starters, most people don't even need the huge drives they already have - If you gave them just the pair of RAID0 36GBs, they'd never use even half that the capacity, so no need for ever moving files to the slow storage. Then failing that, the members of the Sixpack family that manage to store hundreds of GB only fill it with downloaded porn, music, and movies - Uses that really don't need fast drives, just tons of space.


      So while it sounds useful in theory - in practice, such a setup would just add cost and complexity without providing any tangible benefit to most users. I suspect even most Geek users would rarely notice the difference (aside from OS load times), and would only make such a setup for bragging rights.

  3. Already have teirs... by Kaenneth · · Score: 3, Insightful

    Registers, CPU cache, on-chip cache, RAM, local disk, Network/Removable Media, Paper/Human memory...

    It's all about feeding that data hungry CPU, as quickly as possible.

  4. Not so new... by Duncan3 · · Score: 4, Interesting

    I was using systems that did this 10 years ago. Granted, back then it was disk+tape not different speed disks, but it's the exact same thing.

    Looks to me like an excuse to charge 8-10x what you should be paying for storage of that size.

    --
    - Adam L. Beberg - The Cosm Project - http://www.mithral.com/
    1. Re:Not so new... by truthsearch · · Score: 2, Informative

      Ten years ago you had something automated that determined where the files should go and moved them appropriately? It analyzed usage patterns? I'd really like to know what older systems had such features as I've never seen them.

      --
      Developers: We can use your help.
    2. Re:Not so new... by hotrodman · · Score: 3, Insightful


        No kidding. So they find a way to put less-used data on slower disks, that still COST NEARLY AS MUCH. The entry price is still listed as $50,000. Big fuckin' deal. Let me know when you take a bunch of garden-variety servers, and do this, with the super cheap clone raid server with 40 terabytes of SATA as the 'last tier' for slowest files, where I can build 100 terabytes for $50,000.

          And yet, managers will get a woody over this buzzword compliance and want to give these guys millions to have the 'latest and greatest'.

          And have it still work with tape, too, and not tied up in some cumbersome, proprietary protocol owned by one little company that could go out business.

    3. Re:Not so new... by drinkypoo · · Score: 2, Interesting

      I know bugger all about them so I can't vouch for the accuracy of this information but someone who worked in the basement of the Santa Cruz County Courthouse, where the county's servers are - some of those big goofy IBM mainframes that require their own AC system - have been ticking away since time immemorial... and according to one of the sysops they have tiered storage which automatically will put stuff on magtape, and then ask them for the tape again later when the records are accessed. I guess a lot of what they did down there was serve automated tape requests. I hope they're no longer using such a system (disk is cheap) but who knows, it's government.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    4. Re:Not so new... by dpilot · · Score: 4, Informative

      It was called HSM, (Hierarchical Storage Management) it ran on IBM's MVS on mainframes, and it moved your less-used data to cheaper storage, in several stages. IIRC, the first stage was just compression on a different disk, the second stage was a tapes in a jukebox-type thing, and the third stage was tapes that an operator fetched and loaded. Somewhere way back there, data never used for 5 years fell off the end of the belt, but you got warned, first.

      The day after vacation, when you kept getting the message, "DFHSM is recalling dataset xyz for user jkl" as it pulled all of your storage back online was a pain, and we all thought it would be neat to get rid of, as we migrated to workstations. But in retrospect, HSM was great, never having to worry about your data quantity. That's compared with having to root through $HOME every few months to take care of quota problems.

      --
      The living have better things to do than to continue hating the dead.
    5. Re:Not so new... by Doctor+Memory · · Score: 3, Informative
      you had something automated that determined where the files should go and moved them appropriately? It analyzed usage patterns?


      Oh yeah. BITD, there was the archiver, a job that ran every night and moved files that hadn't been accessed in the last N time periods to tape. It left the VTOC entry (kind of like an inode), just marked it "archived" and the label of the tape. Then, the next time that file was accessed, a hook in the open() call would send a message to the console operator telling them to mount tape such-and-such. When the tape was mounted, the archiver would automatically copy the file back into place, the open() call would complete normally, and life was good. Basically transparent to the user (they'd look at their directory and all their files would be there), except for the fact that the file open would take two-three minutes. Then again, since they were paying for disk storage by the block-day, they were generally pretty happy to only pay for a fifty-cent tape mount every quarter instead of keeping that 1200-block file on-line for three months when they weren't using it.
      --
      Just junk food for thought...
    6. Re:Not so new... by drinkypoo · · Score: 2, Informative
      That's not the same thing. At all. Read the article again.

      You know, I did read it, and what they're talking about is that data that is less used/less critical gets moved to slower/less reliable storage automatically.

      And when you have only two kinds of storage, a DASD bank and mag tape, and your system automatically writes least used data to tape and tells you to file it, and asks you for tapes when it needs them - well, I'd say the two are highly analogous. The fact that the slower storage is offline is nothing more than a detail. It's still storage, it's still available - it just has a wicked long seek time.

      I realize that offline storage is different - but only in some details. The majority is the same concept.

      So it's not the same thing, but the "at all" is bullshit. They're quite similar.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    7. Re:Not so new... by Anonymous Coward · · Score: 2, Informative

      Guys, guys, guys, you talk about HSM like it is old and gone. On the contrary, it is the future! We use it everyday where I work, using a program called SAMFS. We have a tape robot and a large disk cache. Data that is used often stays on the cache. Less used data goes to tapes. SAMFS sorts out when/how to do that. The system is great not only because the software figures all of this out for you, but also because it works as your backup software. We switched to this system about 4 years ago when we realized that doing a nightly backup of our TB's of storage was taking hours and was no longer a nightly backup but a morning backup as well. Plus, adding additional disk drives for storage was going to be a very costly thing.

      While it is true that getting less used data OFF of the tape takes some time, it is not that bad if it is data you don't use very often. Depending on the file, it seems to take about 8 minutes or so for us. I think that's because the tape has to find the file, but once it has, it can just copy it off.

      I highly recommend SAMFS. Unfortunately, it only runs on Solaris. :( We examined several Linux solutions at the time, but they were too unstable. Perhaps now there are better ones out there?

      Check it out. Even if you don't use it, it's cool stuff.

      John

  5. Certainly could be done in a desktop by Don853 · · Score: 2, Insightful

    Put two 10k Raptors in Raid 0 for your games and other stuff you need REALLY FAST, and then have a big 250GB 7200RPM drive for everything else. People are doing that already.

    All you would need is some software for automatically moving it around. Though most people with desktop rigs like that probably would rather control what is on which drives themselves.

    1. Re:Certainly could be done in a desktop by COMON$ · · Score: 3, Informative
      Because 2 10K Raptors in Raid 0 isnt worth the speed increase. Last time I checked you may get a 20% increase, and reduced data integrity. I did some research into this a while ago, check out this article, very informative

      http://www.anandtech.com/printarticle.aspx?i=2101

      --
      CS: It is all sink or swim...oh and did I mention there are sharks in that water?
  6. Oh....good.. by JerBear0 · · Score: 5, Insightful

    "idle data goes on slower/cheaper storage"

    So that special little something that you need once a year, but when you need it, you need it RIGHT NOW is tied to the foot of a pigeon fluttering around the warehouse somewhere. Frequency of use does NOT denote importance.

    --
    Bad experience is a school that only fools keep going to.
    1. Re:Oh....good.. by Red+Flayer · · Score: 3, Informative

      That's what metatagging is for. Tag files that are not to be moved to slow storage no matter how infrequently they are accessed. RTFA.

      --
      "Trolls they were, but filled with the evil will of their master: a fell race..." -- J.R.R. Tolkien on Olog-hai
    2. Re:Oh....good.. by Kadin2048 · · Score: 3, Insightful

      Frequency of use doesn't denote importance, but it might denote how quickly you need to be able to recall it. Similarly, importance doesn't imply that quick recall is necessary. If you don't use something frequently, it might be okay to store it somewhere that takes a while to recall from, even if it is "important," as long as you know where it is so that you can get it back.

      As an example, financial records for past years might be very important, but you don't need to be able to access them in a tenth of a second. As long as you can get to them if you really want to (sacrificing a few seconds), then it's all right.

      The way I see this translating to reality is that you'd keep all your old documents in slow-speed storage, but then keep an index in high-speed storage, so that you could easily search (both by name and by content) and decide when to pull stuff out of your archives.

      This is no different than what people have been doing for centuries with paper. Just because the card catalog is located in the center of the library doesn't mean its contents are inherently more valuable than the actual books (which might be in the basement, back shelves, wherever); it just means that the catalog gets accessed much more often.

      Actually, in the physical world, people often exchange speed of recall for certainty of recall. You put important documents in a safe-deposit box, rather than your kitchen counter, because even though it'll take you longer to get them out of the box, they're guaranteed to be there when you need them. Likewise, a system which traded off speed for redundancy would probably be appropriate for "important" but infrequently-accessed electronic documents.

      --
      "Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
  7. No but it is correlated by davidwr · · Score: 3, Insightful

    Apply "frequency of use = urgency" to BIGNUM pieces of data and you will have a very useful albeit sub-optimal algorithm.

    Yes, there are exceptional cases, like the President's access to the Nuclear Briefcase. It hasn't been used for real in a long time if ever but when he needs it it had better be close at hand. However, these special cases can be treated as the special cases they are.

    --
    Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
    1. Re:No but it is correlated by cperciva · · Score: 4, Insightful

      Yes, there are exceptional cases, like the President's access to the Nuclear Briefcase. It hasn't been used for real in a long time if ever but when he needs it it had better be close at hand.

      Oddly enough, I think most people in the world would prefer that it wasn't close at hand when Bush decides he wants it.

      A better example is fire extinguishers -- most of them will literally never be used, but there's a very good reason to ensure that they are readily available.

      --
      Tarsnap: Online backups for the truly paranoid
    2. Re:No but it is correlated by CosmeticLobotamy · · Score: 3, Funny

      A better example is fire extinguishers -- most of them will literally never be used

      If that's the case for you, then I feel sorry for you. You've apparently never known the snowy, probably-toxic joy of Fire Extinguisher Expiration Day. It's the happiest day of the decade.

  8. IDE Neutrality? by sseaman · · Score: 5, Funny

    From its beginnings, the Hard Drive has leveled the playing field for all files. Everday files can have their content read by thousands, even millions of processes.

    The Coalition of Unused Files believes that the desktop is a crucial engine for personal and economic growth. They are working together to urge System Admins to preserve IDE Neutrality, the First Amendment for the Desktop Hard Drive that ensures that the Desktop remains open to innovation and progress.

  9. This is "new"? by Medievalist · · Score: 3, Insightful


    IBM mainframes that literally pumped water were doing this decades ago.

    What, you say water cooling is coming back too?

  10. It already is by malraid · · Score: 3, Insightful

    That's why you have HDD with cache. That's the whole concept of "virtual memory". The next step might be hybrid hdds (solid state / mag platters). But I don't think it will go much farther than that. Multiple raids is overkill for the average desktop.

    --
    please excuse my apathy
  11. Just read TFA: by Ant+P. · · Score: 4, Insightful

    $50k for a 6TB fileserver? What's that extra $40000 paying for that a normal fileserver loaded with RAM can't do just as fast?

    1. Re:Just read TFA: by Anonymous Coward · · Score: 3, Interesting

      Apples and pomegranates you compare;
      Channels of Fiber come not cheap.
      Terabytes 6 with connection of light for less than $50k you will not find.
      Terabytes 6 with connections of wire you may.
      SATA drives, untested are delivered.
      SATA drives with fewer bearings.
      SATA drives with short life.
      Enterprise storage is not easy.

    2. Re:Just read TFA: by Anonymous Coward · · Score: 2, Interesting

      It's not a server, it's a SAN. You connect a server via HBA to the SAN unit. The cost differece is in the performance of the drives you're getting (8 15k 146gb fiber channels and 8 500gb 10k fiber channels), these aren't the same Maxtor 250 GB SATA drives you picked up at Best Buy last week. (then there's the enclosures, controllers, io cards, etc....)

  12. Just like my kitchen by Red+Flayer · · Score: 5, Funny

    Cheetos go in the easy-to-reach cabinet next to the fridge.

    Beer goes in the fornt on the top shelf of the fridge, milk (eventually cheese, typically) goes on the bottom shelf in the back.

    This is automated, since I simply shove things onto the shelves when I get home from the supermarket. Anything I consume and replace ends up at the front. Anything I buy because I 'should' be eating it (like fiber biscuits, or whatever) ends up pushed to the back.

    It's automated via metatag, too. Anything tagged 'ice cream' goes in the door of the freezer, anything tagged 'vegetable' gets relegated somewhere in the back, where it quickly develops an inch of ice crystals, to slowly dry out to a freezer-burnt state of suspended animation until I buy a new fridge unit.

    This costs no more than regular kitchen storage space, but if you'd like a custom design for you and your loved ones, my consulting fee is $75/hr, or a bag of chips and a six-pack.

    --
    "Trolls they were, but filled with the evil will of their master: a fell race..." -- J.R.R. Tolkien on Olog-hai
  13. Yes, Kinda... by ThinkFr33ly · · Score: 4, Informative

    Automatic tiered storage is definitely coming, but probably not in the form of multiple disks that run at different speeds or RAID levels.

    Microsoft announced a while back that Windows Vista would support three technologies designed to improve disk speed called SuperFetch, ReadyBoost, and ReadyDrive. SuperFetch is simply a way of preloading applications and data when the OS anticipates that you'll be loading those soon.

    ReadyBoost and ReadyDrive both utilize persistent memory caches to speed up access to the disk.

    ReadyBoost treats normal USB keys and flash disks like temporary caching locations for data from the disk.

    ReadyDrive is essentially the term Microsoft uses to described their support for hybrid hard drives, which are disks that have a built in flash memory module that's used as a persistent cache.

    Not only do hybrid disks dramatically increase performance, but they also result in huge power savings for mobile devices like laptops and media players.

  14. I could see a use. by Kadin2048 · · Score: 2, Interesting

    I could see a use for something like this. Personally, I've stopped throwing stuff away. With the exception of temporary and cache files, storage is cheap enough that I just don't delete anything on the off chance that I might want it again. Every email, every instant message, every dictated note (I use a little Olympus digital recorder), every digital photo, it's all saved. By the time I fill up my main hard drive with stuff, I can just buy another one that's probably between two and five times the size, dump everything onto it, and keep the old one as a historical backup. (I keep online backups as well, but I won't bore you with it here.)

    I don't think I'm that atypical in this regard. GMail brought the idea of saving all your email, forever, to the masses; Flickr gives you an unlimited amount of photo storage; and technologies like Apple's Spotlight make it relatively easy to search through gigabytes of saved information and pull up related items. What we haven't seen yet is a lot of popular interest in redundant backup systems: that'll come later, once people start realizing how much of their lives they're stored away on the crummy OEM drive in their Dell. (Probably after a lot of them fail and we hear some real horror stories.)

    It's not hard to imagine a near future where people just get used to not throwing anything away. In that situation, tiering storage -- allocating the fastest media to the most frequently accessed information -- could have big performance gains. And assuming that you have a relatively static amount of frequently-accessed information, and basically only add information to the "infrequenly accessed" category, a tiered system means that you only really have to add storage to the bottom tier. It's a pyramid where the base gets larger and larger, but the upper part remains basically the same size.

    So for example, as you save more and more emails (infrequently accessed information), they automatically get saved onto inexpensive, slower drives, which are then mirrored to each other for redundancy. A single, fast drive could hold the system -- maybe solid state storage? -- and more frequently-accessed data. A smart system would know what information needs to be moved up to faster storage to be very useful (uncompressed digital video, for example, wouldn't be much fun to work with off of a slow drive), and what can be left there as it's accessed (MP3s and compressed video could be played directly from slower media).

    I think it's an interesting technology with a lot of possible applications, but as with a lot of other things, it'll be the home user who arrives last to the party, because their storage is the least centralized. Unless there's a move away from storage on individual desktop PCs and towards storage on per-home servers, it'll be a while before most people require or see the benefit in such a thing.

    --
    "Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
  15. Liars, damn liars, and statistics by Medievalist · · Score: 4, Informative
    Decades ago, we used to laugh at the mainframers and their automated hierarchical storage systems because they'd make exactly these kinds of statements.

    Frequency of use DOES denotes importance, at the very least STATISTICALLY.
    No. Absent other data, it only denotes frequency of use, period. Playboy.com gets more hits than the general ledger webapp if you unblock your company firewall, but the general ledger is more important to the company.

    Just because you want "that special little something" once a year; does not mean you can degrade the speed of information which is instantly needed.
    There is actually very little correlation between what the average user wants and what s/he needs, as is empirically obvious. If the image from the "fly-fishing.com" website that they've set to come up as their background image every morning fails to load, they can still work, but if the once-a-year corporate audit checklist gets put on slow, old storage and then gets lost in a hardware failure, the company stock price may flutter and certainly heads will roll in the corporate IS department.

    This is an obvious fact
    I don't think that word means what you think it means.
  16. I'd love to see it... by drinkypoo · · Score: 2, Insightful

    ...but we can't seem to even get a fucking trashcan right.

    I should never have to empty my recycle bin manually, except where I want to perform a security erase - which should be a function delivered with my operating system. This is the height of stupidity.

    It's not even a hard problem! There's functions which programs use to check for free space. Lie to them. Don't count files in the recycle bin against the available free space. If you're about to run out of space, delete the least recently used file. Perhaps you might also base things based on total number of accesses, or other criteria, but I believe (perhaps naively) that making the trash can an automatic FIFO from which files are automatically deleted when disk space is low would be about a hundred times better than what we have now.

    Also, I want this functionality on all operating systems. Unless I explicitly request deletion, no file should ever be unlinked, deleted, or whatever you call it when I delete it, whether through the command line or the GUI.

    This is not hard and it would make everyone a lot happier.

    --
    "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    1. Re:I'd love to see it... by mrsbrisby · · Score: 3, Interesting

      Also, I want this functionality on all operating systems. Unless I explicitly request deletion, no file should ever be unlinked, deleted, or whatever you call it when I delete it, whether through the command line or the GUI.

      The problem with this is, is that it causes a significant reduction in performance.

      Ideally, the operating system chose the best possible spot for that file when it got written. Once that file is deleted, that spot will once again be the fastest best possible spot- for at least something. If the operating system skips that spot for a new file, then this new file isn't going to be accessed quite as quickly.

      Truly automatic tiered storage solves this problem by splitting the directory services from the storage system- that is, the file's _name_ is no longer tied to the volume that the file happens to live on (and no, this isn't the same thing as symlinks or shortcuts). This allows the decision as to what the best spot for a file is to be deferred until later- and even spanned across multiple volumes!

      Unfortunately, such a beast is very difficult- if we make a reduction in our requirements- say that performance isn't very important- or perhaps that we can stop using our computer for a few hours each evening, then it's probably possible. What we need is a new kind of file system that supports either atomic moves between disks, or a filesystem that splits the names from the storage.

      A few research projects have been focused on these kinds of changes- but they all tend to break UNIX semantics (Amoeba immediately springs to mind)- and those UNIX semantics are, in-fact, the most widely used and recognized semantics for filesystems anywhere (Even Windows uses them!)-- people who develop a filesystem incapable of supporting them, really need to have a real good reason for breaking everyone's hard work.

      While they often do, it hasn't yet been seen as good enough for general purpose stuff.

    2. Re:I'd love to see it... by mrsbrisby · · Score: 2, Informative

      Filesystems may be automatically and intelligently defragmented (while live, if the filesystem is decent) when disk I/O is at a minimum.

      But my filesystem is never idle, or even nearly so. Nonetheless, fragmentation isn't exactly a bad thing, and doesn't necessarily have to cause problems (such as lost performance) by itself.

      Worse still: How does the defragmenter know to avoid using this block? Or how does it know that it's a good candidate to be moved to the other end of the disk?

      We could make a record of every block that we access and sort it- as later information for the defragmenter, but would it really help? Wouldn't the cost of updating that information be too high?

      These real questions have been examined and explored, and the short answer of it is simply "we don't know."

      Access patterns are so complicated it's not even funny.

      That's where tiered storage comes in- instead of trying to make the perfect defragmentation utility, we simply "copy" all of our storage to the lesser speed disks and update all the pointers.

      We'd still never know when it was safe to remove the data from the first disk, but at least we'd never have to know- if we needed space, we could simply reclaim any storage from the fast disk that had already been mirrored. We might guess wrong, but on next access, we could reinforce that it guessed wrong by copying it back.

      As a result, we don't have to do extra writes in the middle, and we don't need a fancy defragmenter...

      Currently, some operating systems (e.g. Windows XP) optimize file location for minimizing boot time. I don't see any reason the same concept couldn't be extended. It might not be a bad idea even in the absence of these other concepts.

      NTFS does this by having a (small) table near the beginning of the disk. The ``files'' in it are packed in such a way that the entire table is read into memory- as the boot procedure "knows" that they will all be needed shortly.

      Unfortunately, updating this table is expensive, and it isn't very large, so it cannot keep very many files.

  17. Hot File Adaptive Clustering? by Kadin2048 · · Score: 3, Informative
    I have heard mention of this as well, but I'd never seen any details. I tried to dig up some information; here's what I found.

    Apple's "About disk optimization with Mac OS X" (basically telling you that you don't need to defrag), says "Mac OS X 10.2 and later includes delayed allocation for Mac OS X Extended-formatted volumes. This allows a number of small allocations to be combined into a single large allocation in one area of the disk." ... "Mac OS X 10.3 Panther can also automatically defragment such slow-growing files [that data is continually appended to]. This process is sometimes known as "Hot-File-Adaptive-Clustering.""

    There's also a reference to a "hot band," a region of the drive where data is written that's used during startup, in order to increase performance and I assume lessen boot times.

    There's also reference to some automatic defragging in this macosxhints article on HFAC:
    There are 2 separate file optimizations going on here.

    The first is automatic file defragmentation. When a file is opened, if it is highly fragmented (8+ fragments) and under 20MB in size, it is defragmented. This works by just moving the file to a new, arbitrary, location. This only happens on Journaled HFS+ volumes.

    The second is the "Adaptive Hot File Clustering". Over a period of days, the OS keeps track of files that are read frequently - these are files under 10MB, and which are never written to. At the end of each tracking cycle, the "hottest" files (the files that have been read the most times) are moved to a "hotband" on the disk - this is a part of the disk which is particularly fast given the physical disk characteristics (currently sized at 5MB per GB). "Cold" files are evicted to make room. As a side effect of being moved into the hotband, files are defragmented. Currently, AHFC only works on the boot volume, and only for Journaled HFS+ volumes over 10GB.
    So that seems to be the deal; if anyone else has more information, I'd be interested to hear about it.

    There's also a MacSlash article on HFAC and a discussion on Ars that includes a post of the source code.
    --
    "Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
  18. Re:Coming full circle -- good idea! by ratboy666 · · Score: 3, Informative

    And my favorite commands on the ol' HP-2000 mini:

    SANCTIFY and DESECRATE

    "Sanctify file" moved the file to drum (basically, one-drive RAID 0 for all you young-uns). Desecrate moved it to the regular hard disk.

    YMMV

    Ratboy

    --
    Just another "Cubible(sic) Joe" 2 17 3061
  19. Gee, a New Idea, only 43 years old! by CAOgdin · · Score: 2, Informative

    It was about 1962, when IBM was touting something they called "Percolate & Drip" storage. The idea was that things that were used often "percolated" up to the fastest storage medium, while data that was only infrequently used would "drip" down to the most capacious media. Why do children get to claim everything they imagine is somehow NEW? Mature adults try to stand on the shoulders of giants.