Slashdot Mirror

← Back to Stories (view on slashdot.org)

Garbage Collection Algorithms Coming For SSDs

Posted by Soulskill on from the take-out-the-tash dept.

MojoKid writes "A common concern with the current crop of Solid State Drives is the performance penalty associated with block-rewriting. Flash memory is comprised of cells that usually contain 4KB pages that are arranged in blocks of 512KB. When a cell is unused, data can be written to it relatively quickly. But if a cell already contains some data, even if it fills only a single page in the block, the entire block must be re-written. This means that whatever data is already present in the block must be read, then it must be combined or replaced, and the entire block is then re-written. This process takes much longer than simply writing data straight to an empty block. This isn't a concern on fresh, new SSDs, but over time, as files are written, moved, deleted, or replaced, many blocks are a left holding what is essentially orphaned or garbage data, and their long-term performance degrades because of it. To mitigate this problem, virtually all SSD manufacturers have incorporated, or soon will incorporate, garbage collection schemes into their SSD firmware which actively seek out and remove the garbage data. OCZ, in combination with Indilinx, is poised to release new firmware for their entire line-up of Vertex Series SSDs that performs active garbage collection while the drives are idle, in order to restore performance to like-new condition, even on a severely 'dirtied' drive."

119 of 156 comments (clear)

  1. The logical next step... by Joce640k · · Score: 3, Insightful

    A weakness was found in first generation drives, the second generation drives fixed it.

    Film at 11.

    --
    No sig today...
    1. Re:The logical next step... by Anonymous Coward · · Score: 1, Informative

      This is the third generation, the second was to fix speed degradation through fragmentation.

    2. Re:The logical next step... by miknix · · Score: 5, Funny

      This is the third generation, the second was to fix speed degradation through fragmentation.

      And the fourth generation will fix SSD's small life longevity due to massive GC activity.

    3. Re:The logical next step... by Magic5Ball · · Score: 2, Interesting

      Not to be cynical, but these new algorithms, if implemented poorly, have the potential to run down the limited number of write cycles on the cells. Not that this could be strategically manipulated in any way...

      --
      There are 1.1... kinds of people.
    4. Re:The logical next step... by Anonymous Coward · · Score: 2, Funny

      The garbage man can, Marge, the garbage man can!

    5. Re:The logical next step... by zach297 · · Score: 5, Informative

      From the summary: "This isn't a concern on fresh, new SSDs, but over time, as files are written, moved, deleted, or replaced, many blocks are a left holding what is essentially orphaned or garbage data, and their long-term performance degrades because of it." The are talking about clearing sectors of garbage data that is no longer in use. It would have to be done anyways before the sector can be reused. The new firmware is simply doing that time consuming step early while it is in idle. The actual number of write cycles is not changing.

    6. Re:The logical next step... by jbolden · · Score: 1

      Good point I'm worried about that too. SSD already has poor life I'm not so sure about the garbage collection idea.

    7. Re:The logical next step... by Magic5Ball · · Score: 1

      The actual number of write cycles is not changing.

      That would be ideal, but if the designers of the algorithm could predict with perfect accuracy the usage patterns of the drive, they would be more profitably in a different line of business. The "would have to be done anyway" assumes quite a bit of foreknowledge, and "properly implemented" is an assumption that cannot be tested except through time in the field.

      Contemplate the following: some periodic process (log, network manager, etc) makes a few 3KB files which aren't grouped through NCQ, which only need to last 30 minutes each. As a hard disk, you don't know that second part, so when is the disk "idle" enough for this cleaning to take place? Active GC on a soon to be vacated block is at least pointless write when the block can just be marked empty. At what point *should* a sector be reused in this situation?

      Now, think about the multitude of ways in which even relatively simple things like a nightly append and tar of a column store (or even things like volume shadow copies) can fall outside assumptions about what is, or is not, data that needs to be consolidated.

      I would imagine that drives with a "server profile" GC could sell for much more than drives with a "desktop profile" GC...

      --
      There are 1.1... kinds of people.
    8. Re:The logical next step... by amRadioHed · · Score: 2, Insightful

      How would this make a difference? The blocks would have to be wiped out next time they are written to anyway, the only difference here is that the blocks are cleared during idle time so you don't have to wait for it.

      --
      We hope your rules and wisdom choke you / Now we are one in everlasting peace
    9. Re:The logical next step... by ion.simon.c · · Score: 1

      ...cause if I buy an SSD and it wanders around eliminating random files cause I haven't accessed them in 2 months...

      Surely you're suggesting this in jest?

    10. Re:The logical next step... by broken_chaos · · Score: 2, Informative

      I *think* you're misunderstanding how this works, actually.

      When a block is written to, the entire block (512KiB) has to be wiped and rewritten from a blank state. When a block is emptied entirely, it does not get touched - just marked as empty. When new data is written to it, the 'empty' block has to actually be wiped, and then the new data written on the just-blanked block.

      What this seems to be proposing is to, periodically, actually wipe the blocks marked as empty, when the SSD is otherwise idle - meaning deletes are still fast, and new writes would speed up. I imagine rewrites would stay comparatively slow, though.

      (I might be way off on this - someone correct me if I am.)

    11. Re:The logical next step... by zach297 · · Score: 4, Informative

      Read http://en.wikipedia.org/wiki/Flash_memory#Programming . Clearing out an entire block is different than a write. Writing to an SSD is only possible by setting the value to 0. So when I save something to the SSD it is really only writing down the 0's of my file and just leaving the 1's alone. This is not the destructive part of using flash. The part that uses up actual write cycles is clearing a block back to 1's. This is explained in http://en.wikipedia.org/wiki/Flash_memory#Erasing .

      Taking from your list of actions: Pick a random block:
      1. GC comes along, swoops up block, eliminates junk by flashing entire block into 1's (awhile later)
      2. OS requires write, swoops up block, writes only the 0's from the file leaving everything else untouched.

      In this manner each step does half of the writing amounting to one write when combined. This is exactly how all SSDs work. The major difference announced in the article is that they are separating the two steps.

      Normally this is impossible because the SSD doesn't know if something can be cleared until the OS is trying to overwrite it. This makes writes take longer. The new firmware hopes to make writes faster by moving the first step into the idle time of the drive (by figuring out when a overwritten block is unused) sort of like how you can set up a download to only run when your not using the internet connection. It allows for more efficient use of time that the drive would otherwise be doing nothing with.

    12. Re:The logical next step... by jbolden · · Score: 1

      Block A is full on day 100
      10 cells are invalid (i.e. ready to be deleted) but 99% of the blocks still have space

      10 cells get wiped cleaan. While allows for 10 new writes. It fills up on day 120 with 10 cells invalid....

      day 200 there aren't free blocks anymore
      block A has 40 cells that are invalid
      they all get deleted. Since there are 40 free cells this lasts till day 280. Say on day 300 it gets deleted and 40 cells are free again....

      So in one situation you are flipping every 20 days, the other every 100 days.

    13. Re:The logical next step... by pantherace · · Score: 3, Interesting

      Anything measures in rewrites over hours or larger time spans is not (or shouldn't be) that much of a problem for modern flash. Someone calculated that you'd have to be reflashing a particular device every 15 minutes for 5 years to reach the flash's rewrite limit. That was several years ago. (It may have been 5 minutes as opposed to 15, but I'll give the less reliable number. This number appears to be from 2000 or 2001, as the device was the Agenda VR3 dating from about then.)

      Assuming it's as good as the flash from that example, rewriting every hour results in 20 years. I don't know about you, but I don't have many hard drives from 20 years ago.

      Now, if it's rewriting all the time, that could go down drastically, and quality might be different, but every 20 days shouldn't be a problem unless you've got really really crappy flash, by the standards of 9 years ago.

    14. Re:The logical next step... by the_one(2) · · Score: 1

      Wouldn't the number of erases drastically drop? Imagine that the every block on the SSD has been written two once. If you want to write anything now you'd have to read a page, erase, write back with the change. So if one assumes that on the filesystem this page was mostly empty a whole lot of garbage was just written unnecessarily. If this page was preemptively erased you'd only have to write the new stuff and the rest of the page will still be free.

    15. Re:The logical next step... by jbolden · · Score: 1

      Where are you getting that from? To the best of my knowledge 10,000 rewrites is pretty standard except for the Intel X25e which is 68k.

      There are 10,000 minutes in a week (roughly). You can burn through it very quickly. Or if you were to do it every 15 minutes you would burn through the rewrites in under 4 months and every 5 minutes in a bit over a month.

      You do have a good point though I hadn't thought about. If I expect the drive to last 5 years I can do 5 rewrites a day and still come in under 10,000. Or another way of thinking of that is I can rewrite 1% of the drive every 5 minutes and be fine. So as long as the drive is rotation materials around (which it didn't look they were) caching should be fine.

    16. Re:The logical next step... by Jah-Wren+Ryel · · Score: 1

      Assuming it's as good as the flash from that example, rewriting every hour results in 20 years. I don't know about you, but I don't have many hard drives from 20 years ago.

      It isn't as good. Back then the only kind of flash was SLC which gets roughly 1 million writes before death. Most consumer-grade flash drives today use MLC which is generally only good for an order of magnitude less writes.

      --
      When information is power, privacy is freedom.
    17. Re:The logical next step... by jbolden · · Score: 1

      I looked this up and 10k is out of date. It looks like 100k - 2m is the norm for new drives. That's probably fine for all but large scale date servers if you include wear leveling. People just don't flip enough data often enough.

    18. Re:The logical next step... by timeOday · · Score: 1

      SSD drives have been out for over a year now. If there were really an issue here, there would be thousands of reports from people with unusual usage patterns who write more in a year than most people write in 5. That hasn't happened. It's a non-issue.

    19. Re:The logical next step... by Lothsahn · · Score: 5, Informative

      I think you're somewhat close, but there are some inaccuracies...

      Block devices (typically HD's) have two operations (read and write). These operations are what most modern operating system use. Flash SSD's emulate a block device, but the underlying flash memory uses three operations (read, write, and erase). The main difference, therefore, between the block device (what the OS references) and the underlying flash itself is the extra erase operation.

      To write to a flash drive, assuming a cell has already been erased, all a user must do is a write operation. This operation is typically fast and does not affect the lifespan of the flash. A write can change any or all of the bits in a block from 1 to 0. Once this is complete, the requested data is written. However, if a user wants to overwrite or change existing data, they must first perform a block erase. This sets every bit in the block back to 1, and is typically very slow (compared to a write). In addition, this is what wears out the flash block, so we really want to avoid these operations.

      Since flash blocks each have their own lifespan, we want to spread the erase operations around the disk. This is called wear leveling. To do this, the flash device appears like a block device to the operating system, but it remaps where the data is actually located at the physical flash layer with a remap table. For instance, let's say you overwrite a block in Linux. If there is an available free flash block, it may not even overwrite that block--it may allocate a new block for the file and write it there (updating the remap table). This avoids an erase command. Furthermore, there are a few files on a filesystem which change frequently, and if we did not move their location around the physical flash, we would wear out one cell in flash extremely quickly, even though the remainder of the cells had plenty of life left.

      The garbage collection comes in due to this remapping. Typical block sizes for most OS filesystems are around 4k, but flash blocks are typically 512KB in flash devices. This means that if you send data to a SSD, it may or may not take up an entire page, as you may only be using 4k of actual data. Eventually, as writes are leveled around the drive and often fragmented (as we may not be occupying the entire 512KB block), future writes begin taking one (or more erase cycles). For instance, if you request that 512KB of data be written to the drive, but all the cells in the flash are physically occupied by a small amount of data, then data from multiple cells must be combined into one cell (multiple reads+erase+write), and then the destination cell that you are writing to must also be erased and written. This is what causes flash SSD's performance to significantly degrade over time.

      By performing this recombining in the background (as a garbage collection), this should allow flash SSD's to maintain like-new performance even when containing a lot of data. In essence, they are performing background defragmentation on the SSD. As a sidenote, NEVER defragment an SSD from the Operating System, as this defragments the filesystem, but performs a ton of erase+write operations to the flash. At best, on new SSDs (Intel, Indilinx), this will wear out the drive sooner. On old SSD's, this will also increase fragmentation at the flash remap layer, causing further performance loss.

      So to address your initial comment, rewrites would also see a performance increase by this garbage collection, as "rewriting" data in flash is virtually equivalent to a new write, since the remap table essentially moves the data anyway.

      Source:
      http://en.wikipedia.org/wiki/Flash_memory#Block_erasure

      --
      -=Lothsahn=-
    20. Re:The logical next step... by Jah-Wren+Ryel · · Score: 1

      Are you disputing my numbers about SLC and MLC?

      --
      When information is power, privacy is freedom.
    21. Re:The logical next step... by Amouth · · Score: 1

      "I imagine rewrites would stay comparatively slow, though."

      thats the extra beauty in this method.. a rewrite.. doesn't have to actualy rewrite the same block.. it can read the block merg and write to a "empty&wiped" block.. then go back and mark the source "empty" and let GC go back later and "&wiped".. meaning that your rewrites would speed up also .. as it wouldn't have to wipe the source block after reading it.

      --
      '...if only "Jumping to a Conclusion" was an event in the Olympics.'
    22. Re:The logical next step... by jbolden · · Score: 1

      Wiping near clean blocks still creates the same sort of situation the block gets wiped more frequently as it moves towards being more empty.

    23. Re:The logical next step... by jnork · · Score: 1

      "It would have to be done anyways before the sector can be reused."

      Quite correct. But there's more.

      If you don't know it's garbage you have to preserve it -- which costs another write for most of the cells. But if it's empty, the only cells being re-written are the ones that change. Writing FF over a cell that's been cleared (they clear to FF) doesn't cost anything, but writing anything else costs you a write cycle. So pre-clearing a garbage sector could save write cycles when you go to re-use part of it.

      I can see this actually saving write cycles in the long run.

      --
      Cleverly disguised as a responsible adult.
    24. Re:The logical next step... by shic · · Score: 1

      Thanks for your exposition, I found it clear and interesting.

      Specifically, I am very interested by the 512KB claim (I assumed, here, you meant KiB - i.e. 512*(2^10 bytes)... though that's not what interests me...)

      I've read various claims from extremely non-authoritative sources about 'block sizes' for flash devices - and I've read about everything from 512 bytes up-to 64KiB - and 512KiB is the largest I've heard about.

      It strikes me that, in the context of applications that store data on flash, where the application is free to define a storage format, that considerable performance optimisation is possible here. I'd love to find out:

      1. Do flash device manufacturers (especially those selling SD cards or USB sticks, as well as SSDs) document their block-size? If so, can anyone point me at such a document?

      2. Is there any 'standard' interface (on Windows or Unix) to reliably discover the block-size of an attached flash device?

      3. Is there any method - other than empiric experiment (i.e. timing lots of writes) - to determine the block size of flash devices?

      Thanks for any hints...

    25. Re:The logical next step... by Lothsahn · · Score: 1

      Shic:

      Yeah, I suppose I meant KiB, not technically KB. In general, which one KB refers to is unambiguous, and so I don't go through the trouble of looking up which specific term I need. :) If I ever write a hardware spec, though, I'll be sure to specify which one I mean!

      Good questions, unfortunately I'm afraid I may not have most of the information you're looking for...

      1. The answer to this would be that each manufacturer may or may not document this, at their whim. It would certainly be easy for them to do so, if they wanted. I don't know where any such documents are--I suspect that if Google doesn't know, you're kinda left in the dark. Also note that manufacturers may change these "implementation details" at any time, while not changing the model number (see also, WRT54G or almost any consumer-grade network product today). Many of these products have MAJOR internal details changed (flash memory, CPU speed, RAM, OS) without any major change in the model number. For instance a WRT54Gv1 contains twice the ram and flash and half the CPU of a WRT54Gv5, and it runs Linux instead of VxWorks. To test, I quickly googled the block size of the Intel SSD's and didn't see anything obvious in the first 30 seconds or so...

      2. None that I know of. Doesn't mean one doesn't exist. However, the vendor specific firmware tools may give such info. For instance, I got some information when I flashed my Intel SSD with the later firmware. I don't remember if block size was included.

      3. Absolutely. If you broke open the SSD and looked at the flash chips themselves, they would likely be printed with a model number. I'm 99% sure that Googling such a number would give you the exact flash manufacturer, block size, and a slew of other items about the chips. However, this would likely void warranty, so that may not recommended.

      In short, unless you're a geek and want to physically play with the device, the implementation details (block size, etc) often don't matter as much as one would think. For instance, Intel combines writes together, so even small writes go much faster than they may otherwise go. This means that the block size on these devices matters MUCH less than other SSD's. That being said, the block size is special--if we knew what the size actually was, it would allow us to set the block size in our OS filesystem to match, which would theoretically give improved performance.

      Also, if you're looking at a SSD purchase in the near future, I would avoid any product other than an Indilinx or Intel at the current time. Samsung SSD's are also decent, but I haven't found as much information about their design, specifically if their performance will degrade over time, and how much. I'm currently using a Generation 1 Intel SSD and it works very very well.

      HTH

      --
      -=Lothsahn=-
    26. Re:The logical next step... by shic · · Score: 1

      Thanks for your reply. I'm actually considering writing some software that needs non-volatile storage of blocks of data - where constant-time random-access (for read, and especially write) is very desirable. I am free (at this stage) to chose an arbitrary block size (or collection of supported block sizes) if it will make any difference. I'm expecting that it will make a difference (if I get the block-size right) since RAM-IO has a bandwidth significantly higher than Fash-IO... making flash IO the bottleneck that counts.

      The blocking approach of my application requires an entire block to be written (to a new location) - even if it is only 1 byte different to a previous version. A naive approach to optimisation might be to use a large block size to maximise throughput in the context of large changes... but, I strongly expect, this will only yield a benefit up-to a block size that matches the device itself.

      Another reason for wanting to match block sizes relates to reliability. As far as I'm concerned, if a device exhibits a single failure, I'd prefer it to fail to read a single application-level block rather than two blocks... hence demanding an application block size as a multiple of the device block size...

      A complication with respect to 'suck-it-and-see', for me, is that I'm running with USB devices (for now) making performance characteristics less obvious given a shared bus - and the possibility that IO bandwidth might be constrained by this interface rather than Flash itself.

      I'd definitely like to find more information (especially about cheap flash storage) - since I don't intend to use any conventional file-system when I use flash to store my blocked data... a file system like FAT would be an overhead I don't need or want for my (specific) purposes.

  2. Do cleanup in the OS by mrcaseyj · · Score: 1

    It seems like this function should be performed in the operating system. The firmware should just make available the info and commands an OS needs to do the right thing.

    1. Re:Do cleanup in the OS by mattventura · · Score: 4, Informative

      I think it ends up being like NCQ. The drive's processor can be much more specialized and can do the processing much more efficiently. Not to mention, it might require standards to be changed, since some busses (like USB, IIRC) don't provide commands to zero-out a sector on a low level. On an SSD, just writing a sector with zeros doesn't work the same as blanking the memory. It just makes the drive use a still-blank sector for the next write to that sector. The problem only comes when you run out of blank sectors.

    2. Re:Do cleanup in the OS by jtownatpunk.net · · Score: 1

      Um...How 'bout NO? This kind of thing absolutely should NOT be handled by the operating system. It should be entirely platform independent.

    3. Re:Do cleanup in the OS by fuzzyfuzzyfungus · · Score: 1

      I suspect that this is one of those "side effect of standardization" things: "SSDs", in this context, are quantities of flash with an embedded controller that pretends to be an ordinary magnetic drive(and, even there, most magnetic drives have been lying about their internal workings since shortly after MRL went out of style).

      If you want the OS to handle it, the option already exists. MTD, on the linux side(I assume WinCE has an equivalent, I've no idea what; the NT series OSes don't) is a mechanism for doing exactly that. The OS sees the "raw" flash chip directly, and you then run JFFs or another flash-optimized filesystem on top of it.

      On the plus side, this skips the cost, and power drain, of the dedicated controller(great for embedded systems), and allows these sorts of fiddly revisions to be made in software. On the minus side, you lose compatibility. Pretty much every motherboard ever knows how to talk to something that looks like a hard drive, and OSes are equally compliant. Not so much with raw flash chips... This is already common, if not dominant, on the embedded side; but I suspect that it is a fair ways off, if ever, on the normalish X86 side(though, curiously, the OLPC actually works this way).

    4. Re:Do cleanup in the OS by RiotingPacifist · · Score: 3, Interesting

      why? its low level but it doesn't affect the above filesystem.
      on the list of reasons why it SHOULD be done by the OS not the firmware are:
      *OS has a better clue about idleness
      *OS can create idleness by holding unimportant writes for a while (ext4 style) and using this time to do GC
      *OS can decide to save power by not doing this while on batterypower
      on the list AGAINST i only have:
      *jtownatpunk.net thinks it should be platform independent and thinks this can't be achieved without doing it in firmware

      put out the essence of the driver in public-domain and code a version for windows/mac if required, that way all oses will use the same logic even if they have completely different drivers.

      --
      IranAir Flight 655 never forget!
    5. Re:Do cleanup in the OS by v1 · · Score: 1

      Will the firmware in the drive be able to do this without understanding the filesystem?

      Just off the top of my head I can see where the onboard controller would have a big advantage. If we simplify the case and say the drive uses 2k blocks and the file system can't be modified to use 2k blocks, (lame!) then the onboard controller should watch for situations where a cell (of four 512 byte blocks) is frequently being reflashed because a single one of the four is being changed. Then if it could take a look at history and determine that somewhere else are three more blocks that always are getting changed at the same time, it could remap them to all four use the same flash block, so next time save a file, something that the controller has no higher understanding of, it only has to reflash one block instead of 2 (or 3 or 4).

      Of course ideally the OS could just be intelligent to be told the device uses 2k blocks instead of 512 byte which would make the above totally unnecessary.

      --
      I work for the Department of Redundancy Department.
    6. Re:Do cleanup in the OS by Hal_Porter · · Score: 3, Informative

      How does the firmware know what sectors are empty if it doesn't understand this stuff?

      I am curious how it works, if it doesn't need knowledge of the filesystem. FAT, NTFS, UFS, EXT2/3/4, ZFS, etc are all very different.

      The filesystem tells the SSD "LBA's x to y are now not in use" using the ATA trim command.

      http://www.theregister.co.uk/2009/05/06/win_7_ssd/

      Over-provisioned SSDs have ready-deleted blocks, which are used to store bursts of incoming writes and so avoid the need for erase cycles. Another tactic is to wait until files are to be deleted before committing the random writes to the SSD. This can be accomplished with a Trim operation. There is a Trim aspect of the ATA protocol's Data Set Management command, and SSDs can tell Windows 7 that they support this Trim attribute. In that case the NTFS file system will tell the ATA driver to erase pages (blocks) when a file using them is deleted.

      The SSD controller can then accumulate blocks of deleted SSD cells ready to be used for writes. Hopefully this erase on file delete will ensure a large enough supply of erase blocks to let random writes take place without a preliminary erase cycle.

      Actually I used to work on an embedded system that used M Systems' TrueFFS. There the flash translation layer actually understood FAT enough to work out when a cluster was freed. I.e. it knew where the FAT was and when it was written it would check for clusters being marked free at which point it would mark them as garbage internally.

      --
      echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
    7. Re:Do cleanup in the OS by maxwell+demon · · Score: 1

      On an SSD, just writing a sector with zeros doesn't work the same as blanking the memory.

      I think you mean a sector with all ones.
      Is there a reason why the firmware couldn't detect a write of an all-ones, and treat it as command to clear the sector instead of remapping it? Note that semantically, it would still do the same thing (i.e. the next read of that sector will indeed give an all-1), and since writes of all-1 blocks are otherwise rare, it wouldn't hurt performance of normal operation.

      --
      The Tao of math: The numbers you can count are not the real numbers.
    8. Re:Do cleanup in the OS by Trahloc · · Score: 2, Interesting

      Your right that the OS file system can do it more efficiently than the onboard SSD controller. Problem is, how many years do you want to wait until we get usable SSD's? Every OS would need their filesystem updated to speak to SSD drives or we present the same IDE/SATA/SCSI/SAS command interface to the OS like we have for decades and handle all the new specialized items internally. Which gets the product out there as quick as possible? When you've spent years R&Ding a product most companies would rather get it out there as quick as possible to recoup their costs and then fix bugs/errors as they go along rather than wait for some thirdparty they have no control over to do it for them ... someday ...

      While your solution is the *better* one, the current method is more practical and yes that matters. We can only hope that eventually SSD's create an alternate interface that allows the OS to bypass the SCSI/whatever interface and speak in raw SSD lingo to the device.

      --
      The Goal: A long simple life filled with many complex toys.
    9. Re:Do cleanup in the OS by smallshot · · Score: 2, Insightful
      I disagree with your list of reasons why the OS should do it:
      1. "idleness" should refer ONLY to the SSD not being used, not the OS being idle, in which case they both have approximately the same information.
      2. The OS should NOT hold unimportant writes for very long, and doing GC during this time does not change the possibility for important writes to be necessary
      3. I agree here. However, on battery power, I would expect an SSD would be sent to standby very quickly as if it were physically spinning, in which case the SSD firmware should know better than to do GC during standby, and with a definition of idle being 5+ minutes of zero-usage, it could be effectively the same as the OS not doing it on battery power. But I am only speculating

      On my list of AGAINST the OS doing GC i have:

      1. EVERY OS would have to support it in their next update to be as effective as having GC done in the firmware.
      2. OS software engineers are not likely to know the best way to design a GC for SSDs, leave it to the manufacturers who build the thing.

      There are some pros to letting the OS manage it, but I think it's riskier. I would suggest the firmware do the GC but have an interface for the OS to override the GC for flexibilities sake.

    10. Re:Do cleanup in the OS by KingMotley · · Score: 1

      Personally I will never buy a SSD disk until crap like wear leveling is unecessary and random write performance does not suck.

      Oh, so you bought a SSD 9 months ago?

    11. Re:Do cleanup in the OS by atamido · · Score: 1

      I was wondering if anyone was going to mention the TRIM command.

      If the OS and the SSD both support the TRIM command, the problem is solved. Next news item please.

  3. Who had to creative/hates "defragmentation"? by sznupi · · Score: 4, Insightful

    "Garbage collection" has already quite different usage in CS. And while what has to be done to those SSDs isn't technically the same as defragmentation on HDDs, it is still "performing drive maintenance to combat performance-degrading results of prolonged usage, deletion of files".

    --
    One that hath name thou can not otter
    1. Re:Who had to creative/hates "defragmentation"? by CountOfJesusChristo · · Score: 5, Informative

      So, I delete a file off of a drive such that the Filesystem no longer holds any references to the given data, and the firmware moves in and performs operations to improve the performance of the device. Its not really rearranging files in to contiguous sections like defragmentation does, its restoring unused sections to an empty state, probably using an algorithm similar to many garbage collectors -- sounds like garbage collection to me.

    2. Re:Who had to creative/hates "defragmentation"? by sznupi · · Score: 1

      I wonder if it can do that while block has data in some of it cells; or does it have to move, while idle, chunks of data from cells all around the disk to fill some other block and then restore, now empty of data, "garbage blocks" to pristine state...which has again some similarities to defragmentation.

      But really, I wasn't going so much into technical details, more into language conventions/familiarity. From the point of view of...almost everybody this new SSD mechanism is practically synonymous with defragmentation, even if concepts behind implementation would be more reminiscent of good old garbage collection (which I doubt; it doesn't free unneeded space, it repairs unused one) - the latter only helpful to firmware/drivers/filesystem writers.

      --
      One that hath name thou can not otter
    3. Re:Who had to creative/hates "defragmentation"? by natehoy · · Score: 2, Informative

      Right, but recall that SSD can only be erased in large blocks, though it can be written to in smaller ones. Erases are what eventually kill a block.

      So if I take a block that has only 25% garbage and I want to wipe it, I have to copy the good data over to another block somewhere before I can do that. So I've written 3/4 of a wipable sector's worth of data to a new sector to get rid of the 25% of garbage. Do that a lot, and you do a lot of unnecessary erases and the drive dies faster.

      If, instead, you take a sector that is 90% garbage, you only have to use 10% of a new sector to move off the good stuff before you can wipe it. So if you want the drive to last as long as possible, do garbage collection only when absolutely necessary.

      But allow garbage to grow too high, and you'll have to tell the operating system to wait while you rearrange data to make room when a write request comes in for a large file.

      Do you want the drive to be neatly optimized with no garbage all the time, or do you want the drive to last? I'm not saying one answer is more or less right than the other, but it's a tradeoff.

      --
      "This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
    4. Re:Who had to creative/hates "defragmentation"? by Bryan+Ischo · · Score: 1

      It's my understanding that the balance that SSD manufacturers like Intel have struck is a drive with excellent performance that is expected to life at least 3 or 4 years of heavy usage. Being extra conservative with erases at the expense of performance would increase the lifespan of the device, but in most circumstances, 3 or 4 years is good enough. Most people will have upgraded to a new drive by that point anyway. And quite a few hard drives die within that span as well.

    5. Re:Who had to creative/hates "defragmentation"? by vidnet · · Score: 1

      "performing drive maintenance to combat performance-degrading results of prolonged usage, deletion of files"

      This is what defragmentation does from the user's point of view, but not what it means. Defragmentation, as the word implies, is the process of reducing fragmentation, i.e. make files contiguous. This is achieved by moving chunks of data around, and the performance benefit comes from reducing the number of HD seeks required to read the data. Since a SSD is random-access, it doesn't have to seek so defragmentation is pointless. It just eats precious write cycles.

      Garbage collection, on the other hand, is the process of reclaiming memory that is no longer used and putting it back in the usable memory pool. This is exactly what this technology does.

    6. Re:Who had to creative/hates "defragmentation"? by sznupi · · Score: 1

      Well, it's a bit of a semantics issue (like my whole point ;p ), but...no, garbage collection isn't that. It is reclaiming memory that IS used, but no longer needed. In case of SSDs it's NOT putting it into usable pool, just making memory that isn't used already a bit more healthy.

      Furthermore, the mechanism by which SSD degrades is very similar to fragmentation.

      --
      One that hath name thou can not otter
    7. Re:Who had to creative/hates "defragmentation"? by complete+loony · · Score: 1

      Sure it does. Consider the following; Flash has 512K blocks that must be erased as a unit. Hard drives write in 4K blocks. A 512K block has been filled, and the OS then re-writes over some of those blocks. Now they can't be overwritten in place, they are instead remapped to another empty block. So now the first filled block has an empty hole in it that the SSD knows about, but can't use until the rest of the 512K block has been relocated.

      Sounds a bit like generational garbage collection to me.

      --
      09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
    8. Re:Who had to creative/hates "defragmentation"? by natehoy · · Score: 1

      Yup. Ideally, they'd have a slider you could set that indicates how aggressively you want to do data collection, with the warranty tied to where you set that slider. :)

      But it really depends on how much you use the drive (for writes/updates, I mean). Put a swap partition on SSD, and the drive's going to get burned up faster than an OS drive.

      --
      "This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
  4. when drives are "idle" ? by Gothmolly · · Score: 1

    If you have an app that needs SSD, when will the drives ever be idle ?

    --
    I want to delete my account but Slashdot doesn't allow it.
    1. Re:when drives are "idle" ? by basementman · · Score: 1

      They will be Idle whenever Slashdot needs something stupid to post.

      --
      A Magic the Gathering Article and Forum Aggregator
    2. Re:when drives are "idle" ? by Wesley+Felter · · Score: 1

      All PCs can benefit from SSDs, and they are often idle. Technology isn't just for those who "need" it.

    3. Re:when drives are "idle" ? by fuzzyfuzzyfungus · · Score: 1

      I (very strongly) suspect that any drives based on these indilinx controllers are not going to be aimed at the 24/7 database crowd, where your concern would definitely apply. In the "fancy laptop that makes no noise and doesn't die when you drop it", "l33t gaming box/workstation that boots in 15 seconds and loads Quake N/Maya in another 10", and similar, there will be periods of intense load; but also plenty of downtime.

      The indilinx guys are definitely a step above jmicron, who just suck, but tend to come in a decent bit below intel(though they are cheaper), much less the sort of stuff that Fusion IO is putting out(which is massively faster, and doesn't bother emulating a hard drive at all). Based on that, I suspect that they are aiming for the high end desktop/laptop/enthusiast segment.

    4. Re:when drives are "idle" ? by 644bd346996 · · Score: 2, Informative

      The drives don't have to be idle, just the portion being garbage collected. Flash drives typically consist of many independent memory chips united by a single drive. If the block being erased by the GC is on a chip that isn't being read from at that time, then the controller can issue the erase command without affecting the latency of any request from outside the drive. It would take a very full, random workload (and a very fast disk interface) to be able to detect the garbage collection, and even then, it couldn't be worse than the current method of erasing on an as-needed basis.

    5. Re:when drives are "idle" ? by tepples · · Score: 1

      If you have an app that needs SSD, when will the drives ever be idle ?

      When the battery on the device that runs the app is charging.

    6. Re:when drives are "idle" ? by tepples · · Score: 1

      The drives don't have to be idle, just the portion being garbage collected.

      It depends on how the drive controller maps the sector numbers. If the upper bits of the sector number select a chip, you have the equivalent of a JBOD array, and you can optimize the first 1/4 of the drive while the OS is doing stuff on the rest. But if the lower bits of the sector number select a chip, you have striping as in RAID 0, and no chip will be "idle" for any reasonable amount of time unless the whole array is idle. And then there are the tiny-form-factor (CompactFlash size) SSDs that may only have one flash chip.

  5. The cops'll love it. by domatic · · Score: 4, Interesting

    So what does this do when forensics are being done on one of these drives? Is the firmware just doing a better job of marking a dirty block available or do the dirty blocks have to be zeroed at some point. Even if the blocks are just marked will they output zeros if 'dd'ed by an OS?

    1. Re:The cops'll love it. by The+MAZZTer · · Score: 1

      I imagine they would still contain the data, but once you write to anywhere in that block the garbaged data would be zeroed out because it isn't being read into the "combine-read-data-with-new-to-be-written-data" buffer.

      Privacy advocates will love it.

    2. Re:The cops'll love it. by distantbody · · Score: 1

      SSD circuitry dictates the creation of pages and blocks.

      A page is the minimum amount you can write, say 4KB

      A block is the minimum you can delete, and it is made up of multiple pages, say 5.

      If a block contains pages to keep and pages to delete, the circuitry has to delete the whole block, then rewrite the valid pages just to clear the deletable pages. This takes more time.

      Current SDDs when told to delete a file just update the file table and leave the time-consuming delete task to when something needs to be written to a deleted but non-empty page.

      The TRIM command deletes blocks as the file is deleted from the file table, so the write operation doesn't have to wait around for the sequence of: cache-valid-pages>delete-block>rewrite-valid-pages>write-new-pages.

      So no, it wont make drive forensics easier, in fact it will remove any trace of a deleted file.

      My summation. It might be off the mark. See the great AnandTech SDD explanation, where I got the information from.

    3. Re:The cops'll love it. by Kjella · · Score: 1

      If you want a SSD wiped, best to use the manufacturer's low level format tools. There's literally gigabytes of extra space that might not be erased. If you wanted something kept safe it's generally better to keep it encrypted from the start anyway. I'm guessing forensic firms will do well on recovering from SSDs...

      --
      Live today, because you never know what tomorrow brings
    4. Re:The cops'll love it. by iwein · · Score: 1

      If I get this right the new firmware will erase unused blocks. Current SSD's will contain more interesting data than the ones with new firmware. I don't know if the firmware would erase all unused blocks if idle long enough, but it will at least erase a bit more than the current SSD's do.

      --
      Show a man some news, distract him for an hour. Show a man some mod points, distract him for the rest of his life.
  6. Leave the garbage (ie my porn) alone!!! by syousef · · Score: 4, Funny

    I don't want my porn garbage collected thank you very much. Who died and made you king of deciding what's garbage.

    --
    These posts express my own personal views, not those of my employer
  7. Filesystem info by wonkavader · · Score: 3, Interesting

    Wouldn't the drive benefit from a real understanding of the filesystem for this sort of thing? If it knew a sector was unallocated on a filesystem level, it would know that sectors were empty/unneeded, even if they had been written to nicely. Or should computers now have a way of tagging a sector as "empty" on the drive?

    Either way, it looks like an OS interaction would be very helpful here.

    Or are modern systems already doing this, and I'm just behind the times?

    1. Re:Filesystem info by blaster · · Score: 5, Informative

      There is an extensions that was recently added to ATA, the TRIM command. The TRIM command allows an OS to specify a blocks data is no longer useful and the drive should dispose of it. No productions support it, but several beta firmwares do. There are also patches for the Linux kernel that adds support to the black layer along with appropriate support to most filesystems. Windows 7 also has support for it.

      There is a lot of confusion about this on the OCZ boards, with people thinking GC somehow magically obviates the needs for TRIM. As you pointed out the GC doesn't know what is data and what is not with respect to deleted files in the FS. I wrote a blog post (with pictures and everything) explaining this just a few days ago

    2. Re:Filesystem info by Wesley+Felter · · Score: 3, Informative

      You're about two months ahead of the times. The ATA TRIM command will allow the filesystem to tell the SSD which sectors are used and which are unused. The SSD won't have to preserve any data in unused sectors.

    3. Re:Filesystem info by grotgrot · · Score: 1

      Wouldn't the drive benefit from a real understanding of the filesystem for this sort of thing?

      There is no need as a standard ATA TRIM command exists by which the OS can tell the device when a block is no longer in use. LWN wrote about this almost a year ago.

    4. Re:Filesystem info by Hal_Porter · · Score: 1

      Remember here is that you can't write 4KB to flash, only the 512KB cells.

      That's not correct. NAND flash has pages, typically 512 bytes to 4KB, and blocks, typically 128KB to 512KB. You can write any page you want. The limitation is that you can write a pages only once before you erase the block containing that page.

      --
      echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
    5. Re:Filesystem info by noidentity · · Score: 1

      Wouldn't the drive benefit from a real understanding of the filesystem for this sort of thing? If it knew a sector was unallocated on a filesystem level, it would know that sectors were empty/unneeded, even if they had been written to nicely. Or should computers now have a way of tagging a sector as "empty" on the drive?

      I have a radical concept: let the filesystem manage the drive, rather than making the drive do it. Isn't that the point of file systems, to provide optimal use of the media? Some filesystems used to schedule accesses that were near each other, to reduce head travel in a hard drive. Why not have them know about SSDs and do the read-rewrite themselves? It seems the problem is that there is an accepted interface to them that hides this information, thus requiring that the drive do it. At some point the interface should allow a filesystem to optionally take over this task.

    6. Re:Filesystem info by atamido · · Score: 1

      It's worth noting that while SSDs are just now trying to implement this, Linux and Windows 7 have supported it for at least 6 months, and its use has essentially been finalized for almost two years (draft in the specification). Adding this feature to SSDs is long overdue.

    7. Re:Filesystem info by bill_mcgonigle · · Score: 1

      Isn't that the point of file systems, to provide optimal use of the media?

      They want to sell these to people who run Windows.

      --
      My God, it's Full of Source!
      OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  8. At what cost? by igny · · Score: 3, Interesting

    The Garbage collector restores performance of the drive. Nothing comes free, so a question - at what cost?

    --
    In theory there is no difference between theory and practice. In practice there is. - Yogi Berra
    1. Re:At what cost? by slimjim8094 · · Score: 4, Insightful

      Possibly shorter drive life. If each cell can be rewritten 100,000 times (don't remember exactly) then - for exactly the same reason you're doing this in the first place (rewriting an entire cell on every write) you'll be wearing out the cell.

      Probably a net gain, though. This and wear-leveling algorithms probably will make drives last longer.

      Don't be quite so cynical. Usually I'd agree with you - but SSD (not flash) is so new that improvements can be made for free by just changing some techniques.

      --
      I have developed a truly marvelous proof of this comment, which this signature is too narrow to contain.
    2. Re:At what cost? by The+MAZZTer · · Score: 1

      The cost would be that it would have to keep track of which blocks of data are in use, so it would have to have a small bit of the SSD storage set aside for this purpose.

      There is no performance or lifespan penalty since this only affects what happens when data is written -- currently, the block is always read, combined, and then written. If the block is marked as not in use, the first two steps can be skipped. If the block is in use, we're just doing old behavior, no loss (except we needed to look up to see if the block was in use, but I doubt the performance loss would be noticeable).

    3. Re:At what cost? by natehoy · · Score: 5, Informative

      Simple. Well, not really, but...

      SSD's can be written to in small increments, but can only be erased in larger increments. So, you've got a really tiny pencil lead that can write data or scribble an "X" in an area to say the data is no longer valid, but a huge eraser that can only erase good-sized areas at a time, but you can't re-write on an area until it's been erased. There's a good explanation for this that involves addressing and pinouts of flash chips, but I'm going to skip it to keep the explanation simple. Little pencil lead, big eraser.

      Let's call the small increment (what you can write to) a "block" and the larger increment (what you can erase) a "chunk". There are, say, 512 "blocks" to a "chunk".

      So when a small amount of data is changed, the drive writes the changed data to a new block, then marks the old block as "unused". When all the blocks in a chunk are unused, the entire chunk can then be safely wiped clean. Until that happens, if you erase a chunk, you lose some data. So as time goes on, each chunk will tend to be a mix of current data, obsolete data, and empty blocks that can still be written to. Eventually, you'll end up with all obsolete data in each chunk, and you can wipe it.

      However, it's going to be rare that ALL the blocks in a chunk get marked as unused. For the most part, there will be some more static data (beginnings of files, OS files, etc) that changes less, and some dynamic data (endings of files, swap/temp files, frequently-edited stuff) that changes more. You can't reasonably predict which parts are which, even if the OS was aware of the architecture of the disc, because a lot of things change on drives. So you end up with a bunch of chunks that have some good data and some obsolete data. The blocks are clearly marked, but you can't write on an obsolete block without erasing it, and you can't erase a single block - you have to erase the whole chunk.

      To fix this, SSD drives take all the "good" (current) data out of a bunch of partly-used chunks and write it to a new chunk or set of chunks, then marks the originals as obsolete. The data is safe, and it's been consolidated so there are fewer unusable blocks on the drive. Nifty, except...

      You can only erase each chunk a certain number of times before it dies. Flash memory tolerates reads VERY well. Erases, not so much.

      So if you spend all of your time optimizing the drive, you're moving data around unnecessarily and doing a LOT of extra erases, shortening the hard drive's life.

      But if you wait until you are running low on free blocks before you start freeing up space (which maximizes the lifespan of the drive), you'll run into severe slowdowns where the drive has to make room for the data you want to write, even if the drive is sitting there almost empty from the user's perspective.

      So, SSD design has to balance between keeping the drive as clean and fast as possible at a cost of drive life, or making the drive last as long as possible but not performing at peak all the time.

      There are certain things you can do to benefit both, such as putting really static data into complete chunks where it's less likely to be mixed with extremely dynamic data. But overall, the designer has to choose somewhere on the continuum of "lasts a long time" and "runs really fast".

      --
      "This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
    4. Re:At what cost? by Anonymous Coward · · Score: 1, Interesting

      Possibly shorter drive life.

      -1, braindead. The block has to be cleared anyway before it can be reused. It's just being cleared in advance instead of at the time when it's needed. No extra wear is occurring.

    5. Re:At what cost? by bonch · · Score: 1

      The garbage collection is doing what already must be done before writing to a cell. This is just doing it at an earlier point when write performance is not a concern.

    6. Re:At what cost? by Anonymous Coward · · Score: 1, Insightful

      Needn't be. Suppose I have 128Kb free wherein I do garbage collect. I could still write a 4 block to it, if I had not done GC before.
      Thus, the shorter lifetime.

    7. Re:At what cost? by Hal_Porter · · Score: 1

      You drive works fine for a while. After that demons visit and eat you alive.

      --
      echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
    8. Re:At what cost? by angelbunny · · Score: 1

      No more un-erase. What is gone is really gone, there is no recovering lost data.

    9. Re:At what cost? by sl149q · · Score: 1

      Assuming 8GB flash, with a write speed of 10MB/s, 100000 erases allowed per sector, and a perfect wear leveling algorithm....

      Then your device will last (8 * 1024 * 1000000) / 10 / (3600 * 24) = 948.14 days... give or take

      Assuming you can write to it continuously at 10MB for that length of time (and assuming that the underlying hardware will do the required erases etc fast enough...)

      If you don't wear level and continuously erase same sector it will last about an hour or two.

      You *might* wear one out... but really its unlikely that many people will.

    10. Re:At what cost? by redhog · · Score: 1

      This sounds like something you could improve by something analogous to "generational GC"?

      --
      --The knowledge that you are an idiot, is what distinguishes you from one.
    11. Re:At what cost? by KingMotley · · Score: 1

      You obviously don't understand the technologies behind flash. There is no extra write.

      In either case, if you wrote to the non-GC'ed sectors, you need to do a sector clear, then write (because flash can only write to cleared sectors -- essentially). Although writes are really free, it's the clear that has a limited number of uses per sector. Reads are free, writes are free, clears are limited. You can't write to a non-clear sector, so for simplicity -- people say flash has a limited number of write cycles.

      Flash writes can only change bits from 0 to 1. If any bit in that sector is currently a 1, and needs to be a 0, then you have to clear the sector first. So if the sector is say 00000001, and you want to write 00010001, then you don't NEED a clear, but if it is currently 00010001, and you want to write 00000001, then you need to first clear the sector, and then write. The odds of any garbage sector having all the 1's in the correct spots so a new write to the same sector eliminates the need for a write is so small, that in most cases you might as well think of being able to write to the same sector multiple times as not being possible. Yes, there are cases in which it COULD happen, but it is so rare, that outside of the drives own housekeeping, I don't think the drive even considers it an option in most firmwares.

    12. Re:At what cost? by KingMotley · · Score: 1

      That should have read "eliminates the need for a clear is so small"

    13. Re:At what cost? by natehoy · · Score: 1

      Thanks for marking me as Foe, at least. It marks your comments so I can ignore them more easily.

      Have a great day.

      --
      "This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
  9. Damn it! by Anonymous Coward · · Score: 1, Funny

    The new drive deleted all my %INSERT_POPSTAR% songs!

    Captcha is "tragedy", how fitting...

  10. Re:Lifetime of drives by Wesley+Felter · · Score: 1

    Not necessarily. An SSD has to collect garbage sometime; whether it GCs proactively or lazily causes the same wear.

  11. Comment removed by account_deleted · · Score: 1

    Comment removed based on user account deletion

  12. Re:More wearing? by Anachragnome · · Score: 1

    Imagine that. Integrated obsolescence disguised as a feature.

    What ever will they think of next?

  13. Take out the "Tash"? by Opyros · · Score: 1

    Not on your life! Unless I've got Aslan to help me, of course.

    1. Re:Take out the "Tash"? by derGoldstein · · Score: 1

      aw man, you beat me to it. Still, Tash can mean more than one thing.

      What I'm inferring from the context within the sentence is that "take-out-the-tash" means he wishes to assassinate Rico Smith.

      --
      Entomologically speaking, the spider is not a bug, it's a feature.
  14. Check out NILFS by rrohbeck · · Score: 1

    No, not MILFs.

    http://www.linux-mag.com/cache/7345/1.html

    This GC stuff is only needed as long as the FS uses small blocks. File systems should be able to use arbitrarily sized blocks.
    Hopefully btrfs will be able to use large blocks efficiently too.

    --
    thegodmovie.com - watch it
    1. Re:Check out NILFS by derGoldstein · · Score: 1

      Glossary:
      GC: Garbage Collection
      FS: File System
      BTRFS: B-tree File System
      NILFS: New Implementation of a Log structured File System
      (I had to look NILFS up though...)

      --
      Entomologically speaking, the spider is not a bug, it's a feature.
    2. Re:Check out NILFS by swilver · · Score: 1

      IMHO, GC on SSD's is not going to be good enough, 512 kB blocks are way too large. Instead, they should build SSD's that can erase smaller blocks.

      There's good reason most filesystems donot use extremely large blocksizes:

      1) Using large block sizes means you reduce the maximum IO operations/second (something which SSD's are supposed to be good at). On traditional Hard Drives, you won't notice this until you go past 64 kB due to seek latency taking the largest part of such a transfer. On SSD's however you may notice this much earlier since there's supposed to be almost no seek time, meaning that the biggest advantage of SSD's (low latency) basically gets nullified if you started using insanely large blocks in your filesystem.

      2) Cached blocks will contain more junk that was never requested (a cache with 1k granularity can hold far more pertinent data than one with 64 kB blocks)

      3) Tail packing is no longer optional when using larger block sizes. Filesystems that donot support this therefore are unsuitable.

      4) In memory manipulation of large block sizes could also affect performance (like rehashing/checksumming data, re-arranging block contents) depending on algorithms used.

      5) Using larger blocksizes means the FS will likely rewrite such a block MORE often, as it contains more updatable data and more "free" space to add new data. Filesystems that never rewrite blocks won't be spared either as those blocks still need to be written somewhere (just not "overwritten") -- this makes keeping snapshots or going back in time a lot less efficient too. Writing such a large block will still require an Erase + Write, and always a full transfer of 512 kB (which takes 1.6 ms @ 300 MB/s -- quite close to the seek latency of traditional drives).

      There's probably more good reasons. I don't see the new filesystems fixing them all. Half a megabyte block sizes are really insanely large. It reminds me of FAT16 filesystems storing a 20 byte file in a 32k block -- your basically throwing around huge amounts of data for even the smallest of operations...

  15. Is this what dreams are for ? by Latinhypercube · · Score: 1

    Is this what dreams are for ?

    I feel like my brain goes 'garbage collecting' every night...

  16. Re:Lifetime of drives by natehoy · · Score: 1

    An SSD does have to collect garbage sometime, but waiting until an entire section is marked obsolete until you do the wipe is the cheapest/least wearing method. Remember - SSD can be written to in small increments, but only erased in large increments, and once an increment has been written to it must be erased to be written to again. And each increment can only be erased a certain number of times before it dies.

    Let's say we have erasable blocks of 512K with 512 writable sectors of 1K each. A proactive algorithm might move data into unused blocks when, say, half the block has "garbage" and the other half has good data. So when we hit more than 256K of garbage in the sector, we move the good stuff off it and wipe it.

    So we take the good data out of two blocks, and write it all to a single new block, then we can wipe both old blocks safely.

    But in doing so, we've "used up" a fresh block for no good purpose except to free up space. A noble goal, but I presume it's pretty obvious that if we had waited until each block was down to 128K of good data, we could have copied the good data out of FOUR old blocks to a single block, thereby cutting the "maintenance wear" by half.

    Garbage collection of an SSD drive is a continuous choice between:

      - allowing the drive to get clogged up with junk data and being as lazy about maintenance as possible (which statistically will extend drive life, but will mean there are times when you have to wait to do a write because the drive has to find the most-garbagey chunks available and clear them to make room). If you put this task off as long as possible, when you DO wipe a block it's more likely to be mostly garbage, so you're moving good data around as little as possible.

    and

      - obsessively keeping all garbage data off the drive by moving all data from ANY partially-used sectors to new sectors so all garbage can be kept out (which optimizes continuous performance, but means you may wipe an entire block when only a small percentage of it was actual garbage, so you're moving a lot of data around frequently and burning out sectors faster). If you do this task quickly and frequently, you move a lot of good data around from place to place. But you do ensure that free blocks are always available.

    The ideal is somewhere around the middle. You pick a point at which you consider a specific block to be "too garbagey" and you clean those up as you go. Where is that point? Depends on your needs, how full the drive really is, and your relative values on performance versus reliability or drive longevity.

    --
    "This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
  17. OCZ already released the GC tool, but for Win only by slagell · · Score: 2, Interesting

    I see OCZ already released some sort of garbage collection tool, but it only works on Windows. Kind of annoying since I bought their "Mac Edition" drive for my MacBook. Hopefully they'll put this in a firmware update, too, and hopefully I won't have to boot DOS on my Mac to update the firmware with a utility that blows over my partition table this time. That was a lot of fun going from version 1.10 to 1.30 firmware.

  18. Wrong data in article? by thePig · · Score: 2, Informative

    In the article it says

    But if a cell already contains some data--no matter how little, even if it fills only a single page in the block--the entire block must be re-written

    Is this correct?
    From whatever I read in AnandTech, it looked like we need not rewrite the entire block unless the available data is less than total - (obsolete + valid) data.

    Also, the article is light in details. How are they doing the GC? Do they wait till the overall performance of the disk is less than a threshold and rewrite everything in a single stretch or do they rewrite based on local optima? If the former, what sort of algorithms are used (and are the best for it) ?

    --
    rajmohan_h@yahoo.com
    1. Re:Wrong data in article? by blaster · · Score: 3, Informative

      No, what the actual situation is is that a block consists of some number of pages (currently on the flash used in SSDs it tends to be 128). The pages can be written individually, but only sequentially (so, write page 1, then page 2, then page 3), and the pages cannot be erased individually, you need to erase the whole block.

      The consequence of this is that when the FS says "Write this data to LBA 1000" the SSD cannot overwrite the existing page it is stored without erasing its block, so instead it find somewhere else to store it, and in its internal tables it marks the old page as invalid. Later when the GC is sweeping blocks for consolidation the number of valid pages is one of the criteria it uses to figure out what to do. If a block has very few valid pages and has been completely filled then those pages will probably be copied to another block that is mostly valid and the block the data was originally in will be erased.

  19. Re:OCZ already released the GC tool, but for Win o by Wesley+Felter · · Score: 2, Informative

    No, OCZ released wiper, which is a trim tool. Trim and GC are different; in particular, GC requires no tools or OS support.

  20. you're not moving the data unnecessarily by YesIAmAScript · · Score: 1

    Next time you write into that block, this operation will be performed anyway. This is why some SSDs have huge delays on writes, because they delay your write until the data merging is done. Also, not every block needs merging anyway, an area of a file that spans 512K (128 pages) is written in one chunk anyway and never needs re-merging.

    To be honest, the data retention time on NAND (where the data just drains out like DRAM) is becoming as big a factor as write wear anyway. You're going to have to move the data around a little to make sure it stays valid.

    Honestly, I'm pretty spooked about SSDs right now, the only reason I use one is a friend gave it to me for free to test.

    --
    http://lkml.org/lkml/2005/8/20/95
    1. Re:you're not moving the data unnecessarily by natehoy · · Score: 1

      The retention issue is a good one. I don't know how long the retention time is, or if the memory location can simply be "refreshed" with the same data rather than overwritten (with less damage)

      Keep in mind, though, the major difference between "lazy" and "active" maintenance (which is really a matter of degree, not an absolute) is how early you reclaim garbage sectors. If you take a sector that's nearly full of good data, then you have to "use up" an erase of almost a full sector to reclaim just a bit of empty space.

      Eventually, more of each sector should become garbage, so if you wait, you'll find sectors that have a higher percentage of garbage. So if you wait until you have ten sectors that are 90% garbage, you're only "using up" one sector to move the data off ten sectors that can now be reused.

      It's the moving of the good data that makes garbage collection accelerate the wear of the drive.

      --
      "This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
  21. Which department? by Voyager529 · · Score: 1

    from the take-out-the-tash dept.

    Is "tash" a play on words regarding SSD's, or does the taking-out-the-TRASH department have a job opening for a grammar nazi?

  22. Re:More wearing? by Voyager529 · · Score: 1

    What ever will they think of next?

    Why, they'll think of disguising integrated obsolescence as a feature next, but they'll call it a Data Relocation Mechanism (DRM).

  23. Oh Wow by Nom+du+Keyboard · · Score: 1

    Oh Wow, SSDs still have issues. Hey it's new technology, it's still very expensive relative to the technology it replaces, we haven't yet seen how well it holds up long term, and everyone who jumped to it early because it was the new bright and shiny thing should consider being a bit more cautious next time.

    --
    "It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
    1. Re:Oh Wow by Bryan+Ischo · · Score: 4, Informative

      You need to read up much, much more on the state of SSDs before making such sweeping, and incorrect, generalizations.

      There are algorithms in existence, such as clever "garbage collection" (which is a bad name for this process when applied to SSDs - it's only a bit like "garbage collection" as it is traditionally known as a memory management technique in languages like Java) combined with wear levelling algorithms, and having extra capacity not reported to the OS to use as a cache of "always ready to write to" blocks, that can keep SSD performance excellent in 90% of use cases, and very good in most of the remaining 10%. Point being that for the majority of use cases, SSD performance is excellent almost all of the time.

      Intel seems to have done the best job of implementing these smart algorithms in its drive controller, and their SSD drives perform at or near the top of benchmarks when compared against all other SSDs. They have been shown to retain extremely good performance as the drive is used (although not "fresh from the factory" performance, there is some noticeable slowdown as the drive is used, but it's like going from 100% of incredibly awesome performance to 85% of incredibly awesome performance - it's still awesome, just not quite as awesome as brand new), and except for some initial teething pains caused by flaws in their algorithms that were corrected by a firmware update, everything I have read about them - and I have done *alot* of research on SSDs, indicates that they will always be faster than any hard drive in almost every benchmark, regardless of how much the drive is used. And they have good wear levelling so they should last longer than the typical hard drive as well (not forever, of course - but hard drives don't last forever either).

      Indilinx controllers (which are used in newer drives from OCZ, Patriot, etc) seem to be second best, about 75% as good as the Intel controllers.

      Samsung controllers are in third place, either ahead, behind, or equal to Indilinx depending on the benchmark and usage pattern, but overall, and especially in the places where it counts the most (random write performance), a bit behind Indilinx.

      There are other controllers that aren't benchmarked as often and so it's not clear to me where they sit (Mtron, Silicon Motion, etc) in the standings.

      Finally, there's JMicron in a very, very distant last place. JMicron's controllers were so bad that they singlehandedly gave the entire early-generation SSD market a collective black eye. The one piece of advice that can be unequivically stated for SSD drives is, don't buy a drive based on a JMicron controller unless you have specific usage patterns (like, rarely doing writes, or only doing sequential writes) that you can guarantee for the lifetime of the drive.

      I've read many, many articles about SSDs in the past few months because I am really interested in them. Early on in the process I bought a Mtron MOBI 32 GB SLC drive (I went with SLC because although it's more than 2x as expensive as MLC, I was concerned about performance and reliability of MLC). In the intervening time, many new controllers, and drives based on them, have come out that have proven that very high performance drives can be made using cheaper MLC flash as long as the algorithms used by the drive controller are sophisticated enough.

      Bottom line: I would not hesitate for one second to buy an Intel SSD drive. The performance is phenomenal, and there is nothing to suggest that the estimated drive lifetime that Intel has specified is inaccurate. I would also happily buy Indilinx-based drives (OCZ Vertex or Patriot Torx), although I don't feel quite as confident in those products as I do in the Intel ones; in any case they all meet or exceed my expectations for hard drives. I've already decided that I'm never buying a spinning platter hard drive again. Ever. I have the good fortune of not being a movie/music/software pirate so I rarely use more than a couple dozen gigs on any of my systems anyway, so the smal

    2. Re:Oh Wow by lukas84 · · Score: 1

      I have an OCZ Vertex 120GB in my Laptop. Performance is okay, though not phenomenal.

      The new X25-M 34nm 160GB i bought for my Desktop on the other hand is awesome. Everything is near instantenous - it's like a new PC.

    3. Re:Oh Wow by amorsen · · Score: 1

      everyone who jumped to it early because it was the new bright and shiny thing should consider being a bit more cautious next time.

      The only regret I have about my X25-M is that I didn't get one when they first came out but waited till 6 months ago. The only comparable speed increase in Linux I have ever experienced was when I upgraded my parents' 486 from 8MB to 20MB RAM.

      --
      Finally! A year of moderation! Ready for 2019?
    4. Re:Oh Wow by bertok · · Score: 1

      You must have had a faulty setup in some way.

      I put an OCZ 256GB Vertex into my laptop, and it got so fast it's just unbelievable. I did have to upgrade the firmware, the stock firmware was both buggy and slow. Since then, several major revisions of the firmware have been released. I'll try this new GC firmware when I upgrade to Windows 7.

      Still, I have more confidence in the long term reliability of the Intel drives. A friend of mine purchased a Vertex drive as well, and it died a week after he got it, and mine had data corruption issues at first until I patched the firmware.

    5. Re:Oh Wow by lukas84 · · Score: 1

      I'm running Firmware 1.30 on my OCZ Vertex, which is the newest. I've been running Windows 7 RC, and updates to Windows 7 RTM yesterday, using a full format.

      I'm running Bitlocker on the laptop, which should not impact performance more than with the HDD when the SSD supports TRIM (which the Vertex with 1.30 should).

      Maybe "okay" was an understatement - it's certainly faster than with the previous 200GB 7.2kRPM HDD, but it wasn't the experience i got when upgrading to an X25-M on my desktop.

      FYI:
      Laptop: ThinkPad W500, 4GB, 2.53 Ghz C2D
      Desktop: Self-Built, 6GB, 2.66 Ghz i7

    6. Re:Oh Wow by timeOday · · Score: 1

      Oh Wow, SSDs still have issues

      Even with none of the proposed improvements, quality SSD's already slaughter mechanical drives' performance. It's the biggest single advance in mass storage since the PC was invented.

    7. Re:Oh Wow by bertok · · Score: 1

      Hmm... one thing that might be going on is that when you upgrade to an SSD, suddenly everything becomes CPU limited.

      I noticed that before my SSD, the CPU would only rarely go above 30% for any length of time, but now it hits 100% regularly.

      The Intel Core i7 in your desktop is at least 3x the speed of your laptop CPU: it has double the cores, and each core is more efficient.

      I saw a similar thing recently: I have a Vertex drive in a 2.66 GHz C2D laptop (4GB, running Windows 2008 x64), and a co-worker put an identical drive into a quad-core desktop. The desktop system was quite noticeably faster.

      We've all become used to CPU upgrades having little effect because of the bottleneck of the disk system, so when the disk IO issues go away, it can be surprising how much difference a decent CPU makes.

  24. What about Flash memory camcorders? by twosat · · Score: 1

    How would this affect camcorders that record to flash memory? I'm interested in getting a camcorder that uses flash memory for its inherent ruggedness and low power consumption. If the memory ages with time is it better not to get the ones with built-in memory that are not easily replaced by simply inserting a new memory stick? Does it mean that the memory at the beginning of the stick will age more, given that it will be overwritten more?

  25. So, in short, data gets jumbled.... by Tomsk70 · · Score: 1

    ....defrag, anyone?

  26. Impact on battry life? by howardd21 · · Score: 1

    I also wonder, does this impact battery life in a laptop with these SSDs?

    --
    no comment
  27. Misconceptions / errors in parent article by AllynM · · Score: 5, Informative

    I have been working closely with OCZ on this new firmware and wanted to clear things up a bit. This new firmware *does not*, *in any way at all*, remove or eliminate orphaned data, deleted files, or anything of the like. It does not reach into the partition $bitmap and figure out what clusters are unused (like newer Samsung firmwares). It does not even use Windows 7 TRIM to purge unused LBA remap table entries upon file deletions.

    What it *does* do is re-arrange in-place data that was previously write-combined (i.e. by earlier small random writes taking place). If data was written to every LBA of the drive, then all files were subsequently deleted, all data would remain associated with those LBAs. This actually puts OCZ above most of the pack, because their algorithm restores performance without needing to reclaim unused flash blocks, and does so completely independent of the data / partition type used. This is particularly useful for those concerned with data recovery of deleted files, since the data is never purged or TRIMmed.

    Slashdot-specific Translation: This firmware will enable an OCZ Vertex to maintain full speed (~160 MB/sec) sequential writes and good IOPS performance when used under Mac and Linux.

    Hardware-nut Translation: This firmware will enable OCZ Vertex to maintain full performance when used in RAID configurations.

    I'll have my full evaluation of this firmware up at PC Perspective later today. Once available, it will appear at this link:
    http://www.pcper.com/article.php?aid=760

    Regards,
    Allyn Malventano
    Storage Editor, PC Perspective

    --
    this sig was brought to you by the letter /.
    1. Re:Misconceptions / errors in parent article by atamido · · Score: 1

      Is there any information about when TRIM support will be added? It's nice to be able to undelete files, but most of us are fine with the consequences.

  28. You fail, Slashdot by yerktoader · · Score: 1

    An entire article and posts about garbage files and not one person mentions Ellingston Mineral and it's Gibson computers....I am dissapoint.

  29. Re:Ion.SIMIAN.c: Time to "face the music", once mo by Random+Destruction · · Score: 1

    Nothing like a crazy, off-topic, poorly written wall-of-text comment with scores of unnecessary bold and caps to get your point across.

    especially when the point is that you don't exist to write pointless wall-of-text-comments

    bravo, sir

    --
    :x
  30. Why not use indirection? by StCredZero · · Score: 1

    You'd need a separate memory to keep a table of pointers, just like an object table in some OO language implementations. This way, you wouldn't need Garbage Collection so much as just plain old compaction. The extra level of indirection wouldn't be so bad, because it's Flash -- random low-latency access is what it's good at.

    (The separate memory would have a granularity of just 64 bits, so rewriting just one pointer will have less overhead.)

    1. Re:Why not use indirection? by pdbaby · · Score: 1

      They already do this, as I understand it. The controller on the SSD is responsible for making data stored in various places on flash chips appear like a boring old block device

      --
      Global symbol "$deity" requires explicit package name at line 2. - If only $scripture started "use strict;"