Slashdot Mirror

← Back to Stories (view on slashdot.org)

The Lies Disks and Their Drivers Tell

Posted by Soulskill on from the designed-at-odds dept.

davecb writes "Pity the poor filesystem designer: they just want to know when their data is safe, but the disks and drivers try so hard to make I/O 'easy' that it ends up being stupidly hard. Marshall Kirk McKusick writes about the difficulties in making the systems work nicely together: 'In the real world, many of the drives targeted to the desktop market do not implement the NCQ specification. To ensure reliability, the system must either disable the write cache on the disk or issue a cache-flush request after every metadata update, log update (for journaling file systems), or fsync system call. Both of these techniques lead to noticeable performance degradation, so they are often disabled, putting file systems at risk if the power fails. Systems for which both speed and reliability are important should not use ATA disks. Rather, they should use drives that implement Fibre Channel, SCSI, or SATA with support for NCQ.'"

12 of 192 comments (clear)

  1. almost clicked the link... by adturner · · Score: 5, Funny

    But you lost me the moment you mentioned ATA drives.

    1. Re:almost clicked the link... by Lunix+Nutcase · · Score: 5, Insightful

      And yet fails to name any. Looking at Seagates site about NCQ pretty much every consumer model since 2004 has NCQ. This seems overblown.

    2. Re:almost clicked the link... by TheGratefulNet · · Score: 4, Informative

      yeah, well, I have quite a bit of experience with samsung (not seagate branded but the older samsungs) drives.

      they REPORTED having ncq but you always had to disable them.

      I got so that I do this at bootup:

      if [ -e /sys/block/sda/device/queue_depth ] ; then
            echo " sda NCQ now off"
            echo 1 > /sys/block/sda/device/queue_depth
      fi

      and so on.

      performance does not suffer (that I would care about) BUT the data reliab was more than making up for it. no more timeouts, no more syslog 'scaries'.

      vendors really do fuck up the protocol implementations. seagate is 'strange' in ways, so is WD, so is hitachi and ibm (I know they are not even in the biz anymore, at least for consumer drives).

      windows has a 'blacklist' of what things to not use when talking to drives and so does linux. its a fact of life.

      drive vendors are borderline idiots. sad but true ;(

      --

      --
      "It is now safe to switch off your computer."
    3. Re:almost clicked the link... by Eponymous+Hero · · Score: 5, Informative
      you didn't bother to RTFA, good for you. it says quite plainly that (only part of) the problem is not drives that don't support ncq, but those drives that have it and disable it. and that was a relatively small portion of TFA. here's how the disks lie:

      File systems need to be aware of the change to the underlying media and ensure that they adapt by always writing in multiples of the larger sector size. Historically, file systems were organized to store files smaller than 512 bytes in a single sector. With the change in disk technology, most file systems have avoided the slowdown of 512-byte writes by making 4,096 bytes the smallest allocation size. Thus, a file smaller than 512 bytes is now placed in a 4,096-byte block. The result of this change is that it takes up to eight times as much space to store a file system with predominantly small files. Since the average file size has been growing over the years, for a typical file system the switch to making 4,096 bytes the minimum allocation size has resulted in a 10- to 15-percent increase in required storage.

      just to clarify what the author's point was:

      The conclusion is that file systems need to be aware of the disk technology on which they are running to ensure that they can reliably deliver the semantics that they have promised. Users need to be aware of the constraints that different disk technology places on file systems and select a technology that will not result in poor performance for the type of file-system workload they will be using. Perhaps going forward they should just eschew those lying disks and switch to using flash-memory technology—unless, of course, the flash storage starts using the same cost-cutting tricks.

      if you want to argue that, great, go nuts. nobody who actually RTFA thinks the argument is really about ncq. the ac you responded to said

      the way I interpret TFA, the problem also applies to SATA drives which do not implement the NCQ specification.

      well, here's what TFA actually said:

      Luckily, SATA (serial ATA) has a new definition called NCQ (Native Command Queueing) that has a bit in the write command that tells the drive if it should report completion when media has been written or when cache has been hit. If the driver correctly sets this bit, then the disk will display the correct behavior.

      In the real world, many of the drives targeted to the desktop market do not implement the NCQ specification. To ensure reliability, the system must either disable the write cache on the disk or issue a cache-flush request after every metadata update, log update (for journaling file systems), or fsync system call. Both of these techniques lead to noticeable performance degradation, so they are often disabled, putting file systems at risk if the power fails. Systems for which both speed and reliability are important should not use ATA disks. Rather, they should use drives that implement Fibre Channel, SCSI, or SATA with support for NCQ

      i hope it's painfully obvious by now that the point about ncq is not that some drives don't have it; it's that some don't use it -- mostly so you don't go giving their drives bad reviews for being slow but unnoticeably reliable. if it's disabled, you can enable it. what sata drives don't have ncq? i asked wikipedia:

      SATA revision 1.0 (SATA 1.5 Gbit/s) .... During the initial period after SATA 1.5 Gbit/s finalization, adapter and drive manufacturers used a "bridge chip" to convert existing PATA designs for use with the SATA interface. Bridged drives have a SATA connector, may include either or both kinds of power connectors, and, in general, perform identically to their PATA equivalents. Most lack support for some SATA-specific features such as NCQ. Native SATA products quickly eclipsed bridged products with the introduction of the second generation of SATA drives.

      so yeah, probably not a whole lot of these drives being sold new, but there are lots of shops that buy used gear because it's cheap. these older sata drives haven't all just disappeared when revision 2.0 came out.

      --
      insensitive clod overlords obligatory xkcd car analogy russian reversals whoosh pedant fanbois ftfy in 3...2...1..PROFIT
    4. Re:almost clicked the link... by hoggoth · · Score: 5, Insightful

      LOSE LOSE LOSE LOSE! YOU WILL LOSE DATA!

      Sorry... I'm usually a calm rational person. I almost never become a grammar-nazi, spelling nazi, or troll. It's just that I see this so often I'm afraid one day Webster will just give up and switch the definitions of Lose and Loose.

      --
      - For the complete works of Shakespeare: cat /dev/random (may take some time)
    5. Re:almost clicked the link... by anomaly256 · · Score: 4, Informative

      Green drives from Seagate do not appear to have NCQ. As per below, I have 1 normal and 4 greens in this box:

      ~$ cat /sys/block/sd?/device/queue_depth
      31
      1
      1
      1
      1

      ~$ cat /sys/block/sd?/device/queue_type
      simple
      none
      none
      none
      none

    6. Re:almost clicked the link... by hoggoth · · Score: 4, Funny

      "and then..."

      and then all hell will break lose, obviously.

      --
      - For the complete works of Shakespeare: cat /dev/random (may take some time)
  2. 2 out of 3 by ardmhacha · · Score: 4, Insightful

    Cheap, fast and reliable.

    Pick any two.

  3. Not about ATA, about enterprise data storage by MSTCrow5429 · · Score: 4, Informative
    1) This article isn't about ATA, ignore it.

    2) The article's point on NCQ is that many consumer drives do not implement it correctly, and disable the write cache on the disk and issue cache-flush requests to increase performance, but leading to possible file-system failures if there is a power outage.

    I think this article is saying that for the enterprise, buy enterprise drives, not consumer drives. Most consumers use laptops now, so power failure doesn't fit in, and consumers prefer speed over reliability, which is why I've always been stuck using laptops lacking ECC RAM.

    --
    Slashdot: Playing Favorites Since 1997
    1. Re:Not about ATA, about enterprise data storage by ChumpusRex2003 · · Score: 5, Informative

      The "Turn off Windows write-cache buffer flushing on the device" option activates an ancient windows bug, and should never be used.

      When Windows 3.11 was released, MS accidentally introduced a bug, whereby a call to "sync" (or whatever the windows equivalent was called) would usually be silently dropped. At the time, a few programmers noticed that their file I/O appeared to have improved, and attributed this to MS's much marketed new 32-bit I/O layer. What a lot of naive developers didn't notice was that the reason their I/O appeared to be faster was that the OS was handling file steams in an aggressive write-back mode, and then calls to "sync" were being ignored by the OS.

      Because of this, there was a profusion of office software, in particular, accounting software, which would "sync" frequently - some packages would call "sync" on every keypress, or everytime enter was pressed, or the cursor moved to the next data entry field. As on 3.11, this call was effectively a NOP, a lot of packages made it onto client machines, and because it was fast, no one noticed.

      With Win95, MS fixed the bug. Suddenly, corporate offices around the world had their accounting software reduced to glacial speed, and tech support departments at software vendors rapidly went into panic mode. Customers were blaming MS, Win95 was getting slated, lawyers were starting to drool, etc. Developers were calling senators and planning anti-trust actions. The whole thing was getting totally out of hand.

      In the end, MS decided the only way to deal with this bad PR, was to put an option into windows, where the bug could be reproduced for software which depended upon it. The option to activate the bug was hidden away reasonably well, in order to stop most people from turning it on, and running their file-system in a grossly unstable mode. However, in Win95 - Vista, it had a rather cryptic name "Advanced performance", which meant that a lot of hardware enthusiasts would switch it on, in order to improve performance, without any clear idea of what it did. At least in Win7 it now has a clear name, even though it still doesn't make clear that it should only be used for when using defective software.

  4. NCQ - Native Command Queueing by wonkey_monkey · · Score: 4, Informative

    Native Command Queueing

    Because not everybody knows everythingTM

    --
    systemd is Roko's Basilisk.
  5. Get Hardware RAID by FranTaylor · · Score: 4, Insightful

    The people who make hardware RAID know all about the lying drives, they get good information from the manufacturer on how to make the drives play nice with the RAID controller.

    Just read the compatibility charts for your RAID controller, many drives have footnotes with minimum drive firmware requirements and other odd behavior.