TRIM and Linux: Tread Cautiously, and Keep Backups Handy

An anonymous reader writes: Algolia is a buzzword-compliant ("Hosted Search API that delivers instant and relevant results") start-up that uses a lot of open-source software (including various strains of Linux) and a lot of solid-state disk, and as such sometimes runs into problems with each of these. Their blog this week features a fascinating look at troubles that they faced with ext4 filesystems mysteriously flipping to read-only mode: not such a good thing for machines processing a search index, not just dishing it out. "The NGINX daemon serving all the HTTP(S) communication of our API was up and ready to serve the search queries but the indexing process crashed. Since the indexing process is guarded by supervise, crashing in a loop would have been understandable but a complete crash was not. As it turned out the filesystem was in a read-only mode. All right, let's assume it was a cosmic ray :) The filesystem got fixed, files were restored from another healthy server and everything looked fine again. The next day another server ended with filesystem in read-only, two hours after another one and then next hour another one. Something was going on. After restoring the filesystem and the files, it was time for serious analysis since this was not a one time thing.

The rest of the story explains how they isolated the problem and worked around it; it turns out that the culprit was TRIM, or rather TRIM's interaction with certain SSDs: "The system was issuing a TRIM to erase empty blocks, the command got misinterpreted by the drive and the controller erased blocks it was not supposed to. Therefore our files ended-up with 512 bytes of zeroes, files smaller than 512 bytes were completely zeroed. When we were lucky enough, the misbehaving TRIM hit the super-block of the filesystem and caused a corruption."

Since SSDs are becoming the norm outside the data center as well as within, some of the problems that their analysis exposed for one company probably would be good to test for elsewhere. One upshot: "As a result, we informed our server provider about the affected SSDs and they informed the manufacturer. Our new deployments were switched to different SSD drives and we don't recommend anyone to use any SSD that is anyhow mentioned in a bad way by the Linux kernel."

Comments

Maybe we can create a superior alternative to TRIM

[Rinikusu]

I suggest we call it SNATCH.

Is there a site maintaining a list of "bad" SSDs?

[msobkow]

I'll Google in a moment, but I was wondering if anyone knew of any good sites that maintain lists of good/bad SSDs for Linux. With the number of vendors out there nowadays, having to scan the source seems like a poor way to track the information.

Apple

[Anonymous Coward]

This is why Apple doesn't support TRIM in third-party SSDs...

Re:Apple

[Lumpy]

And it's trivial to get around.

Re:Apple

[drinkypoo]

Sure, if you want to roll the dice.

You roll the dice when you buy Apple equipment, just like everything else. They've had their hardware failures, and their design failures too. They simply put out less models than other companies do.

Re:Apple

[PlusFiveTroll]

This bug has nothing to do with standard TRIM. Send TRIM as a single command after flushing the drive queue and it works fine. This has to do with the newest SATA specification allowing queued TRIM blowing up. Apple just wants to sell their more expensive drives.

TRIM -- command of mass destruction

[m.dillon]

The only TRIM use I recommend is running on it on an entire partition, e.g. like the swap partition, at boot, or before initializing a new filesystem. And that's it. It's an EXTREMELY dangerous command which results in non-deterministic operation. Not only do SSDs have bugs in handling TRIM, but filesystem implementations almost certainly also have ordering and concurrency bugs in handling TRIM. It's the least well-tested part of the firmware and the least well-tested part of the filesystem implementation. And due to cache effects, it's almost impossible to test it in a deterministic manner.

You can get close to the same performance and life out of your SSD without using TRIM by doing two simple things. First, use a filesystem with at least a 4KB block size so the SSD doesn't have to write-combine stuff on 512-byte boundaries. Second, simply leave a part of the SSD unused. 5% is plenty. In fact, if you have swap space configured on your SSD, that's usually enough on its own (since swap is not usually filled up during normal operation), as long as you TRIM it on boot.

-Matt

Re:TRIM -- command of mass destruction

[Anonymous Coward]

Man Linux users are hilarious. TRIM has worked and been safe on every other platform for ages.

LOL.

Do you know who you're replying to? Matt Dillon is the principal developer of DragonflyBSD, and the HAMMER fileystem.

While he probably does use Linux from time to time, I think you're more likely to find him at a BSD system.

Re:TRIM -- command of mass destruction

[cmaxx]

*So* not a kludge. o.O

Historically TRIM was a smart addition to block-storage commands in the enterprise SAN space, to better enable things like thin-provisioning, which at least at the time went with server virtualisation like eggs go with bacon.

It just-so-happened that TRIM also fulfilled a very similar, and similarly smart, use-case on flash-based SSDs of all levels soon after/at the same time.

Given the characteristics of the current flash devices in SSDs, it's a perfectly reasonable thing to add to filesystems, SANs and drives. It's vitally important that it be correctly implemented, but frankly, that's true for everything in IT.

Re:TRIM -- command of mass destruction

[tlhIngan]

No, TRIM support doesn't matter to SSDs.

TRIM is an optimization that when properly used, can make an SSD faster. Emphasis on can, and properly used. It can also be used to reduce write amplification (the ratio of number of writes to flash over the number of writes the host computer actually did. More later).

You can use an SSD with a non-TRIM aware OS and it will work just fine.

In an SSD, you have pages which are mapped to sectors. As you overwrite a logical sector, the wear levelling algorithm marks the old pages as "dirty" and stores the updated contents in a new page elsewhere in the storage array. If you repeated hammer a sector, that sector's contents will be repeated across the array - there will only be one valid sector, and a bunch of older ones marked as dirty.

Depending on SSD load, a garbage collector may have enough time to run, and it will look at each flash block, move all the pages still valid in it elsewhere, then erase the block and make it ready for use. It's not important this runs as if you issue a write and there's no free blocks, the flash controller will migrate valid sectors elsewhere and erase a block on demand.

Note I said valid sectors - since a modern flash array may have many (128 or more) 512 byte sectors mapped to a block, rarely will all sectors be dirty at once, and long-standing data will often need to be moved elsewhere so a block can be freed up. But moving a sector incurs a write into the array, which means if you wrote the sector once at the beginning, and never touched it again, that sector may be rewritten multiple times throughout the SSD's life - so a 512 byte sector write may incur many megabytes of writes as it's moved throughout the flash array. That is write amplification - you only wrote 512 bytes, but incurred megabytes of wear on the flash. (Ideally you want 1.0 - every sector you write is written to the array exactly once)

Now let's examine what happens with TRIM. When you delete a file, the OS merely updates a few sectors that free up the directory entry and add the disk blocks to the free bitmap. As the OS writes to the drive, the sectors belonging to the deleted file get moved about (the SSD doesn't know which sectors it no longer needs to move around), incurring write amplification and making the SSD a bit slower as it has to move sectors that the OS isn't really using.

That's where TRIM comes in - when you delete a file, TRIM tells the SSD that the sector belongs to a deleted file, and the SSD will treat that sector as empty until the OS writes to it. It will not move the sector elsewhere as the OS said its contents were no longer relevant. This means the SSD won't need to move the contents of a deleted file around, lowering write amplification, as well as saving the need to move that sector, making the SSD faster.

There is nothing in an SSD that requires TRIM - just that TRIM support in an OS and SSD makes storage faster and last longer. Early SSDs lacked TRIM, as did earlier OSes, so you need to be able to treat an SSD like a regular block device.

In fact, in older SSDs, they often got slower as time went on because they were moving around deleted data - the only way to speed them up was to use an ATA_SECURE_ERASE which basically reset the entire array. Nowadays most OSes will do an equivalent TRIM if you delete the partition.

Oh yeah, because of this, SSDs are terrible for forensics...

Name and shame

[Anonymous Coward]

see ata_blacklist_entry

(reformatted to get past Slashdot's 'junk' filter)

static const struct ata_blacklist_entry ata_device_blacklist [] = {
see ata_blacklist_entry

static const struct ata_blacklist_entry ata_device_blacklist [] = /* Devices with DMA related problems under Linux */
WDC AC11000H, NULL, ATA_HORKAGE_NODMA ,
WDC AC22100H, NULL, ATA_HORKAGE_NODMA ,
WDC AC32500H, NULL, ATA_HORKAGE_NODMA ,
WDC AC33100H, NULL, ATA_HORKAGE_NODMA ,
WDC AC31600H, NULL, ATA_HORKAGE_NODMA ,
WDC AC32100H, 24.09P07, ATA_HORKAGE_NODMA ,
WDC AC23200L, 21.10N21, ATA_HORKAGE_NODMA ,
Compaq CRD-8241B, NULL, ATA_HORKAGE_NODMA ,
CRD-8400B, NULL, ATA_HORKAGE_NODMA ,
CRD-848[02]B, NULL, ATA_HORKAGE_NODMA ,
CRD-84, NULL, ATA_HORKAGE_NODMA ,
SanDisk SDP3B, NULL, ATA_HORKAGE_NODMA ,
SanDisk SDP3B-64, NULL, ATA_HORKAGE_NODMA ,
SANYO CD-ROM CRD, NULL, ATA_HORKAGE_NODMA ,
HITACHI CDR-8, NULL, ATA_HORKAGE_NODMA ,
HITACHI CDR-8[34]35,NULL, ATA_HORKAGE_NODMA ,
Toshiba CD-ROM XM-6202B, NULL, ATA_HORKAGE_NODMA ,
TOSHIBA CD-ROM XM-1702BC, NULL, ATA_HORKAGE_NODMA ,
CD-532E-A, NULL, ATA_HORKAGE_NODMA ,
E-IDE CD-ROM CR-840,NULL, ATA_HORKAGE_NODMA ,
CD-ROM Drive/F5A, NULL, ATA_HORKAGE_NODMA ,
WPI CDD-820, NULL, ATA_HORKAGE_NODMA ,
SAMSUNG CD-ROM SC-148C, NULL, ATA_HORKAGE_NODMA ,
SAMSUNG CD-ROM SC, NULL, ATA_HORKAGE_NODMA ,
ATAPI CD-ROM DRIVE 40X MAXIMUM,NULL,ATA_HORKAGE_NODMA ,
_NEC DV5800A, NULL, ATA_HORKAGE_NODMA ,
SAMSUNG CD-ROM SN-124, N001, ATA_HORKAGE_NODMA ,
Seagate STT20000A, NULL, ATA_HORKAGE_NODMA ,
2GB ATA Flash Disk, ADMA428M, ATA_HORKAGE_NODMA , /* Odd clown on sil3726/4726 PMPs */
Config Disk, NULL, ATA_HORKAGE_DISABLE , /* Weird ATAPI devices */
TORiSAN DVD-ROM DRD-N216, NULL, ATA_HORKAGE_MAX_SEC_128 ,
QUANTUM DAT DAT72-000, NULL, ATA_HORKAGE_ATAPI_MOD16_DMA ,
Slimtype DVD A DS8A8SH, NULL, ATA_HORKAGE_MAX_SEC_LBA48 ,
Slimtype DVD A DS8A9SH, NULL, ATA_HORKAGE_MAX_SEC_LBA48 , /* Devices we expect to fail diagnostics */ /* Devices where NCQ should be avoided */ /* NCQ is slow */
WDC WD740ADFD-00, NULL, ATA_HORKAGE_NONCQ ,
WDC WD740ADFD-00NLR1, NULL, ATA_HORKAGE_NONCQ, , /* http://thread.gmane.org/gmane.linux.ide/14907 */
FUJITSU MHT2060BH, NULL, ATA_HORKAGE_NONCQ , /* NCQ is broken */
Maxtor *, BANC*, ATA_HORKAGE_NONCQ ,
Maxtor 7V300F0, VA111630, ATA_HORKAGE_NONCQ ,
ST380817AS, 3.42, ATA_HORKAGE_NONCQ ,
ST3160023AS, 3.42, ATA_HORKAGE_NONCQ ,
OCZ CORE_SSD, 02.10104, ATA_HORKAGE_NONCQ , /* Seagate NCQ + FLUSH CACHE firmware bug */
ST31500341AS, SD1[5-9], ATA_HORKAGE_NONCQ |
ATA_HORKAGE_FIRMWARE_WARN ,
ST31000333AS, SD1[5-9], ATA_HORKAGE_NONCQ |
ATA_HORKAGE_FIRMWARE_WARN ,
ST3640[36]23AS, SD1[5-9], ATA_HORKAGE_NONCQ |
ATA_HORKAGE_FIRMWARE_WARN ,
ST3320[68]13AS, SD1[5-9], ATA_HORKAGE_NONCQ |
ATA_HORKAGE_FIRMWARE_WARN , /* Seagate Momentus SpinPoint M8 seem to have FPMDA_AA issues */
ST1000LM024 HN-M101MBB, 2AR10001, ATA_HORKAGE_BROKEN_FPDMA_AA ,
ST1000LM024 HN-M101MBB, 2BA30001, ATA_HORKAGE_BROKEN_FPDMA_AA , /* Blacklist entries taken from Silicon Image 3124/3132
Windows driver .inf file - also several Linux problem reports */
HTS541060G9SA00, MB3OC60D, ATA_HORKAGE_NONCQ, ,
HTS541080G9SA00, MB4OC60D, ATA_HORKAGE_NONCQ, ,

Re:Name and shame

[Rockoon]

The way I am reading the comments, the issue is that the buggy SSD's are flagging physical blocks as RZAT or DRAT when a trim request on a logical block is ignored. The bug presents itself later if the SSD performs wear leveling that swaps out the logical block with another, the bug being leaving the physical block tagged RZAT or DRAT.

TLDR

[Anonymous Coward]

Don't buy Samsung SSDs.

Apple TRIM Whitelist?

[LDAPMAN]

I wonder if this issue has anything to do with why Apple only supports TRIM on specific drives they OEM?

Re:Apple TRIM Whitelist?

[Anonymous Coward]

Yes. Apple also has custom firmware to support temperature sensors (instead of just using the standard SMART commands), which is why the fans in their iMacs/Macbooks will run at full speed if you swap out their drive for a 3rd party one... OS X assumes that the sensor is broken and goes into safe mode to avoid an overtemp burnout.

Re:Apple TRIM Whitelist?

[Anonymous Coward]

Its a good bet.

As apple is probably quite aware, being probably the biggest seller of non-windows PCs, there is an endemic problem with a whole lot of hardware shipped claiming to be "compliant" with any given standard.

Most vendor's testing methodology pretty much comes down to "Works on windows? Ship it"

Linux has been dealing with this problem for decades. Power management implementations in laptops (and some desktop motherboards) are often outright broken and don't behave anything close to what the "standard" dictates. (Its so bad in laptops that Microsoft's power management maintains a hardware checklist with custom hacks for laptops with known bad implementations. On many systems it does not even /attempt/ to use standard calls)

Linux developers attempt to access hardware in a manner according to how documented standards state and end up tripping all sorts of bugs from mild to hardware-bricking. Flabbergasted hardware vendors often respond with "It works in windows!"

(Fortunately this shit doesn't fly in the server space where Linux is now pretty much King.. Well, at least in theory)

So yes, I'd be willing to bet that Apple found that enabling trim in any old SSD led to an unacceptable chance of filesystem corruption and decided to implement a white list. So, you know, they don't catch shit for someone else's broken hardware.

Re:Is there a site maintaining a list of "bad" SSD

[Ken_g6]

It takes a couple of links and searching through source code to get there. So here's the list of problematic drives, better formatted but still in regular expression format:

/* devices that don't properly handle queued TRIM commands */
Micron_M500*
Crucial_CT*M500*
Micron_M5[15]0*
Crucial_CT*M550*
Crucial_CT*MX100*
Samsung SSD 8*

So, basically, all the ones I thought were the best. The list of whitelisted drives after it only includes those brands, Intel, and ST-something. So other brand may be unknowns.

Re:trim

[Nkwe]

"we don't recommend anyone to use any SSD that is anyhow mentioned in a bad way by the Linux kernel"

???? SERIOUSLY???

While poorly written, I think the author was suggesting that any model of SSD for which the Linux kernel has specific special handling logic should be avoided. In my opinion, it is not an unreasonable statement.

Re:Is there a site maintaining a list of "bad" SSD

[Anonymous Coward]

The Crucial MX100 with the latest MU02 firmware is now whitelisted by the Linux Kernel, and has it's TRIM ability re-enabled.

Re:And the offending SSD drives are....

[ihtoit]

it's a long list, apparently... the whitelist is shorter. I'd go with the Apple one. They seem to have their QA department high and tight.

Another Deceptive Slashdot Title

[idontgno]

Correct title: "TRIM and Any Fucking Operating System: Don't Buy Defective SSDs"

It's not as if Windows or MacOS has any magic that makes queued TRIM work with non-compliant and poorly-coded hardware, right?

Seriously, WTF, over?

Re:Another Deceptive Slashdot Title

[Anonymous Coward]

Windows and MacOS do not issue Queued TRIM in the first place. They only issue the regular TRIM command, which has to stop all data in flight and quiesce the entire submission queue (all tags, etc).

Linux is ultra-high-IO-load optimized, queued TRIM is a must when dealing with high-performance storage (not just SSDs). Maybe it should stop trusting devices that are neither attached to a SAS or FC transport by default when they claim to actually implement advanced features, though.

Re:Another Deceptive Slashdot Title

[Trogre]

Dear Microsoft,

Thank you for your generous donation to our staff social club. As promised, please find attached drivers that utilise the *real* TRIM commands for our SSDs.

Sincerely yours,
A. Manufacturer

Re:Is there a site maintaining a list of "bad" SSD

[idontgno]

ObPedant: those aren't regexes, they're globs. Otherwise (for instance), the Samsung entry would match

Samsung SSD<space>
Samsung SSD<space>8
Samsung SSD<space>88
Samsung SSD<space>888
.
.
.

ad nauseam: the "*" regex operator means "zero or more occurrences of the previous pattern", which in this case is the character "8".

At least, I hope they're not supposed to be regexes. Otherwise, the kernel blacklist itself will have some serious issues known-bad SSDs because someone never learned how to create a regular expression.

Re:Is there a site maintaining a list of "bad" SSD

[Anonymous Coward]

There's also an upgrade path for Micron's older SSDs - I just upgraded my Crucial M550 from MU01 to MU02 using a bootable ISO from Micron's support site:

      http://www.crucial.com/usa/en/support-ssd-firmware

Re:Is there a site maintaining a list of "bad" SSD

[Gaygirlie]

Not directly an answer to your question, but related: after Googling for a bit I actually cannot find any mention of Samsung SSD 840 PRO having issues with TRIM under Windows. If it was, indeed, a controller - problem then it would have to happen under all OSes as long as TRIM is enabled, but all the evidence I'm finding only points towards to Linux or these guys own setup as being the culprit.

Re:Is there a site maintaining a list of "bad" SSD

[FatdogHaiku]

Linus is on vacation so you'll have to wait on your next SSD purchase for him to return to merge the patches....

What? "The most influential individual economic force of the past 20 years" gets to just wander off?

Re:Software RAID issue with SSDs?

[sexconker]

Your RAID controller would have to pass the TRIM commands to the SSDs in the array for this bug to show up. The controller simply having TRIM support doesn't mean it actually passes it to drives that are part of an array.
Intel controllers since the Z-77 do pass TRIM along to the drives in the array, but only for RAID 1 and RAID 0.

Re:Is there a site maintaining a list of "bad" SSD

[MrBingoBoingo]

There is are two easy solutions to Ext4 vs. SSD problems. The first is ReiserFS which is still eminently usable on Gentoo. The second is UFS which is available on the BSD's.

Re:Is there a site maintaining a list of "bad" SSD

[Anonymous Coward]

I assume that Windows does not submit queued trim commands, thereby avoiding this problem.

Re:Is there a site maintaining a list of "bad" SSD

[Anonymous Coward]

You will only find SSDs from the very best vendors there... because the crap ones don't claim to support queued TRIM in the first place.

It is interesting that the Micron M500, *which is an enterprise datacenter SSD*, is listed. Rather bad PR for Micron, that: an enterprise datacenter SSD that corrupts data and has not been fixed?!

As usual, good PR for Intel... too bad their SSDs self-destruct based on a timer, instead of trying to soldier on until things actually get really broken (and only *then* self-destruct).

Re:Is there a site maintaining a list of "bad" SSD

[drinkypoo]

Wait, what?

When Intel SSDs decide they are bad, they just brick themselves instead of going into read-only-good-luck-your-data-may-be-bad-mode. This probably makes sense for Enterprise RAID, and for absolutely no other use case.

Re:Btrfs?

[BitZtream]

COW doesn't solve the problem that TRIM solves.

Once you write over the entire drive once, then all blocks of flash are dirty and MUST be erased before any new writes can take place. At this point, you can't even write the meta data without a sector erase, then you can write to it ... just to tell it that you've added another ref to an existing block.

With TRIM, blocks are erased when they are no longer used, so they do not need an erase cycle when before writing to them.

I don't use BTRFS, but do use ZFS and it most certainly benefits from TRIM on an active drive, which is certainly what all your SSDs are going to be.

Re:Is there a site maintaining a list of "bad" SSD

[PlusFiveTroll]

Because Windows doesn't do queued TRIM.

TRIM in Windows and Linux before now worked more like this. -DATA- -DATA- -FLUSH ALL COMMANDS TO DRIVE- -WAIT- -TRIM- -DATA- -DATA- When I drive was doing the trim thing it could not do anything else, there could be no other in flight commands to the drive.

This is different. -DATA- -DATA- -TRIM- -DATA- -TRIM- -DATA- -DATA- -DATA-

TRIM is part of the NCQ and is an operation occurring with other instructions in the SATA queue. Problem is some disk manufactures have pissed this up. It seems likely that a firmware update will be able to fix this issue.

https://en.wikipedia.org/wiki/...

Re:Is there a site maintaining a list of "bad" SSD

[complete loony]

From the SSD Endurance Experiment;

The drive's media wear indicator ran out shortly after 700TB, signaling that the NAND's write tolerance had been exceeded. Intel doesn't have confidence in the drive at that point, so the 335 Series is designed to shift into read-only mode and then to brick itself when the power is cycled. Despite suffering just one reallocated sector, our sample dutifully followed the script. Data was accessible until a reboot prompted the drive to swallow its virtual cyanide pill.

Re:trim

[Nkwe]

While poorly written, I think the author was suggesting that any model of SSD for which the Linux kernel has specific special handling logic should be avoided. In my opinion, it is not an unreasonable statement.

It probably is an unreasonable statement. If Linux has special logic to handle the drive, then someone else probably already had the problem and now there's a fix in so it probably won't happen to you.

Perhaps. But if the drive was broken and someone had to write special software to fix it, how can you be sure that it was fixed correctly and completely? Can you also be sure that the "fix" works for all versions of firmware on the drive? While you might be confident of these things, I would suggest that it would be better to use a drive that follows the standards and doesn't require special code to make it work right. Granted that as always, your mileage may vary -- and it could vary in either direction.

Re: Is there a site maintaining a list of "bad" SS

[Bert64]

If your booting from the SSD, chances are the machine will crash...
Would be much better to just stay in readonly mode, and give you the chance to copy data off (and yes im aware this is no substitute for a backup, but think of the use case of a travelling laptop far away from its backup server etc).

Re:Mentioned in a bad way by the Linux kernal?

[Mike Frett]

I could shove a dead Rat in the CPU slot and Windows would work with errors galore while Linux would actually tell me I have a dead Rat in the CPU slot. Linux exposes bad Hardware.

TRIM+NCQ

[DrYak]

it's not trim by itself that is problematic.
it's the combined use with NCQ (= reordering of commands).

The latest firmware (the one that fixes the speed decay) has started to falsely advertise support for this combination, whereas the drive doesn't actually support it.
The drive isn't actually able to re-order TRIM commands, and the wrong bit might end up being erased due to NCQ.

So this but will only show up:
- your drive is a Samsung 840 EVO (850 aren't affected by the speed decay and didn't get the faulty upgrade)
- if you have upgraded the firmware to the latest (faulty) version.
- you run an OS which is actually able to use TRIM+NCQ (Linux and BSD, basically)
- your OS also follows the standards (asks the drive what is supported and gets the false advertisement of TRIM+NCQ).
- you actually enable TRIM on the drive.

remove any one point of this chain and the bug doesn't happen.