Slashdot Mirror
Ubuntu Wants To Enable SSD TRIM By Default
jones_supa writes "During the first day of the latest virtual Ubuntu Developer Summit, Canonical developers finally plotted out the enabling of TRIM/DISCARD support by default for solid-state drives on Ubuntu 14.04. Ubuntu developers aren't looking to enable discard at the file-system level since it can slow down delete operations, so instead they're wanting to have their own cron job that routinely runs fstrim for TRIMing the system. In the past there has been talk about the TRIM implementation being unoptimized in the kernel. Around when Linux 3.0 was released, OpenSUSE noted that the kernel performs TRIM to a single range, instead of vectorized list of TRIM ranges, which is what the specification calls for. In some scenarios this results in lowered performance."
acronyms, too.
Well it's about time.
I'm still waiting for for firefox or chrome to make themselves SSD friendly. I know we all have RAMdisk, but I swear, after the OS, web browsers seem to generate the next highest number of 'writes'.
to Computer Science graduate? You know, down from kernel hacker?
What? I still count as a nerd and this IS news for nerds...
Windows 7 incorporated TRIM support for SSDs back in 2009. I know the Linux kernel can do it with the right mount options and has been able to for some time, but after a while you just assume distros are setting things automatically as expected (there's very few situations where TRIM is a bad idea, particularly on a desktop-focused distro like Ubuntu).
There's a reason I still feel like using a poor-man's system when using Linux on the desktop. They just don't think hard enough about automating stuff. Heck, Ubuntu (and no other distro I believe) doesn't enable Wake-on-lan when you shutdown, whereas Windows 7 and onwards does. This is something you have to script in yourself. Why the fuck aren't distros doing things you can reliably expect in commercials operating systems!?!
I swear, after the OS, web browsers seem to generate the next highest number of 'writes'.
I'd bet a lot of these writes are for caching received HTTP response bodies to disk. Otherwise, desktop browsers in low-memory environments would have to act like Firefox for Android and Chrome for Android. When I open multiple pages in tabs in these mobile browsers, they tend to discard entire pages as soon as I switch to another tab and reload them when I switch back. This interferes with my common use case of opening multiple pages in tabs and then reading them while offline and riding transit. Firefox for X11/Linux can keep pages open on a Dell Inspiron mini 1012 laptop with 1 GB of RAM, but Firefox for Android can't on a first-generation ASUS Nexus 7 tablet with the same amount of RAM. I guess the difference comes from two differences in the environment: swapping is more acceptable on X11/Linux than on Android, and desktop browsers are more likely to keep things in disk cache than mobile browsers.
Of source it doesn't implement the standard because it's a OS kernel, not a hard drive.
The drives implement TRIM, Linux just doesn't take full advantage of its capabilities.
More like Linux doesn't follow the best practice recommendations of the standard when it wouldn't be all that hard to do so.
I really don't know about windows TRIM support, but It'd better do it only if the HDD supports it.
What happens when the kernel sends a TRIM command to a drive that does not recognize TRIM commands?
I imagine that discussing the suggestions on Slashdot first is a way to avoid presenting half-baked suggestions to busy developers.
"As long as that SSD doesn't stall trying to pull blocks off the top of that queue, it really doesn't matter how deep it is. So if you have 10GB of free space on your partition, you only need to call wiper.sh / fstrim once every 10GB worth of file deletions."
This isn't necessarily true. Earlier Trim will improve the performance of the SSD drive because the drive knows more free space -- more free space allows the drive to 1) pre-emptively erase flash 2) coalesce fragmented blocks 3) more efficiently combine write blocks 4) perform wear levelling operations with less overhead.
Early trimming can have a similar effect to the manufacturer increasing slack space which increases performances on nearly all SSD's.
A filesystem just notes which blocks are erased, it doesn't actually erase them. [...] With TRIM, the OS can tell the disk which blocks are no longer needed, so that the disk can treat them like empty blocks
Why couldn't an operating system just write a big block of 0xFF bytes to an unused sector, which the SSD's controller would compress into an efficient representation of a low-information-content sector, instead of needing a dedicated command?
This is not so simple.
The original TRIM command is non-queued. It can kill drive performance on servers, so enterprise drives are designed to work well without TRIM. If you want better, and more importantly consistent performance then you should overprovision the drive. Overprovisioning means that you do not partition 20-40% of a new drive (or a used drive, after a secure erase). Those blocks will never be used, therefore the drive always have plenty of free space, so there is no need for trim.
Queued TRIM command appeared only in the SATA 3.1 specification, so only new drives support it.
Still my preferred Linux distro for desktop productivity (where the important points are A) easy to tell it what I want it to do, and B) it does it well, without needing a lot of hand-holding but also without needing me to fix anything afterward). Backtrack (I suppose I should really upgrade to Kali now...) and FreeBSD in VMs, for work and play respectively.
There's no place I could be, since I've found Serenity...
Of source it doesn't implement the standard because it's a OS kernel, not a hard drive.
The drives implement TRIM, Linux just doesn't take full advantage of its capabilities.
The drive does shit (shit that you don't get to know the details about) when issued a TRIM command.
The OS is responsible for sending that TRIM command.
TRIM tells the drive when data is deleted, allowing the drive to do whatever it thinks is best when writing pages of data or erasing blocks of data.
Without TRIM, the drives considers all previously written data to be valid because it doesn't know about deletions (they're done at the logical level within the file system).
TRIM enables your drive to have much more flexibility when writing (and overwriting) data, and when load balancing and garbage collecting. It also reduces the need for load balancing and garbage collecting.
All decent modern SSDs support TRIM for good reason. All decent modern OSs should as well.
Now if I could just get Intel to enable TRIM on RAID 0 for my chipset (1 generation behind the cutoff), I'd be set.
Linux fully supports TRIM and failure to enable it will not damage the device in any way. What will happen is the device will slow down and spend more time freeing blocks as-needed if the drive is increasingly full.
Of course, if your SSD is your boot drive and you have /home elsewhere, chances are you aren't going to suffer and current drives are significantly faster than older ones (and at their worst, still significantly faster than rotating media.)
I'm surprised this is still being figured out. I thought TRIM was old-hat, long since turned-on by default and working as intended.
So it's off by default because the Linux implementation slows regular IO, and even when it's on it's sub-optimal due to lack of multiple trim ranges.
Microsoft has had this working in Windows 7 circa 2009. With multiple trim ranges.
WTF? SSDs are important Linus. What is the major malfunction here?
The drive does shit (shit that you don't get to know the details about) when issued a TRIM command.
My point exactly. the drive implements the standard. Not the OS.
I already setup a cron job to fstrim my drives so this is a welcome addition that will save me a step during new installations.
ayottesoftware.com
To make it work, the disk would have to look at every written block and see if it's "the empty block".
A lot of flash controllers already use compression of some sort to reduce write amplification. So a controller could just store which sectors compress to the smallest sizes. That wouldn't be much more effort than storing which sectors are TRIMmed, and it'd ensure a well-defined response when the kernel attempts to read a TRIMmed sector.
TRIM is faster and easier.
Unless, as Immerman pointed out, a particular drive chokes on it. Sending TRIM to a device that doesn't correctly support TRIM produces "undefined behavior", and even some drives that claim to support TRIM don't in fact. No drive chokes on constant fill.
A sector full of only 0xFF might mean "I don't need this anymore" as well as "this file have a sector worth of 0xFF stored there".
Either way, when you read that sector back, the drive would decompress it to a string of 0xFF. This is true whether it's an "empty sector" or a row of white pixels in a BMP file.
Using trim, the FS/OS/whatever's on the line can tell the drive "ok, this part I don't need anymore, go play with it" in a non-ambiguous way.
What happens when the kernel attempts to read back a sector that hasn't been written since it was last TRIMmed?
I'm aware of that. But what should I do if I want the browser to write the page to flash memory, such as if I'm about to go offline for an hour?
WTF? SSDs are important Linus. What is the major malfunction here?
There was a fully functioning implementation on its way into the kernel, but Linus lost it when his only copy of the code was lost because his SSD failed.
Why go to the trouble of implementing a command implicitly when you can implement it explicitly and avoid unintended side effects?
Because the explicit command causes unintended side effects in drives manufactured prior to the command's introduction.
What in the fuck, you seriously consider some dude's admitted speculation as proof that this is a real risk?
Here's some vastly more likely semi-informed speculation: Like most modern standards, SATA has many optional features, and standardized discovery methods to inform software what each device can and cannot do. If a drive says it doesn't support TRIM (or, to be more precise, lacks some new capability tag which says it can do TRIM), the OS simply never issues a TRIM command.
It's like you (and that Immerman dude) have no concept of how sane people design protocols to safely accommodate future extensions.
Not to mention operating systems would still need to change the way they handle the disk to support the 0xFF method
Any file system supporting "secure" deletion should be filling deleted files' sectors in the background anyway.
Which operating system would that be? I'm not aware of any mainstream OS which even tries to implement secure deletion at the kernel level. I know of exactly one (OS X) which implements it in a user-visible way at all, and there it's just an option to "secure empty" Trash (read: it overwrites file contents in place). This is really not very secure since it is 100% userland code which has no way of tracking all the blocks ever occupied by that file and scrubbing them all -- it can only scrub blocks currently occupied by the file. It can defeat casual attempts to recover file contents, but there aren't any guarantees.
But it's worse than that. Let's say you tried to write an OS For Paranoids which, at the kernel level, scrubbed all blocks as the file was deleted or as they ceased to be part of a file. This still would not be truly secure. ATA (and other abstract block device protocols like SCSI) implements semantics somewhat like this: "If you read block N, you'll get back whatever was last written to block N, or undefined data if N was TRIMmed and not subsequently written to". There is absolutely no guarantee that the device did not make invisible, unaddressable extra copies of the contents of block N behind the operating system's back. As a matter of fact, SSDs frequently do just that -- it's required if you're going to implement effective wear leveling. Even HDDs frequently leave invisible copies of old data behind whenever they have to spare out a bad sector. Merely overwriting blocks is not, and never has been, a true guarantee that the contents are actually gone from storage media.
This is why good SSDs implement the ATA Secure Erase command, which (if implemented according to spec) truly erases 100% of the media, including all overprovisioned space, any invisible copies of old data which it may contain, and so on. But wait, oh mah gawd that's another new command- HOLY SHIT IT WILL NEVER WORK BECAUSE NOBODY CAN EVER ADD NEW THINGS TO A STANDARD!!! why did I not see this before?!
Tepples, please, for the sake of my sanity, please consider no longer arguing endlessly with posters who are explaining how shit works in the real world. Next time, instead of succumbing to the temptation of raising objection after objection based on your ideas about how things ought to work, consider the possibility that those ideas are naive and wrong.
> osx and windows both support it for YEARS
So you've proven yourself an irrational Linux-hater. Why are you here?
With my MacBook, here is what I had to do to enable TRIM:
http://www.mactrast.com/2011/07/how-to-enable-trim-support-for-all-ssds-in-os-x-lion/
With my work laptop that is a Dell, I had to download a utility from Intel to enable TRIM:
https://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=18455
Care to apologize for your lies?
There is nothing you do special in Windows to enable TRIM support. It is included support directly in the OS and the drivers, automatically. The only time you have to do anything special in Windows for TRIM support is when you're actively using Intel SSDs in a RAID configuration using the Intel Rapid Storage Driver--and even that is merely a driver update, and boom, RAID0 TRIM support passed from OS to driver to device.
That's it. Under all other circumstances TRIM is automatically enabled and there are no extra utilities needed under any circumstances.
This isn't quite as accurate. Most SSDs these days are built with space already overprovisioned. For example, a 128GB SSD might actually have 160GB of NAND flash inside of the device, with the remaining flash unreachable by the operating system and is used by the device.
Essentially, the age old adage of "over provisioning" is not actually that necessary on modern systems.
Those blocks will never be used, therefore the drive always have plenty of free space, so there is no need for trim.
It's not quite that simple either.
SSDs write in pages, but erase in blocks of pages. When a page is changed it gets rewritten to another block. The original page is marked as free, but it can't be erased until the whole block is free. Therefore the SSD performs garbage collection of free pages, re-packing them into complete blocks.
On its own the SSD only knows which pages it freed during rewrites - it doesn't know about pages that COULD be freed because they're deleted. Overprovisioning prevents blocking when there are no free pages (that's a huge win), but the drive still wastes lots of time and wear-life moving deleted data around during GC. TRIM provides the necessary hint to prevent that waste.
Any file system supporting "secure" deletion should be filling deleted files' sectors in the background anyway.
You don't seem to understand the basics of how SSD's work or you would haven't said this. Such secure file deletion doesn't actually work on SSDs. The LBA's overwritten with zeros or random data are written to different, already erased physical pages, while the original pages containing the data are simply flagged for erase. It isn't possible to directly overwrite SSD pages. They have to be erased first.
What the hell reason would it not be enabled by default?
a.) Because the spec was poorly written making the command a non-queuing command, therefore file systems can't just spit out a series of TRIM commands every time a file is deleted because the queue has to be cleared first. This slows down everything, reads and writes. With multiple file systems per drive, a given file system doesn't necessarily know the drive is idle so some other process would need to do the delayed TRIM which is what Canonical is suggesting.
b.) Some manufacturers have implemented very aggressive erase cycles upon TRIM commands being received and that stalls the drive also. This is also not very smart.
I just assumed, since osx and windows both support it for YEARS, that forward thinking linux did. Wow.
OK OS X only supports it for Apple branded drives, it's disabled for most (maybe all) 3rd party SSDs. No doubt Apple found certain edge cases where it was causing a problem and instead of white or blacklisting piles of drives they decided not to let manufacturers use users as guinea pigs. Conversely, Microsoft decided it's probably better than not doing it and after all the manufacturer's and their industry trade association and standards body need to sort out this mess rather than being bailed out of it.
It makes you wonder what other optimizations are not being done but are done in Windows.
I'll toss in one: in Windows 8, an USB memory stick is automatically powered down if it is mounted but has not been in use for a while.
It's also worth noting that there are two modes for TRIM. One enforces the invariant that two reads of the same block will give the same value if there are no writes between them, the other only requires this invariant for blocks that have not been marked as erased. There's often a performance difference between them, because the latter lets the drive just leave the block as spare and overwrite it during a GC and then reallocate it on write (a good implementation would mark it as zero'd and always return zeros until there is a write, but this requires slightly more space in the remapping tables and so isn't always done). It sounds like Linux is still using the former version, which can be very slow on some drives.
I am TheRaven on Soylent News
I moved all the cache directories of Firefox/Chrome/IE and even the temp directory of OS and all applications point to a cheap USB thumb drive.
I have not looked around to find out if there is more writes happening elsewhere other than cache/temp...but I guess a super majority is taken care of.
The above might be the reason the SSD did not 'disappear' when a power outage happened.
Now, as far as I know the only cache I cannot control is when the virtual machines are booted up - swap spaces remain in the SSD.
Tat Tvam Asi
And TRIM is enabled by default on OS X if you use the stock drives. If you use an after-market replacement then you do need to explicitly enable it (which sucks). With FreeBSD, it's enabled for UFS by default since 9.0 and ZFS by default since 9.2. It is also enabled in software raid configurations by default in 10.0. I'm very surprised Linux doesn't enable it by default.
I am TheRaven on Soylent News
With multiple file systems per drive, a given file system doesn't necessarily know the drive is idle so some other process would need to do the delayed TRIM which is what Canonical is suggesting.
Why would a filesystem need to know? On FreeBSD, the filesystem just spits a BIO_DELETE command into the GEOM layer, and it is up to GEOM to schedule when to dispatch it - it's free to reorder it, as long as it doesn't move it after a BIO_WRITE with an overlapping range. If the filesystem needs to know about the status of other filesystems then that's a serious layering problem. The FS should not be making the decision about whether to send the BIO_DELETE, because it's the responsibility of something lower down the stack to decide what to do with it. For example, a RAM disk can use it to free the underlying memory. You don't want every filesystem to have to know about every possible kind of device.
I am TheRaven on Soylent News
The word you are looking for is API. An implementation is on the side of the thing that exposes the API. The OS uses the API, it does not implement the standard. Implement would mean providing the API. Not making use of an API doesn't mean you're not standards compliant. Eg. None of my OSs make use of (U)EFI (because coreboot is amazing). My OSs are following the boot standards, They just aren't using the API that was implemented for (U)EFI.
You "execute" TRIM through the API of its implementation. Calling it on delete is an optional part of the protocol. Not calling it on deletes is not deviation from the standard.
Linux fully supports TRIM and failure to enable it will not damage the device in any way.
Linux does not fully support TRIM. It is the very reason why many distros do not automatically enable "discard" in fstab. As noted in the summary: "the kernel performs TRIM to a single range, instead of vectorized list of TRIM ranges, which is what the specification calls for. In some scenarios this results in lowered performance".
Such as? The post you linked to explicitly said they were simply guessing.
On the other hand, compression algorithms do have plenty of weird side effects, from increased latency to randomly varying write speeds to the impossibility of estimating actual free space - because for every bit a lossless compression algorithm shaves off one file, it must add to some other file (because if a bigger file is made to look like a smaller file, the smaller file that looks like its compressed form must be moved out of the way, and the only place with room is what the bigger file just vacated). Thus some files are actually made bigger than their size implies.
Secure against what, exactly speaking? Even physical hard drives can remap blocks. And how many files actually need to be unrecoverable? Not many. And drives are cheap. So, if you store sensitive information, don't trust "secure erase" or even "drive wipers", just physically destroy the drive when you're done with it (but if you're too cheap for that, a drive wiper is still better than trusting an OS "secure" delete; even if the OS is trustworthy, filling the entire drive with garbage several times over has a much higher chance of pushing out your secret data than filling just a tiny fraction of it, as "secure" delete does). And given that, "secure" deletion is a pointless waste of resources that will actually lower people's security by giving them promises it can't keep.
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.
Windows 8, an USB memory stick is automatically powered down if it is mounted
Yeah, but in reality it gets it wrong half the time and closes Firefox instead.
Join the Slashcott! Feb 10 thru Feb 17!
Those blocks will never be used, therefore the drive always have plenty of free space, so there is no need for trim.
Clue me in: how in the world does the drive firmware know which blocks the filesystem data structures have pointers to as valid, currently unused ranges?
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
for every bit a lossless compression algorithm shaves off one file, it must add to some other file
Pigeonhole principle. I'm aware of that.
Thus some files are actually made bigger than their size implies.
By about one byte, a marker that the sector is uncompressed. In a real file system, this overhead of one byte per sector is made up for by real files that contain real redundancy, such as the last sector of a file (or the only sector of a small file). On average, the last sector of a file will contain a run of half a sector's worth of $00 bytes. This and other cases where sectors can be compressed allow more logical sectors to fit into a single erase page.
Because that would require the flash to do deduplication and know that the blocks were full of FF and so could be copy-on-write (and, in your scheme, block-level compression)
Block-level run-length encoding is exactly what I had in mind. This way more logical sectors can be packed into one 64-128K erase page. A sector that has been compressed into "0x00, then 4095 more of the last byte" is as good as TRIMmed. It needs to be done anyway for file tails.
How should one make sure a particular model of SSD is not broken before buying it? I was under the impression that by the time all the bugs got shaken out of a product, it ended up close to EOL.
The firmware is not interested in filesystem pointers. It does know, however, if a block has never been written.
The "everything turned on by default" concept is part of why Windows is bloated and insecure.
This is changing, but part of what I like about Linux is that it makes less assumptions about what you are doing and assumes a (at least somewhat) skilled operator. Part of what I dislike about Ubuntu is that it makes too many assumptions about what I want to do. It's also why multiple Linux distros that target different audiences are a good thing.
The points here are that :
1. All optimizations or assumptions make something better for a certain task or use scenario.
2. Any optimization that makes something better for a certain task or scenario will be worse for a different task/use scenario.
Think of strong versus lightweight -- it's a tradeoff.
Competition Good, Monopoly Bad.
Windows does this for hard disks too.
The word you are looking for is API. An implementation is on the side of the thing that exposes the API. The OS uses the API, it does not implement the standard. Implement would mean providing the API. Not making use of an API doesn't mean you're not standards compliant. Eg. None of my OSs make use of (U)EFI (because coreboot is amazing). My OSs are following the boot standards, They just aren't using the API that was implemented for (U)EFI.
You "execute" TRIM through the API of its implementation. Calling it on delete is an optional part of the protocol. Not calling it on deletes is not deviation from the standard.
Holy shit no.
TRIM is a COMMAND, part of the ATA INTERFACE STANDARD.
Hardware devices IMPLEMENT this STANDARD to function. Operating systems IMPLEMENT this STANDARD to support said hardware.
API stands for APPLICATION PROGRAMMING INTERFACE. APIs allow for SOFTWARE to talk to SOFTWARE. APIs do not fucking apply here. TRIM is not part of an API. It is part of the fucking STANDARD. Whatever the drive does in its FIRMWARE doesn't make TRIM an API, nor does it mean the OS doesn't have to IMPLEMENT TRIM as part of the latest SPECIFICATION of the ATA STANDARD.
People like you are why manufacturers have to pay money and get certified to be able to slap a Windows or Mac sticker on their gear, and not the other way around.