All ones or all zeros are a matter of interpretation.
Of course. My statement was imprecise. It would have been more accurate to say that what we (or at least, I) would intuitively expect to represent all 1s - i.e. the storage of charge - actually represents a logical zero when you read the cell.
Graeme.
I'm not against block level access: I just think that most cases where you mix low level access to dead space with simultaneous filesystem (or page) level access are edge cases - whether it's RAM or disk you're talking about.
Looking at the 5 suggestions: I asked for realistic, real-world use cases.
1 - Lazy conversion of filesystems on a single medium using 'dead' space on the disk without creating a temporary filename corresponding to those blocks to cover their ass? This is a 'realistic', 'real-world' use case? Show me a program that does this.
2 - Defragging: similarly, if you're moving data around in dead space without safely duplicating it or having a filename pointing to the blocks in use at any given time, you're not being careful. Also, which defraggers have random 3-minute gaps in operation that would even allow GC to kick in?
3 - Raw filesystem image included as a file inside another filesystem: I don't get it - if the blocks for the image are marked as associated with a filename, where's the problem?
4 - UDF format over NTFS old filesystem. Good point, though I would argue that any UDF formatting tool which leaves in a disk in a state where it could be interpreted ambiguously in terms of it's filesystem, is a broken formatting tool.
5 - Interesting theoretical idea, but not 'real world'. this is not actually the case and MS are well aware of SSD tech, as seen in the latest iteration of Windows. A problem for the future though, for anyone adding new tricks to the GC.
"This is of course a spurious comparison. SMB, FTP, and NFS are presented to the operating system as file trees."
This GC only works with NTFS filesystems. If you are operating an SSD device using an NTFS file tree to store data, then you (as a programmer) are not using the drive as a block device as you suggest; you're accessing it (or you should be accessing it) via the abstract file tree.
"Again, RAM is presented to the OS as logical addresses, and it does faithfully restore the data that was stored in those logical addresses."
While there is a difference in the semantics that occur when marking a reference to a memory page as free (and subsequently losing access to the data) and marking a reference to file data as deleted, they are essentially the same operation in terms of saying 'I don't need to store this data now'.
The only difference is that people seem to take block-level access to the disk for granted; we now know with memory that block-level access isn't really a good idea at all, in the interests of protecting memory from accidental or deliberate attempts to misuse it.
The problem, from my perspective, is that your arguments hinge on the idea that marking data as 'deleted, and the filesystem can now overwrite it at some random future point, perhaps instantly or never' (the HDD model) is in some way better than 'deleted, purge at first opportunity' (the SSD model). From my perspective, I'd prefer the latter; at least then I know what's happened to data after it's been marked for deletion.
I'm still keen to see those realistic real-world use cases. If another poster has posted them, can you provide a link?
"followed by a drive being connected to a non-TRIM OS with a write-blocker"
"Now write 10GB of data to the drive overwriting 10GB of the data written earlier."
Write-blocker. Also, here, we did not do any writes to the drive, only reads. I wasn't asking you to suggest a way in which an OS can (generally) overwrite deleted file data, that's easy. I was asking you to explain how that could happen in the experimental setup we described, if you're suggesting it wasn't the drive-based GC that did it.
SSDs are a great upgrade for macs. Last year I chose a 2-year old core2duo Macbook with an SSD over a shiny new, top-end iMac i7 quad-core. After comparing them side-by-side for most of the work I was doing, 'for most practical purposes' the Macbook/SSD ran better.
"A SSD that performs automatic garbage collection by interpreting the filesystem in firmware is not, in my opinion, a storage device."
Well - you can hold any opinion you like, but the physical fact of reality is that this device can and is being used to store things quite safely by millions of people. To use an analogy: would you say that a library that reorganises their shelves and throws out items marked for removal from time to time - but only when there's no one using the library at 3AM in the morning - was not really a reliable repository for information? Because in reality, that's how most libraries work. You might hold the opinion that such a library was not really be an 'information storage facility' because in fact, you were arranging to have some books thrown out in such a way that a secret message was conveyed to the trashman. But 99.9999% of the population would consider the loss of such a peculiar backchannel a reasonable tradeoff, for say, a 3-fold to 10-fold improvement in how quickly they can find things on the shelf - which is of course the primary purpose of a library.
The analogy is fair, because the disk only runs GC when the disk isn't in use for several minutes; at the OS level we can use write-through caching to ensure there's no logical data left unwritten at this point.
"Suppose I am a filesystem developer. Suppose I want to modify NTFS in such a way that deleted segments of an NTFS disk layout become (in my modified filesystem) a repository for meaningful data. This is not as absurd a concept as it appears. In my line of work (cryptography), storing actual meaningful data in deleted segments might be something that you want to do, for example in steganography.
This is so far into being an edge case it's not funny. The phrase 'for example in steganography' is not reasonable here - can you give me 5-10 other examples to demonstrate this isn't a one-off edge case? Steganography seems like the only example I can think of, and as a stego researcher myself (see my site) I can tell you that you won't get much capacity or reliability from rearranging deleted files - the OS will need to use that space for something. I think huge numbers of deleted files or metadata entries that are not being overwritten would stand out a bit to a forensic investigator too, given that the second thing they'll look for is deleted files.
"In this sense, it is, by definition, impossible for a valid storage device to implement automatic garbage collection at the filesystem level."
I'm guessing you don't consider networked computers (e.g. SMB shares, FTP sites, NFS mounts) to be storage devices either then, since the remote host will merrily overwrite deleted files with other people's data however it likes there too? Why do you think so many people are willing to use remote hosts to store data when they don't have control over how deleted files are garbage-collected/re-used/arranged below the logical layer?
"Sure, those deleted sectors are safe to erase in an NTFS volume, but how do you know that my operating system is using this NTFS volume as an NTFS volume? What if I'm doing steganography or something where those deleted sectors matter?"
What other somethings do you have in mind?
Thanks for your feedback though, it's interesting to see people's gut reactions to this tech.
I will be very interested if you can enumerate some more realistic examples of why performance-boosting GC is a bad thing, other than an edge case (NTFS file deletion stegosystem) which, I would expect, does not presently exist as an implementation.
Graeme.
p.s. Just thought of another example. RAM. Where you store data in memory logically, and how it is arranged physically - including zeroing of dead pages - are completely out of your control and even out of your view. Does this mean you consider RAM not to be a storage device, since you can't reliably construct a stego side-channel using dead pages of memory?
"What you're describing is impossible. It might be possible for some of the more common filesystems"
You seem to have changed your mind rather quickly there?
given the difficulty of supporting NTFS in Linux, I highly doubt that embedded firmware on a drive can parse the NTFS format)
Well, if you think I (and the tech support staff on various SSD manufacturers forums) are wrong, you're welcome to buy an SSD and check for yourself. It's not quite as easy as typing 'it's impossible' a bunch of times, but it's a lot more likely to be correct.
It is utterly impossible in the case of new filesystems. Think about it -- if a piece of hardware predates the creation of ext4, or ext5, or whatever, then how can the hardware understand the filesystem?"
The same way they added GC to older models of SSD drives where it didn't already exist of course, and the same way they update features on any hardware. You flash the firmware with revised code.
"How would it do this without maintaining firmware for all possible types of file systems?"
It wouldn't. But, how many different filesystems are in realworld use on 95%+ of systems? HFS+, ext2/3, NTFS, FAT32?
I can imagine that it does this for simple FAT16 or FAT32, but I doubt they deal with NTFS or EXT3 or HFS.
It does work on NTFS, and if you set up an experimental rig similar to the one we describe at the back of the paper (page 16), hopefully you can satisfy your doubts:-)
"This story comes just 2 weeks after another story on slashdot saying that your data doesn't actually get deleted:"
The two papers are similar in that they both highlight unexpected and poorly documented behaviours of real world SSDs that diverge from traditional hard disk behaviour, and raise issues for digital forensics and data recovery. The irony is that the USENIX authors have shown SSDs may not purge data even when you try your best to make them do so (i.e. a problem for information security); whereas here, we show they may purge data permanently when you don't want or expect them to (i.e. a problem for digital forensics & law).
"The laws regarding hard drives will become much closer to the laws regarding biologicals which can alter themselves between collection and trial, and can be altered by the investigative process."
I agree, that's one way it could go; other options might include 'ignorance is bliss, except for defendants' and 'attempt to legislate ineffectively against a moving target'.
Broadly speaking, I think a can of worms will be opened in the next few years in regard to the legal idea of volition and intention, as computers increasing come under the control of viruses, self-repairing computer systems and so on (this is arguably an example of a self-repairing system).
"To clarify - is this deletion occurring because the drive analyzed the filesystem metadata and decided it would be a good idea to zero some sectors, or is it an asynchronous response to a fast TRIM command applied to an extent covering the whole drive?"
It's not TRIM. We used a non-TRIM OS and a write blocker for this reason. The manufacturer has mentioned a garbage collector that works with NTFS, and the only way it could work is by zero-ing areas of the SSD using information gleaned from the metadata. Hope this clarifies the situation for you!:-)
" but standards of evidence will emerge over time."
Agreed! But, surely this will only happen if courts realise that new standards *need* to emerge because there is a problem with the old ones not keeping up with technology?
You're welcome, it's great to hear people's thoughts about our work and to have the opportunity to get the message out about this result to a wider audience.
"That image in raw data will show that the drive would have been garbage collecting as the image was being taken."
Keep in mind the established court procedure to verify if an investigator's image (and finding) is valid, would be to e.g. run an MD5 over the original disk with a write blocker attached in front of defense and prosecution. That will show 'no match', and by precedent the investigator's copy and findings (not the original) will become devalued, practically to zero.
Also, how could a forensic investigator show a court that his image 'really' was the original version?
There's no authoritative copy representing the data at time of capture to refer to any more, if the original was being modified as he took the copy.
There's also support for GC on the OCZ vertex 2 drives, I think. However, some bad news and good news for you:
The bad news is, the firmware GC implementation we looked at only supported GC on NTFS partitions.
The good news is, you can use NTFS with macos if you like (e.g. through FUSE), or you can wait for MacOS Lion, which is about to be released, and which is rumoured to support TRIM directly. Shouldn't be long!
I don't see anything unusual about being interested in having a discussion about my results with other people - in fact, I'd say that's pretty much the whole idea of scientific publication.
Unfortunately, the way the comment moderation system is set up on this site makes it very hard to talk back and forth without help - new / temporary accounts are by default set to be almost invisible to casual readers. Given the choice between looking good from your perspective, and being able to easily engage in two-way discussion with interested slashdotters about SSDs and forensics, I'll take the latter:-)
On the other hand, in this situation, the thing doing the purging is the drive's firmware itself, not the OS, and the firmware knows for sure where the data is in the cells, and furthermore it's on a mission - to purge cells that it knows the filesystem is no longer using for data.
Inaccurate. These drives are oblivious to the file-system
Hey there. Unfortunately, what you've written isn't correct, and I encourage you to read into modern SSD garbage collection. These drives really do open up the filesystem by themselves, and look for deleted files and purge them, without being asked to do so by the OS. Otherwise, can you explain how file deletion using a non-TRIM OS, followed by a drive being connected to a non-TRIM OS with a write-blocker, would result in data being purged? (we proved this experimentally, and you can reproduce it in your own home using the supplied software and experimental parameters).
But you are wrong that the SSD is doing the purging outside of OS intervention.. the OS must specifically mark pages for purging.
The GC on the drive in question is specifically designed to provide performance improvement for *OS that do not have TRIM*. If you don't believe me, look up the drive model in the paper and google it for forum discussions of its behaviour (or buy one for yourself, and watch it happen in realtime using the probe program! It's freaky to see a drive rapidly purge it's supposedly recoverable data when connected to a non-TRIM OS)
"How can I tell which ssd supports this option?
is there a marketing name for it?"
A few drives offer it, with varying names and degrees of marketing fanfare, but there is little documentation as to the precise workings of the GC features. In the paper PDF you'll see the drive make/firmware for the drive we used in these experiments. You can buy that one if you like:-)
"...a foregone conclusion ever since ATA Secure Erase and TRIM were introduced?"
Yes, TRIM is eventually going to change the game in the sense that data will (probably) be expunged ASAP by the drive when you delete them (as you observe though, it is a request to the drive, not an instruction, and consequences of issuing a TRIM are unknown). But it could be a decade before courts will stop seeing the problem mentioned in this paper. I think the problem will peak in coming years and decline e.g. 4-5 years out as people finally give up on XP/2003, non-TRIM linux implementations of FAT32/NTFS, older Mac OS's, etc.
TRIM only works when you have a firmware and OS that support them. Given the number of people I know using XP (and below) and MacOS Snow Leopard (and below), there's a lot of computers out there who won't be sending ATA TRIM commands to their disk. Yet SSD GC is becoming more common and more aggressive in the meantime...
I would say that court practice and legal precedent is slow to update as far as technology is concerned, and SSD GC has come out of the blue for them, and is a currently a fast-moving technology that works with most non-TRIM OS's out there, and wrecks court expectations and practices that have worked for about 25 years.
"This property of modern SSDs is neither positive nor negative by itself. It's a property you'd be thankful for if you were organizing protests in Iran and stored contacts on an SSD. You'd curse it if it were exploited to destroy incriminating photos by someone who'd stalked you or a loved one."
"So either SSDs are really hard to erase, or really hard to recover. I'm so confused."
Hello, I'm one of the authors of the paper. To explain the apparent paradox in rough terms:
Drive data was traditionally purged manually, by having the computer tell the drive to write something else over the top of the old data. In the absence of such an overwrite, magnetically stored data persists. However, if you try that trick on an SSD, it may not work. The logical address you try to overwrite may be remapped on the fly, so that your 'overwrite' goes to some other physical cell rather than the one which stored the data. From a logical viewpoint, it looks like the overwrite worked - you can't access the data any more through your computer's OS. But from the drives point of view, the data is still there, lurking in some physical cell that is presently out of use as far as the logical sector list is concerned. A cunning firmware or a hacker with a soldering iron might still get at it.
However, separately to this, modern SSD drives use tricks to try and automatically improve their performance, and one of these tricks is to pre-empetively wipe data cells that contain data no longer referenced by the filesystem. Here, the drive is actively attempting to permanently purge everything it can from the drive, all of it's own accord, in the interests of accelerating future writes by having a pool of completely unused cells available.
Summary:
- If you're a computer telling a drive to zero over some data, the drive may lie to you a bit, and not bother to zero it.
- If you're a drive, you do whatever the heck you like, and you see the physical layer directly (unlike the computer). That means the drive can open up the NTFS metadata, looking for data cells which could be preemptively reset, and nuking that data out of existence (when it might traditionally have been recoverable to an expert).
In summary. If your drive wants to nuke something, (and we've shown, they really DO want to nuke everything they can at a few minutes notice), it gets nuked. If your PC wants to nuke something, it may or may not get nuked by attempting an overwrite.
Finally, separate to this is TRIM, which is a hybrid of the two situations - an ATA command by which the OS can signal to the drive that it would like the corresponding physical cell for a particular logical sector address to be nuked, thank you very much.
Slashdot Mirror
All ones or all zeros are a matter of interpretation.
Of course. My statement was imprecise. It would have been more accurate to say that what we (or at least, I) would intuitively expect to represent all 1s - i.e. the storage of charge - actually represents a logical zero when you read the cell. Graeme.
Hey again David,
I'm not against block level access: I just think that most cases where you mix low level access to dead space with simultaneous filesystem (or page) level access are edge cases - whether it's RAM or disk you're talking about.
Looking at the 5 suggestions: I asked for realistic, real-world use cases.
1 - Lazy conversion of filesystems on a single medium using 'dead' space on the disk without creating a temporary filename corresponding to those blocks to cover their ass? This is a 'realistic', 'real-world' use case? Show me a program that does this.
2 - Defragging: similarly, if you're moving data around in dead space without safely duplicating it or having a filename pointing to the blocks in use at any given time, you're not being careful. Also, which defraggers have random 3-minute gaps in operation that would even allow GC to kick in?
3 - Raw filesystem image included as a file inside another filesystem: I don't get it - if the blocks for the image are marked as associated with a filename, where's the problem?
4 - UDF format over NTFS old filesystem. Good point, though I would argue that any UDF formatting tool which leaves in a disk in a state where it could be interpreted ambiguously in terms of it's filesystem, is a broken formatting tool.
5 - Interesting theoretical idea, but not 'real world'. this is not actually the case and MS are well aware of SSD tech, as seen in the latest iteration of Windows. A problem for the future though, for anyone adding new tricks to the GC.
Graeme.
"This is of course a spurious comparison. SMB, FTP, and NFS are presented to the operating system as file trees."
This GC only works with NTFS filesystems. If you are operating an SSD device using an NTFS file tree to store data, then you (as a programmer) are not using the drive as a block device as you suggest; you're accessing it (or you should be accessing it) via the abstract file tree.
"Again, RAM is presented to the OS as logical addresses, and it does faithfully restore the data that was stored in those logical addresses."
While there is a difference in the semantics that occur when marking a reference to a memory page as free (and subsequently losing access to the data) and marking a reference to file data as deleted, they are essentially the same operation in terms of saying 'I don't need to store this data now'.
The only difference is that people seem to take block-level access to the disk for granted; we now know with memory that block-level access isn't really a good idea at all, in the interests of protecting memory from accidental or deliberate attempts to misuse it.
The problem, from my perspective, is that your arguments hinge on the idea that marking data as 'deleted, and the filesystem can now overwrite it at some random future point, perhaps instantly or never' (the HDD model) is in some way better than 'deleted, purge at first opportunity' (the SSD model). From my perspective, I'd prefer the latter; at least then I know what's happened to data after it's been marked for deletion.
I'm still keen to see those realistic real-world use cases. If another poster has posted them, can you provide a link?
Graeme.
"followed by a drive being connected to a non-TRIM OS with a write-blocker"
"Now write 10GB of data to the drive overwriting 10GB of the data written earlier."
Write-blocker. Also, here, we did not do any writes to the drive, only reads. I wasn't asking you to suggest a way in which an OS can (generally) overwrite deleted file data, that's easy. I was asking you to explain how that could happen in the experimental setup we described, if you're suggesting it wasn't the drive-based GC that did it.
Graeme.
SSDs are a great upgrade for macs. Last year I chose a 2-year old core2duo Macbook with an SSD over a shiny new, top-end iMac i7 quad-core. After comparing them side-by-side for most of the work I was doing, 'for most practical purposes' the Macbook/SSD ran better.
Hi David
"A SSD that performs automatic garbage collection by interpreting the filesystem in firmware is not, in my opinion, a storage device."
Well - you can hold any opinion you like, but the physical fact of reality is that this device can and is being used to store things quite safely by millions of people. To use an analogy: would you say that a library that reorganises their shelves and throws out items marked for removal from time to time - but only when there's no one using the library at 3AM in the morning - was not really a reliable repository for information? Because in reality, that's how most libraries work. You might hold the opinion that such a library was not really be an 'information storage facility' because in fact, you were arranging to have some books thrown out in such a way that a secret message was conveyed to the trashman. But 99.9999% of the population would consider the loss of such a peculiar backchannel a reasonable tradeoff, for say, a 3-fold to 10-fold improvement in how quickly they can find things on the shelf - which is of course the primary purpose of a library.
The analogy is fair, because the disk only runs GC when the disk isn't in use for several minutes; at the OS level we can use write-through caching to ensure there's no logical data left unwritten at this point.
"Suppose I am a filesystem developer. Suppose I want to modify NTFS in such a way that deleted segments of an NTFS disk layout become (in my modified filesystem) a repository for meaningful data. This is not as absurd a concept as it appears. In my line of work (cryptography), storing actual meaningful data in deleted segments might be something that you want to do, for example in steganography.
This is so far into being an edge case it's not funny. The phrase 'for example in steganography' is not reasonable here - can you give me 5-10 other examples to demonstrate this isn't a one-off edge case? Steganography seems like the only example I can think of, and as a stego researcher myself (see my site) I can tell you that you won't get much capacity or reliability from rearranging deleted files - the OS will need to use that space for something. I think huge numbers of deleted files or metadata entries that are not being overwritten would stand out a bit to a forensic investigator too, given that the second thing they'll look for is deleted files.
"In this sense, it is, by definition, impossible for a valid storage device to implement automatic garbage collection at the filesystem level."
I'm guessing you don't consider networked computers (e.g. SMB shares, FTP sites, NFS mounts) to be storage devices either then, since the remote host will merrily overwrite deleted files with other people's data however it likes there too? Why do you think so many people are willing to use remote hosts to store data when they don't have control over how deleted files are garbage-collected/re-used/arranged below the logical layer?
"Sure, those deleted sectors are safe to erase in an NTFS volume, but how do you know that my operating system is using this NTFS volume as an NTFS volume? What if I'm doing steganography or something where those deleted sectors matter?"
What other somethings do you have in mind?
Thanks for your feedback though, it's interesting to see people's gut reactions to this tech.
I will be very interested if you can enumerate some more realistic examples of why performance-boosting GC is a bad thing, other than an edge case (NTFS file deletion stegosystem) which, I would expect, does not presently exist as an implementation.
Graeme.
p.s. Just thought of another example. RAM. Where you store data in memory logically, and how it is arranged physically - including zeroing of dead pages - are completely out of your control and even out of your view. Does this mean you consider RAM not to be a storage device, since you can't reliably construct a stego side-channel using dead pages of memory?
Hey,
"What you're describing is impossible. It might be possible for some of the more common filesystems"
You seem to have changed your mind rather quickly there?
given the difficulty of supporting NTFS in Linux, I highly doubt that embedded firmware on a drive can parse the NTFS format)
Well, if you think I (and the tech support staff on various SSD manufacturers forums) are wrong, you're welcome to buy an SSD and check for yourself. It's not quite as easy as typing 'it's impossible' a bunch of times, but it's a lot more likely to be correct.
It is utterly impossible in the case of new filesystems. Think about it -- if a piece of hardware predates the creation of ext4, or ext5, or whatever, then how can the hardware understand the filesystem?"
The same way they added GC to older models of SSD drives where it didn't already exist of course, and the same way they update features on any hardware. You flash the firmware with revised code.
"How would it do this without maintaining firmware for all possible types of file systems?"
It wouldn't. But, how many different filesystems are in realworld use on 95%+ of systems? HFS+, ext2/3, NTFS, FAT32?
I can imagine that it does this for simple FAT16 or FAT32, but I doubt they deal with NTFS or EXT3 or HFS.
It does work on NTFS, and if you set up an experimental rig similar to the one we describe at the back of the paper (page 16), hopefully you can satisfy your doubts :-)
Graeme.
"This story comes just 2 weeks after another story on slashdot saying that your data doesn't actually get deleted:"
The two papers are similar in that they both highlight unexpected and poorly documented behaviours of real world SSDs that diverge from traditional hard disk behaviour, and raise issues for digital forensics and data recovery. The irony is that the USENIX authors have shown SSDs may not purge data even when you try your best to make them do so (i.e. a problem for information security); whereas here, we show they may purge data permanently when you don't want or expect them to (i.e. a problem for digital forensics & law).
Graeme
"The laws regarding hard drives will become much closer to the laws regarding biologicals which can alter themselves between collection and trial, and can be altered by the investigative process."
I agree, that's one way it could go; other options might include 'ignorance is bliss, except for defendants' and 'attempt to legislate ineffectively against a moving target'.
Broadly speaking, I think a can of worms will be opened in the next few years in regard to the legal idea of volition and intention, as computers increasing come under the control of viruses, self-repairing computer systems and so on (this is arguably an example of a self-repairing system).
Graeme.
"To clarify - is this deletion occurring because the drive analyzed the filesystem metadata and decided it would be a good idea to zero some sectors, or is it an asynchronous response to a fast TRIM command applied to an extent covering the whole drive?"
It's not TRIM. We used a non-TRIM OS and a write blocker for this reason. The manufacturer has mentioned a garbage collector that works with NTFS, and the only way it could work is by zero-ing areas of the SSD using information gleaned from the metadata. Hope this clarifies the situation for you! :-)
Graeme.
" but standards of evidence will emerge over time."
Agreed! But, surely this will only happen if courts realise that new standards *need* to emerge because there is a problem with the old ones not keeping up with technology?
Graeme.
"Is there a list of SSDs that do this?"
P64 Corsair and OCZ vertex 2 are known to have GC ; it's probably worth checking firmware notes for any drive you're thinking of buying.
Graeme.
Who would buy it, knowing that the feature could be used against them? Also, there's the legacy problem.
"Your honour, the defendant knowingly purchased an old SSD drive rather than one with ATA Freeze!"
Graeme.
You're welcome, it's great to hear people's thoughts about our work and to have the opportunity to get the message out about this result to a wider audience.
Graeme.
Mirror'd it here for you Andy.
http://graemebell.net/downloads/JDFSL-V5N3-Bell.pdf
Graeme.
Thanks for your comments.
"That image in raw data will show that the drive would have been garbage collecting as the image was being taken."
Keep in mind the established court procedure to verify if an investigator's image (and finding) is valid, would be to e.g. run an MD5 over the original disk with a write blocker attached in front of defense and prosecution. That will show 'no match', and by precedent the investigator's copy and findings (not the original) will become devalued, practically to zero.
Also, how could a forensic investigator show a court that his image 'really' was the original version?
There's no authoritative copy representing the data at time of capture to refer to any more, if the original was being modified as he took the copy.
Graeme.
If you take a look at: http://www.jdfsl.org/subscriptions/JDFSL-V5N3-Bell.pdf , you'll see the drive model we used was the Corsair P64 (see page 16).
There's also support for GC on the OCZ vertex 2 drives, I think. However, some bad news and good news for you:
The bad news is, the firmware GC implementation we looked at only supported GC on NTFS partitions.
The good news is, you can use NTFS with macos if you like (e.g. through FUSE), or you can wait for MacOS Lion, which is about to be released, and which is rumoured to support TRIM directly. Shouldn't be long!
Graeme.
Hello Noughmad.
I don't see anything unusual about being interested in having a discussion about my results with other people - in fact, I'd say that's pretty much the whole idea of scientific publication.
Unfortunately, the way the comment moderation system is set up on this site makes it very hard to talk back and forth without help - new / temporary accounts are by default set to be almost invisible to casual readers. Given the choice between looking good from your perspective, and being able to easily engage in two-way discussion with interested slashdotters about SSDs and forensics, I'll take the latter :-)
Graeme.
Inaccurate. These drives are oblivious to the file-system
Hey there. Unfortunately, what you've written isn't correct, and I encourage you to read into modern SSD garbage collection. These drives really do open up the filesystem by themselves, and look for deleted files and purge them, without being asked to do so by the OS. Otherwise, can you explain how file deletion using a non-TRIM OS, followed by a drive being connected to a non-TRIM OS with a write-blocker, would result in data being purged? (we proved this experimentally, and you can reproduce it in your own home using the supplied software and experimental parameters).
But you are wrong that the SSD is doing the purging outside of OS intervention.. the OS must specifically mark pages for purging.
The GC on the drive in question is specifically designed to provide performance improvement for *OS that do not have TRIM*. If you don't believe me, look up the drive model in the paper and google it for forum discussions of its behaviour (or buy one for yourself, and watch it happen in realtime using the probe program! It's freaky to see a drive rapidly purge it's supposedly recoverable data when connected to a non-TRIM OS)
Graeme.
A few drives offer it, with varying names and degrees of marketing fanfare, but there is little documentation as to the precise workings of the GC features. In the paper PDF you'll see the drive make/firmware for the drive we used in these experiments. You can buy that one if you like :-)
Graeme.
"...a foregone conclusion ever since ATA Secure Erase and TRIM were introduced?"
Yes, TRIM is eventually going to change the game in the sense that data will (probably) be expunged ASAP by the drive when you delete them (as you observe though, it is a request to the drive, not an instruction, and consequences of issuing a TRIM are unknown). But it could be a decade before courts will stop seeing the problem mentioned in this paper. I think the problem will peak in coming years and decline e.g. 4-5 years out as people finally give up on XP/2003, non-TRIM linux implementations of FAT32/NTFS, older Mac OS's, etc.
TRIM only works when you have a firmware and OS that support them. Given the number of people I know using XP (and below) and MacOS Snow Leopard (and below), there's a lot of computers out there who won't be sending ATA TRIM commands to their disk. Yet SSD GC is becoming more common and more aggressive in the meantime...
I would say that court practice and legal precedent is slow to update as far as technology is concerned, and SSD GC has come out of the blue for them, and is a currently a fast-moving technology that works with most non-TRIM OS's out there, and wrecks court expectations and practices that have worked for about 25 years.
Graeme.
"This property of modern SSDs is neither positive nor negative by itself. It's a property you'd be thankful for if you were organizing protests in Iran and stored contacts on an SSD. You'd curse it if it were exploited to destroy incriminating photos by someone who'd stalked you or a loved one."
Exactly right, Jonner.
Graeme.
Hello, I'm one of the authors of the paper. To explain the apparent paradox in rough terms:
Drive data was traditionally purged manually, by having the computer tell the drive to write something else over the top of the old data. In the absence of such an overwrite, magnetically stored data persists. However, if you try that trick on an SSD, it may not work. The logical address you try to overwrite may be remapped on the fly, so that your 'overwrite' goes to some other physical cell rather than the one which stored the data. From a logical viewpoint, it looks like the overwrite worked - you can't access the data any more through your computer's OS. But from the drives point of view, the data is still there, lurking in some physical cell that is presently out of use as far as the logical sector list is concerned. A cunning firmware or a hacker with a soldering iron might still get at it.
However, separately to this, modern SSD drives use tricks to try and automatically improve their performance, and one of these tricks is to pre-empetively wipe data cells that contain data no longer referenced by the filesystem. Here, the drive is actively attempting to permanently purge everything it can from the drive, all of it's own accord, in the interests of accelerating future writes by having a pool of completely unused cells available.
Summary:
- If you're a computer telling a drive to zero over some data, the drive may lie to you a bit, and not bother to zero it.
- If you're a drive, you do whatever the heck you like, and you see the physical layer directly (unlike the computer). That means the drive can open up the NTFS metadata, looking for data cells which could be preemptively reset, and nuking that data out of existence (when it might traditionally have been recoverable to an expert).
In summary. If your drive wants to nuke something, (and we've shown, they really DO want to nuke everything they can at a few minutes notice), it gets nuked. If your PC wants to nuke something, it may or may not get nuked by attempting an overwrite.
Finally, separate to this is TRIM, which is a hybrid of the two situations - an ATA command by which the OS can signal to the drive that it would like the corresponding physical cell for a particular logical sector address to be nuked, thank you very much.
Hope that clears things up.
Graeme.