"Forgive my ignorance, but how is this possible? Does this mean that the drives understand NTFS and are actually zeroing out data on the drive"
- Yes, that's pretty much what they're doing, all under their own control without the PC asking them to. But they are not so much zero-ing out the data, as resetting the cell to prepare it for being written to in future more quickly. Not all drives do it, but increasingly it's built in since the technology is already written now, and people running non-TRIM OS's can benefit from it a great deal.
- Re: zeroing. Oddly enough, I seem to recall that cells are actually reset to all 1s at the physical level...
"I cannot see any government worth its credibility endorse a product which if employed in crime and confiscated (by police), it is almost impossible to use it to prosecute the perpetrators by government agencies and the FBI in the case of these United States."
Hello there, thanks for your comment. Just a couple of quick points:
- SSD garbage collection doesn't destroy data indiscriminately, it destroys data that the user has manually chosen not to retain. In many ways, this is not substantially different to a user choosing to turn off their computer and losing all the data in RAM chips which they have manually chosen not to save permanently. Should we ban RAM as well, along with any device which loses non-archived data in a manner which can't be recovered from?
- Keep in mind the evidence that is destroyed can be used to prove innocence as much as guilt. It's not actually a net loss as far as courts are concerned.
I've spent the last hour writing replies for questions/comments on the thread using this throwaway account: could someone with uber-powers please mod them up a bit?
Thanks very much in advance, and thanks also to everyone making suggestions/comments about the article.:-)
"Plain old magnetic disks used a fairly predictable implementation of that interface so forensics goons got used to having an easy task on their plates."
You've hit the nail on the head.
But, to be fair, consider the amount of digital evidence in the world today vs. 20 years ago. Training up court-approved forensics officers to dismantle drives and their firmware in a manner which doesn't wreck the data, is a heck of a lot harder to do than teaching them to run 'dd' with a write-blocker attached.
Repeating a comment I made elsewhere on this board:
-----
Hey there
Yep, it's the answer that immediately springs to mind when you approach the problem as a techie, and we touched on in the paper (see point 16, page 13).
Keep in mind though that this is extremely non-trivial - e.g. not many court officers will know how to do it - and that a court may have real trouble with a forensics guy telling them that for 'this particular case', they need to not use the standard procedure to preserve the data unmodified, but instead to rip open the disk and muck about with how it processes data.
Courts have accepted practices that are used worldwide; what you propose is not accepted practice; therefore courts will have a problem, at least in the short term.
Also, given the number of variations of controllers and firmware out there too, I'm sure you'll agree there's tremendous potential for a court forensics officer to get it wrong and accidentally nuke the drive data. e.g. how do you find out which firmware was in use, without powering on the drive (which activates the GC)...
Graeme.
p.s. Also keep in mind that data is stored on the chips in a highly non-linear manner; reassembling the logical image may not be easy, since you need to work out how the flash translation layer was being represented. This is getting wildly beyond what a typical court forensics officer is usually trained to do, namely, rip an image of the drive with a write-blocker attached and scan it for jpegs/whatever.
"This is great news for r@ygold, hussyfan, kingpass, vicky series collectors course knowing hard drive manufacturers they will cripple the ssd's somehow so they keep ghosts of the files like older drives did"
Hey there,
Not likely in this situation - here, the GC isn't killing the data for the hell of it, it's killing it because it needs to reset the cell preemptively in order to maintain drive performance at a high level. If you preserve the data in the manner of an old HDD, you nuke performance by forcing the controller to do a reset before future writes.
"On the SSD, the TRIM command caused the device's firmware to wipe the entire device."
Close, but not exactly right. Here, the drive's firmware undertook the operation by itself using a NTFS-aware garbage collector; no ATA TRIM command from the PC was needed. We used a non-TRIM OS for just this reason.
Graeme.
Yep, it's the answer that immediately springs to mind when you approach the problem as a techie, and we touched on in the paper (see point 16, page 13).
Keep in mind though that this is extremely non-trivial - e.g. not many court officers will know how to do it - and that a court may have real trouble with a forensics guy telling them that for 'this particular case', they need to not use the standard procedure to preserve the data unmodified, but instead to rip open the disk and muck about with how it processes data.
Courts have accepted practices that are used worldwide; what you propose is not accepted practice; therefore courts will have a problem, at least in the short term.
Also, given the number of variations of controllers and firmware out there too, I'm sure you'll agree there's tremendous potential for a court forensics officer to get it wrong and accidentally nuke the drive data. e.g. how do you find out which firmware was in use, without powering on the drive (which activates the GC)...
"The drive, however, doesn't know any of this. It updates the requested sectors representing the directory and volume info, and that's about it. It has no idea that blocks somewhere else on the drive are no longer needed, so it will dutifully copy and maintain that data while rewriting blocks."
Actually, this is no longer correct. SSDs (such as the one in this study) are quite capable of examining the filesystem stored on the drive, independently, and the concept of 'dutifully' and ignorantly maintaining deleted data goes out of the window as a result.
The main point of the paper is not that this 'analyse-and-purge' drive GC behaviour happens (which is well known among SSD enthusiasts), but that it fouls up long-accepted court and forensics assumptions about what computer drives are. It can result in loss of evidence (of guilt or innocence) very easily in court cases in a manner which might be incorrectly attributed to malicious human intention.
Hello, I'm one of the authors of the original article.
The drive's firmware is responsible for remapping blocks, and so the OS can't really control what happens whenever the OS tries to permanently delete things by asking the drive nicely to do so. Consequently, if the drive decides it's best to remap the logical block silently while not deleting the cell contents, then the OS doesn't realise the data is still there. That's what the other SSD study noted.
On the other hand, in this situation, the thing doing the purging is the drive's firmware itself, not the OS, and the firmware knows for sure where the data is in the cells, and furthermore it's on a mission - to purge cells that it knows the filesystem is no longer using for data.
The purging that is being done here, is taking place with the specific intention of getting cells freshened up and ready to be written to in future without delay. Consequently we can reasonably expect that the drive firmware really *wants* to nuke cells that contain real data that is no longer needed according to the filesystem metadata since that's the only way to boost performance, and that's the GC's job.
If you want to check for yourself, try carrying out reads at the sector level yourself after running the experiment with the experimental setup described in the paper. You won't get much.
I guess what I'm saying is that both studies are right despite the apparent contradiction, since they're dealing with completely different circumstances and motivations for 'purging' - OS or drive GC controlled.
Graeme.
p.s. Does anyone know of a postdoc position in AI, robotics, or bioinformatics in Grenoble, France? I'm currently looking for a new job.
Hi, I'm one of the authors of the article (Graeme).
You wrote: "So if you're in your secret lab and you hear that the evil enemy is an hour away, you just type "rm/*" and wander off to escape. By the time they get there all your data will be completely wiped. On the other hand if they are breaking down the door to your secret lab and you have only seconds left you can't type "shred/home/joeari/secretfile" and expect it to be perma-deleted like you could with a magnetic drive."
Actually, we found something even more interesting when we prepared this paper.
An evil criminal could run a quick format (about 8 seconds effort) if police were at the door, or they could set up a dead man switch to do the same.
When the investigator powers up the drive, the drive's internal GC runs quickly (we couldn't get any formal documentation about why this is - presumably it begins so quickly since the filesystem metadata is nice and simple to process) and so after about 3 minutes it begins wiping itself.
Just a few minutes later it has finished, and data and old filesystem metadata are gone (at least, gone as far as logical access to drive sectors is concerned).
I'd encourage anyone who finds the result interesting to take a glance at the original paper, it's written to be accessible to people outside the traditional drive forensics community and there are some fun, free script tools you can use to monitor drive GC behaviour in near-realtime.
Hi, I'm one of the authors of the research (Graeme).
It's a good joke, but with a grain of truth in it. If you're concerned, you can buy the drive we used, flash it to the firmware we used, (optionally) buy a write blocker if you like, and run the programs I placed at the back of the paper and see what happens. We carefully documented as many of the experimental setup parameters as we could so it should be possible to reproduce the results exactly.
Skepticism is a good thing - so I hope you'll reproduce the experiment at home and tell the world afterwards if you still think we're running PsyOps for the forensics community:-)
Graeme.
p.s. I'm currently looking for a postdoctoral/research engineer/research scientist position in Grenoble.
At a guess this is caused by mounting with the discard option, or trim as its called in Windows. It tells the drive you don't need the data stored where a deleted file used to be.
Maybe it's still there if you look with a microscope but who really does that?
Hello there, I'm one of the authors of the paper (Graeme).
This finding isn't related to TRIM in any way, though TRIM poses another nightmare for forensic investigators and may make the idea of examining deleted data/metadata redundant in a few years.
What's happening here is that the drive itself has some code, a garbage collector, that reads the NTFS filesystem metadata and wipes any cells it thinks aren't needed any more, so that the next set of writes over those cells can take place more quickly.
Normally this GC has seemed to take a while to kick in (various benchmarking sites suggest e.g. 30-60 minutes and unpredictable behaviour, e.g. not always kicking in when expected); here, we found that after a quick format has taken place, it reliably can be seen to kick in within minutes and also purges the drive within minutes. This is just one example of a case when the GC can kick in, of course.
Current court-accepted forensics practice is to stick a write blocker between the drive and computer, under the assumption that the drive only modifies data when a PC tells it to: but that doesn't help you when the drive itself nukes all it's data cells within minutes of being powered on.
I can imagine a situation where someone connects up the drive to power, (and possibly a write blocker), but not to the PC and makes a cup of coffee for a few minutes - by the time they connect it up for data transfer, it's too late, and they wouldn't even realise it had happened - the drive doesn't exactly make any whirrs or clicks as it wipes itself.
Alternatively, the circumstances we found: in less than the time that would be taken to read an entire image off the disk for forensic study, the GC is racing ahead of you and purging the disk! Yikes. So you can imagine... a forensic investigator takes an image, examines it, presents it to the court, and when the court verifies the original disk that the copy was taken from, nothing is there any more, and the forensic investigator seems to be making it all up....
So this feature has the potential to make it look like the police or forensic investigator themselves tampered with the original disk, as well as potentially destroying evidence that could help establish guilt or help establish innocence. We've put a number of 'interesting legal grey areas' at the back of the paper that we hammered out with reviewers, which are worth being aware of.
Slashdot Mirror
- Yes, that's pretty much what they're doing, all under their own control without the PC asking them to. But they are not so much zero-ing out the data, as resetting the cell to prepare it for being written to in future more quickly. Not all drives do it, but increasingly it's built in since the technology is already written now, and people running non-TRIM OS's can benefit from it a great deal.
- Re: zeroing. Oddly enough, I seem to recall that cells are actually reset to all 1s at the physical level...
Graeme.
Hello there, thanks for your comment. Just a couple of quick points:
- SSD garbage collection doesn't destroy data indiscriminately, it destroys data that the user has manually chosen not to retain. In many ways, this is not substantially different to a user choosing to turn off their computer and losing all the data in RAM chips which they have manually chosen not to save permanently. Should we ban RAM as well, along with any device which loses non-archived data in a manner which can't be recovered from?
- Keep in mind the evidence that is destroyed can be used to prove innocence as much as guilt. It's not actually a net loss as far as courts are concerned.
Graeme.
I've spent the last hour writing replies for questions/comments on the thread using this throwaway account: could someone with uber-powers please mod them up a bit?
Thanks very much in advance, and thanks also to everyone making suggestions/comments about the article. :-)
Graeme.
You've hit the nail on the head.
But, to be fair, consider the amount of digital evidence in the world today vs. 20 years ago. Training up court-approved forensics officers to dismantle drives and their firmware in a manner which doesn't wreck the data, is a heck of a lot harder to do than teaching them to run 'dd' with a write-blocker attached.
Graeme.
-----
Hey there
Yep, it's the answer that immediately springs to mind when you approach the problem as a techie, and we touched on in the paper (see point 16, page 13).
Keep in mind though that this is extremely non-trivial - e.g. not many court officers will know how to do it - and that a court may have real trouble with a forensics guy telling them that for 'this particular case', they need to not use the standard procedure to preserve the data unmodified, but instead to rip open the disk and muck about with how it processes data.
Courts have accepted practices that are used worldwide; what you propose is not accepted practice; therefore courts will have a problem, at least in the short term.
Also, given the number of variations of controllers and firmware out there too, I'm sure you'll agree there's tremendous potential for a court forensics officer to get it wrong and accidentally nuke the drive data. e.g. how do you find out which firmware was in use, without powering on the drive (which activates the GC)...
Graeme.
p.s. Also keep in mind that data is stored on the chips in a highly non-linear manner; reassembling the logical image may not be easy, since you need to work out how the flash translation layer was being represented. This is getting wildly beyond what a typical court forensics officer is usually trained to do, namely, rip an image of the drive with a write-blocker attached and scan it for jpegs/whatever.
Not likely in this situation - here, the GC isn't killing the data for the hell of it, it's killing it because it needs to reset the cell preemptively in order to maintain drive performance at a high level. If you preserve the data in the manner of an old HDD, you nuke performance by forcing the controller to do a reset before future writes.
Graeme.
"On the SSD, the TRIM command caused the device's firmware to wipe the entire device." Close, but not exactly right. Here, the drive's firmware undertook the operation by itself using a NTFS-aware garbage collector; no ATA TRIM command from the PC was needed. We used a non-TRIM OS for just this reason. Graeme.
Yep, it's the answer that immediately springs to mind when you approach the problem as a techie, and we touched on in the paper (see point 16, page 13).
Keep in mind though that this is extremely non-trivial - e.g. not many court officers will know how to do it - and that a court may have real trouble with a forensics guy telling them that for 'this particular case', they need to not use the standard procedure to preserve the data unmodified, but instead to rip open the disk and muck about with how it processes data.
Courts have accepted practices that are used worldwide; what you propose is not accepted practice; therefore courts will have a problem, at least in the short term.
Also, given the number of variations of controllers and firmware out there too, I'm sure you'll agree there's tremendous potential for a court forensics officer to get it wrong and accidentally nuke the drive data. e.g. how do you find out which firmware was in use, without powering on the drive (which activates the GC)...
Graeme.
"The drive, however, doesn't know any of this. It updates the requested sectors representing the directory and volume info, and that's about it. It has no idea that blocks somewhere else on the drive are no longer needed, so it will dutifully copy and maintain that data while rewriting blocks."
Actually, this is no longer correct. SSDs (such as the one in this study) are quite capable of examining the filesystem stored on the drive, independently, and the concept of 'dutifully' and ignorantly maintaining deleted data goes out of the window as a result.
The main point of the paper is not that this 'analyse-and-purge' drive GC behaviour happens (which is well known among SSD enthusiasts), but that it fouls up long-accepted court and forensics assumptions about what computer drives are. It can result in loss of evidence (of guilt or innocence) very easily in court cases in a manner which might be incorrectly attributed to malicious human intention.
Graeme.
The drive's firmware is responsible for remapping blocks, and so the OS can't really control what happens whenever the OS tries to permanently delete things by asking the drive nicely to do so. Consequently, if the drive decides it's best to remap the logical block silently while not deleting the cell contents, then the OS doesn't realise the data is still there. That's what the other SSD study noted.
On the other hand, in this situation, the thing doing the purging is the drive's firmware itself, not the OS, and the firmware knows for sure where the data is in the cells, and furthermore it's on a mission - to purge cells that it knows the filesystem is no longer using for data.
The purging that is being done here, is taking place with the specific intention of getting cells freshened up and ready to be written to in future without delay. Consequently we can reasonably expect that the drive firmware really *wants* to nuke cells that contain real data that is no longer needed according to the filesystem metadata since that's the only way to boost performance, and that's the GC's job.
If you want to check for yourself, try carrying out reads at the sector level yourself after running the experiment with the experimental setup described in the paper. You won't get much.
I guess what I'm saying is that both studies are right despite the apparent contradiction, since they're dealing with completely different circumstances and motivations for 'purging' - OS or drive GC controlled.
Graeme.
p.s. Does anyone know of a postdoc position in AI, robotics, or bioinformatics in Grenoble, France? I'm currently looking for a new job.
You wrote: "So if you're in your secret lab and you hear that the evil enemy is an hour away, you just type "rm /*" and wander off to escape. By the time they get there all your data will be completely wiped. On the other hand if they are breaking down the door to your secret lab and you have only seconds left you can't type "shred /home/joeari/secretfile" and expect it to be perma-deleted like you could with a magnetic drive."
Actually, we found something even more interesting when we prepared this paper.
An evil criminal could run a quick format (about 8 seconds effort) if police were at the door, or they could set up a dead man switch to do the same. When the investigator powers up the drive, the drive's internal GC runs quickly (we couldn't get any formal documentation about why this is - presumably it begins so quickly since the filesystem metadata is nice and simple to process) and so after about 3 minutes it begins wiping itself.
Just a few minutes later it has finished, and data and old filesystem metadata are gone (at least, gone as far as logical access to drive sectors is concerned).
I'd encourage anyone who finds the result interesting to take a glance at the original paper, it's written to be accessible to people outside the traditional drive forensics community and there are some fun, free script tools you can use to monitor drive GC behaviour in near-realtime.
Graeme.
It's a good joke, but with a grain of truth in it. If you're concerned, you can buy the drive we used, flash it to the firmware we used, (optionally) buy a write blocker if you like, and run the programs I placed at the back of the paper and see what happens. We carefully documented as many of the experimental setup parameters as we could so it should be possible to reproduce the results exactly.
Skepticism is a good thing - so I hope you'll reproduce the experiment at home and tell the world afterwards if you still think we're running PsyOps for the forensics community :-)
Graeme. p.s. I'm currently looking for a postdoctoral/research engineer/research scientist position in Grenoble.
At a guess this is caused by mounting with the discard option, or trim as its called in Windows. It tells the drive you don't need the data stored where a deleted file used to be.
Maybe it's still there if you look with a microscope but who really does that?
Hello there, I'm one of the authors of the paper (Graeme).
This finding isn't related to TRIM in any way, though TRIM poses another nightmare for forensic investigators and may make the idea of examining deleted data/metadata redundant in a few years.
What's happening here is that the drive itself has some code, a garbage collector, that reads the NTFS filesystem metadata and wipes any cells it thinks aren't needed any more, so that the next set of writes over those cells can take place more quickly.
Normally this GC has seemed to take a while to kick in (various benchmarking sites suggest e.g. 30-60 minutes and unpredictable behaviour, e.g. not always kicking in when expected); here, we found that after a quick format has taken place, it reliably can be seen to kick in within minutes and also purges the drive within minutes. This is just one example of a case when the GC can kick in, of course.
Current court-accepted forensics practice is to stick a write blocker between the drive and computer, under the assumption that the drive only modifies data when a PC tells it to: but that doesn't help you when the drive itself nukes all it's data cells within minutes of being powered on.
I can imagine a situation where someone connects up the drive to power, (and possibly a write blocker), but not to the PC and makes a cup of coffee for a few minutes - by the time they connect it up for data transfer, it's too late, and they wouldn't even realise it had happened - the drive doesn't exactly make any whirrs or clicks as it wipes itself.
Alternatively, the circumstances we found: in less than the time that would be taken to read an entire image off the disk for forensic study, the GC is racing ahead of you and purging the disk! Yikes. So you can imagine... a forensic investigator takes an image, examines it, presents it to the court, and when the court verifies the original disk that the copy was taken from, nothing is there any more, and the forensic investigator seems to be making it all up....
So this feature has the potential to make it look like the police or forensic investigator themselves tampered with the original disk, as well as potentially destroying evidence that could help establish guilt or help establish innocence. We've put a number of 'interesting legal grey areas' at the back of the paper that we hammered out with reviewers, which are worth being aware of.
Graeme.