Slashdot Mirror
SSDs Cause Crisis For Digital Forensics
rifles only writes "Firmware built into many solid state drives (SSDs) to improve their storage efficiency could be making forensic analysis at a later date by police forces and intelligence agencies almost impossible to carry out to legally safe standards, Australian researchers have discovered. They found that SSDs start wiping themselves within minutes after a quick format (or a file delete or full format) and can even do so when disconnected from a PC and rigged up to a hardware blocker." So either SSDs are really hard to erase, or really hard to recover. I'm so confused.
Deleted, should mean deleted.
At a guess this is caused by mounting with the discard option, or trim as its called in Windows. It tells the drive you don't need the data stored where a deleted file used to be.
Maybe it's still there if you look with a microscope but who really does that?
You need to disassemble the drive and read the memory chips independently of the controller. I believe I read this is how one of the major drive recovery companies is handling SSDs.
Problem solved. People need control over their own privacy. Tough luck Digital Forensic folks.
..destroyed overnight, go with the SSDs. The melting point of a surface mount IC is a lot less than that of a spinning platter.
So either SSDs are really hard to erase, or really hard to recover. I'm so confused.
All I know is that if SSDs were really hard to erase, and I was in the business of recovering data that other people didn't want recovered, this is exactly the kind of story that I would tell them so that they would continue using SSDs.
Not that I'm paranoid or anything.
Why the confusion, dear editor? This should be well understood.
If you want to recover, you can't. If you want to erase, you can't. It's Murphy's Law of Data Storage.
When Mindy the undergrad accidentally deletes her term paper and would be really REALLY grateful for a super smart and kinda cute geek to go in and recover the file with Backtrack... then you'll see the downside.
What? I reject your reality and substitute my own!
Forgive my ignorance, but how is this possible? Does this mean that the drives understand NTFS and are actually zeroing out data on the drive when the OS simply deletes the entry from the FAT table? How can the SSD second guess what the OS is doing? I thought that SSD's use the same interface as regular HD's and should behave the same.
The whole point of the referenced article is that it is somehow a "problem" that data deleted (and intended to be deleted) by the owner of the SSD cannot be later recovered. Why should deleted data be recoverable? Will "police state" now require SSDs to stop this seemingly desirable behavior to ensure evidence be recoverable from an impounded device? I for one applaud the behavior of these new storage devices.
The drives have internal overprovisioning and perform internal garbage collection. This means that marked for deletion data has an unknown lifetime and may disappear at any point without interaction from a controller.
The hard to erase bit means that you really can't be sure something is totally erased without a full specific erase command to all flash blocks. Without that a page marked unused but not erased may be nestled in with a bunch of valid pages. As all pages in a block are erased together that marked unused page can hang around for a wile.
On the other side the firmware does garbage collection it actively looks for blocks with many erased pages and then tries to consolidate things so it can create more free blocks. This means if the drive is powered but not connected to a host machine it can still be doing data moves for consilidation and erasing marked for deletion pages.
There are thresholds for the garbage collection so it won't overwork and try for 100% consolidation. Thus you get both the presence of some really sticky stale marked unused pages and some active erasing of others.
I'm on the fence about this, and it's possible neither pasture is green. On the one hand, I might be the victim of a genuine crime, evidence of which happens to be hiding in an SSD drive. On the other hand, these techniques are just as routinely abused now to go after people for political noncriminal reasons that don't serve the Common Good at all, people and organizations like Julian Assange, Wikileaks, Bradley Manning, the U.S. Chamber of Commerce opponents... you name it.
These techniques are like nuclear physics: just as easily applied for Bad Things as Good. If we can't selectively prevent the abuses, maybe we should err on the side of caution and ban the techniques altogether. They aren't being universally applied to serve justice.
I thought that this was particularly telling. In the article it said:
... the state of the drive cannot be taken to indicate that its owner did or did not interact with it in ways that allow prosecutors to infer guilt or innocence. The fact that data has been purged does not mean a human knowingly did it (e.g. accidental guilt)...
So in other words, until SSDs came along, evidence of purged data was evidence of guilt... at least in Austrailia.
It's not hard really. The drives autoclean themselves. So when you delete the inode reference to a given data file, it will be completely much more quickly that on a normal magnetic drive (which won't reclaim the space till it's needed). On the other hand it won't respond to commands that would FORCE a magnetic drive to completely wipe the file.
So if you're in your secret lab and you hear that the evil enemy is an hour away, you just type "rm /*" and wander off to escape. By the time they get there all your data will be completely wiped. On the other hand if they are breaking down the door to your secret lab and you have only seconds left you can't type "shred /home/joeari/secretfile" and expect it to be perma-deleted like you could with a magnetic drive.
They recover deleted sectors more quickly, but can't be forced to do so in a controlled manner.
I don't need a million points of light, just two points of multi-mode fiber and a 10 Gig-E router.
...a foregone conclusion ever since ATA Secure Erase and TRIM were introduced?
Secure Erase basically tells the SSD that all of its cells are now blank (AFAIK implementations actually zero the drive as well but I'm happy to be corrected on that); therefore as soon as anything is written to the disc, it will be written here, there and everywhere. It took about 30s to run on my first vertex and I couldn't find any trace of
TRIM support in the ATA spec, along with kernel/filesystem support, tells the disc that when file A is deleted, cells X, Y, and ABQ are now officially "empty" and that if the controller feels like it, it can zero them out, shunt other data in there, or have a mardi gras for all it cares. The same happens when a drive is formatted; OS tells drive controller "I've just formatted you" and for the sake of preserving performance the controller goes "Brilliant! I can chuck out all this shit I've been saddled with."
As soon as hard drives start intelligently erasing/shuffling bits of themselves about so that cells are utilised to their utmost efficiency this was bound to happen. Unlike spinning platters where bad blocks were reallocated only if a) the hard disc knew about it and b) the data could actually be read/recovered, it becomes terribly obvious that data on SSD's is going to be read and written and deleted completely and utterly all over the place, without sequential series of sector found in slackspace like you would on a magnetic drive.
Magnetic drives have no performance penalties for not actually erasing the data, so if you work your way around that double negative you'll see that one of the staples of digital forensics (e.g. recovering files from slack) is a by-product of people trying to make magnetic platters as fast as possible by not actually erasing stuff, because as long as the controller knows that sector is blank then it'll just be overwritten as needed. Technology has now changed sufficiently that the performance gains from new solid state tech are helped by a drive controller that erases stuff as soon as possible, since writing over an occupied cell is slower than writing over a blank one.
I'm sure there'll be new methods to mitigate the change in tech, we're just somewhat on the cusp of a completely new tech. They'll probably come to an agreement that TRIM doesn't actually delete stuff until the amount of free space in the cells reaches a certain threshold or something like that.
Disclaimer: I'm not a digital forensic scientist, but am friends with one and we discussed this problem over some exquisite cocktails a few months back. And I don't think TRIM instructions follow the exact specifications I laid out above (e.g. using Brilliant! as an ACK).
Moderation Total: -1 Troll, +3 Goat
The do not completely erase when you hit them with a commercial erase tool because the commercial erase tool does not give the drive a chance to "take a breath". If you hit it with a DOD wipe instead of overwriting old blocks the drive will give you new until it runs out of spares and after that will slowly overwrite some old. At the end some data will remain.
You need to hit it with short write bursts which are alternated by rest periods so that the controller can reorganise the flash especially if you use the TRIM ATA command to mark stuff as "really deleted". As a result the data will be wiped to a standard which no hard drive can achieve.
I can write a perl 10-liner that feeds DD to do that in about half an hour. So any geek can erase one. However the commercial and most importantly certified by banking and govt tools cannot just yet. They definitely will as there is no rocket science here. Until this happens you will hear both sides of the story depending who you talk to.
By the way the same will apply to overlapping recording and drives with flash cache. A "dumb" wipe will not wipe them. At the same time a script any sysadmin can write in his spare time can wipe them so effectively that no forensics can get the data back.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
Most geeks that would try to leverage something like that to get laid will still fail at getting to the getting laid part.
Most girls that would dangle that sort of carrot know that teasing is just as effective as giving where geeks are concerned.
My God! It's full of eval()'s.
Why does the government have this expectation that technology should be built in order to make it easy to spy on citizens?
Lately all you have heard is the complete opposite. That they are impossible to completely erase so it's unsafe to store company/secure data on them. Because even if you erase the file its still left on the disk and just marked as empty. Now they say they erase them self.
you've discovered the paradox that exists in the gulf between "in theory" and "in the real world". In the real world, a lot of shit that people get lathered up about is completely impractical and for all real-world purposes "impossible". But that doesn't stop some tech "editor" on a slow news day from hype pimping some theory as if it's the pending zombie apocalypse.
Business Tip #1: Get payment up front.
Only a problem if the retard windows admin disabled the recycle bin because it interfered with his OCD.
Do you even lift?
These aren't the 'roids you're looking for.
You wrote: "So if you're in your secret lab and you hear that the evil enemy is an hour away, you just type "rm /*" and wander off to escape. By the time they get there all your data will be completely wiped. On the other hand if they are breaking down the door to your secret lab and you have only seconds left you can't type "shred /home/joeari/secretfile" and expect it to be perma-deleted like you could with a magnetic drive."
Actually, we found something even more interesting when we prepared this paper.
An evil criminal could run a quick format (about 8 seconds effort) if police were at the door, or they could set up a dead man switch to do the same. When the investigator powers up the drive, the drive's internal GC runs quickly (we couldn't get any formal documentation about why this is - presumably it begins so quickly since the filesystem metadata is nice and simple to process) and so after about 3 minutes it begins wiping itself.
Just a few minutes later it has finished, and data and old filesystem metadata are gone (at least, gone as far as logical access to drive sectors is concerned).
I'd encourage anyone who finds the result interesting to take a glance at the original paper, it's written to be accessible to people outside the traditional drive forensics community and there are some fun, free script tools you can use to monitor drive GC behaviour in near-realtime.
Graeme.
I think it's about impoliteness when asking favors. Friends help each other: I'm quite ok fixing my non-technical friend's WLAN, just as he's ok with giving me a hand when I'm moving houses. That's the social norm, and thus asking someone for a favor also indicates how you feel about them.
So if you don't like someone, if you would under normal circumstances not want to spend time with them - then you don't ask them for favors. That would be plain rude.
If you don't have an actual friend to provide tech services - just purchase said services.
On the other hand it won't respond to commands that would FORCE a magnetic drive to completely wipe the file.On the other hand it won't respond to commands that would FORCE a magnetic drive to completely wipe the file.
So why isn't a "wipe everything, including the spares" command part of the SATA standard command set? I can think of many times I'd like to completely erase a drive. Basically, any time I run the equivalent of "fdisk, o" in whatever OS I'm using, I'm explicitly telling the drive, "I value nothing here". That would be a prime time to optionally tell a drive to totally erase itself.
There are even non-security-related reasons for doing this. IIRC, the reiserfs recovery tools would scan a partition to look for data laid out in reiserfs-formatted chunks and undelete them. That's normally exactly what you want. However, suppose you're using reiserfs and running a VM that also uses reiserfs on its virtual disk, which lives inside a file on your main filesystem. The recovery tools would see those structures, too, and merge them back into your directory structure. This is typically disastrous.
When programming in low-level languages, it's nice to know that the only thing in your process's heap is data you've explicitly put there. The same holds true for hard drives. While we're obviously pretty good at keeping the wheat separated from the chaff, it'd be nice if you could guarantee that there was no chaff to start with.
Dewey, what part of this looks like authorities should be involved?
It's nearly orthogonal. Confidence and power are what counts. Being nice or a jerk, while displaying weakness or impotency, won't get you anywhere. Being a jerk sometimes projects the illusion of strength, true; but it's not necessary. Being nice without being wishy-washy and fawning is best for a long-term relationship.
You often get laid for being a dick. Hell, if you're in the right place at the right time, with low enough standards, you can get laid just for *having* a dick.
Learn about Photography Basics.
Err...we're talking about getting laid often here...not long term relationships.
Light travels faster than sound. This is why some people appear bright until you hear them speak.........
Plaintiff's Attorney: "Sir, what are the chances of the drive automatically generating the exact sequence of bits required to form this email?"
Expert Witness: "Billions to one, certainly."
Defendant's Attorney: "And how many times will that this 2KB email fit on the drive?"
Expert Witness: "Well, it's a 2TB drive, so... about a billion, give or take."
Defendant's Attorney: "So, assuming the data on the drive is random, then it's safe to say there are at least two billion opportunities on this drive to produce this email?"
Expert Witness: "That's not what I meant..."
Defendant's Attorney: "Yes or no?"
Expert Witness: "Well, yes, but..."
Defendant's Attorney: "No further questions"
Learn about Photography Basics.
It used to be the drive remapping meant that if you deleted something (or even overwrote it), there was no guarantee it would be gone - SSD controllers do wear leveling to avoid having part of the drive get used excessively. Forensic analysts could go pull the raw data from the NAND flash.
The problem was that the wear leveling algorithms need a "free block" pool to work well. Drives that have been used heavily deplete the free block pool, and the drive slows down. For a long time, SSDs would have no knowledge of whether a file had been deleted or not.
The ATA TRIM command was added for just this purpose - with TRIM, when the OS deletes a file that references a block, it can tell the SSD controller that those blocks are free. The SSD controller will then begin erasing those blocks in its free time. (SSDs can be written one block at a time, but must be erased one page at a time. A page consists of multiple blocks. Oh, and I may have page/block swapped here.) So you get a lot of performance improvement by having a bunch of pre-erased pages - these can just have individual blocks written without a read/erase/modify/write on a whole page.
Pre-TRIM, SSDs were probably great for forensic analysts. Post-TRIM, SSDs are not. Oh, and I think the latest ATA standard added a "sanitize" command to make life easier for information assurance types, for whom SSDs have always been a pain.
retrorocket.o not found, launch anyway?
Actually the man page states that on ext3 file systems running in the default manor (data=ordered) shred works fine. If you journal data as well, not so good.
The underlying file system in use is no guarantee either. Software, such as word processors often write out a new file each time you save, and if that is successful, they then delete the old one. This defeats shred because shred requires a file to exist to delete it. (It does not shred un-allocated space).
Sig Battery depleted. Reverting to safe mode.
Not likely in this situation - here, the GC isn't killing the data for the hell of it, it's killing it because it needs to reset the cell preemptively in order to maintain drive performance at a high level. If you preserve the data in the manner of an old HDD, you nuke performance by forcing the controller to do a reset before future writes.
Graeme.
But what happens if you use a different file system the drive makers didn't think of? Do you risk losing data because the SSD misinterprets your file system structures?
The Tao of math: The numbers you can count are not the real numbers.
You've hit the nail on the head.
But, to be fair, consider the amount of digital evidence in the world today vs. 20 years ago. Training up court-approved forensics officers to dismantle drives and their firmware in a manner which doesn't wreck the data, is a heck of a lot harder to do than teaching them to run 'dd' with a write-blocker attached.
Graeme.
I've spent the last hour writing replies for questions/comments on the thread using this throwaway account: could someone with uber-powers please mod them up a bit?
Thanks very much in advance, and thanks also to everyone making suggestions/comments about the article. :-)
Graeme.
However, suppose you're using reiserfs and running a VM that also uses reiserfs on its virtual disk, which lives inside a file on your main filesystem. The recovery tools would see those structures, too, and merge them back into your directory structure. This is typically disastrous.
Sounds like it would be murder on your drive's directory structure.
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
Hello, I'm one of the authors of the paper. To explain the apparent paradox in rough terms:
Drive data was traditionally purged manually, by having the computer tell the drive to write something else over the top of the old data. In the absence of such an overwrite, magnetically stored data persists. However, if you try that trick on an SSD, it may not work. The logical address you try to overwrite may be remapped on the fly, so that your 'overwrite' goes to some other physical cell rather than the one which stored the data. From a logical viewpoint, it looks like the overwrite worked - you can't access the data any more through your computer's OS. But from the drives point of view, the data is still there, lurking in some physical cell that is presently out of use as far as the logical sector list is concerned. A cunning firmware or a hacker with a soldering iron might still get at it.
However, separately to this, modern SSD drives use tricks to try and automatically improve their performance, and one of these tricks is to pre-empetively wipe data cells that contain data no longer referenced by the filesystem. Here, the drive is actively attempting to permanently purge everything it can from the drive, all of it's own accord, in the interests of accelerating future writes by having a pool of completely unused cells available.
Summary:
- If you're a computer telling a drive to zero over some data, the drive may lie to you a bit, and not bother to zero it.
- If you're a drive, you do whatever the heck you like, and you see the physical layer directly (unlike the computer). That means the drive can open up the NTFS metadata, looking for data cells which could be preemptively reset, and nuking that data out of existence (when it might traditionally have been recoverable to an expert).
In summary. If your drive wants to nuke something, (and we've shown, they really DO want to nuke everything they can at a few minutes notice), it gets nuked. If your PC wants to nuke something, it may or may not get nuked by attempting an overwrite.
Finally, separate to this is TRIM, which is a hybrid of the two situations - an ATA command by which the OS can signal to the drive that it would like the corresponding physical cell for a particular logical sector address to be nuked, thank you very much.
Hope that clears things up.
Graeme.
YOU SHOULD REALLY WARN PEOPLE about what you posted! I googled to see what you were talking about and now I'm worried google thinks I'm a pedo thanks to you.
I think you missed the point. Say you've installed Ubuntu on a reiserfs partition inside a VM. The VM's harddisk file is stored at /var/tmp/ubuntu.vdi. For some reason, you need to fsck /var. It reads each block in /var's partition and looks for things that look like inodes and merges them back into the directory structure tree. Guess what happens when it starts scanning blocks that are inside ubuntu.vdi? It finds a lot of things that look like inodes and merges them back into the directory structure tree. Don't believe me? Fine. Maybe Theodore Ts'o can convince you.
If you want to wipe a hard disk, dd if=/dev/zero of=/dev/sda1
I don't think there's a single modern SSD or HDD where that will clear every bit of storage on the device. Both drive types keep spare blocks to swap into rotation whenever old ones become damaged and it's impossible to wipe those blocks with dd. Hence the need for a native drive command to do it. Apparently the drive designers thought so, too, and made a command to do that.
Dewey, what part of this looks like authorities should be involved?
A few drives offer it, with varying names and degrees of marketing fanfare, but there is little documentation as to the precise workings of the GC features. In the paper PDF you'll see the drive make/firmware for the drive we used in these experiments. You can buy that one if you like :-)
Graeme.
Trust me, dd can really make a mess of things, if by intention or mistake.
I was cloning one drive (A) to a larger one (B). It needed to preserve the partition table, MBR, etc.
dd of=/dev/sda if=/dev/sdb bs=1024
About 10 seconds after I hit enter, I saw the mistake, and aborted it. Ya, I dd'd an empty drive over the populated one. How much harm could that possibly do, right? :)
I've done plenty of drive recoveries, with about an 80% success rate. The only ones I can't do are drives that don't spin. I don't have the facilities to transfer the platters.
Two weeks and several recovery attempts, I had recovered about 25% of what I wanted. The partition table, and the MFT were destroyed, and files were on four partitions with two different filesystems (NTFS and ext3). That was in 10 seconds. If you were to say" dd if=/dev/random of=/dev/sda bs=1024", it's rather unlikely even the best would find anything.
From what I understand, the best chance at recovering data from an overwritten media is to look at sectors marked as bad. They are automatically skipped, so they don't get overwritten by anything. Who knows what will be in them though. Most likely on most users machines, it will be pieces of the browser cache, or Windows Update files.
That's not to say if some agency named "No Such Agency", were really bent on finding out what I had on there, they couldn't recover something. Many agencies can't go beyond files that currently exist, or are in the "Recycle Bin". Some go as far as undeleting from an otherwise functional filesystem. Is it worth weeks and thousands of dollars for whatever organization to recover what you have hiding on your desktop? Probably not. If you have secrets, don't hide them on your PC at home, even if you are protected by a nice wood front door with fancy security locks. One brick through the window, and a fast getaway, will defeat any door lock and alarm system. If you kept all of your important information on some random colocated server in another country, AND you have a second person with access to clear your data, it would be untouchable by most agencies. Say I had a server colocated in China, with the normal protections (encrypted virtual filesystem, etc, etc.), and only used my desktop to manipulate them through a terminal session (VNC, SSH, etc) the most that could be found is that I connected to a server in an untouchable place, and by the time they managed to get to it, my 2nd party could have removed it.
Serious? Seriousness is well above my pay grade.
yep, at least on my drive it does.
(note I emulated this by removing the sata cable and not the power cable, as opposed to a nifty diode steered backup battery).
re-connected the drive after 5 min, and re-enumerated the drive, and it was rather blank.
-nB
whois gawk date unzip strip find touch finger mount join nice man top fsck grep eject more yes exit umount sleep dump
You mean, something bizarre like accidentally restoring a boot sector to the wrong partition?
Normally this wouldn't do any harm as long as you fix the problem before you mount the partition again. However, if the drive interprets the data by itself, even immediately fixing this may be too late.
The Tao of math: The numbers you can count are not the real numbers.
You're off by so many orders of magnitude, it hurts.
The chance of generating that particular e-mail, if it's 2 KiB, is 1 in 2^16384. (That's not really the number you want -- you want the chance of generating a similarly-incriminating e-mail. They're roughly equally improbable, though.)
So that's 1 in 2^16384 compared to 2^30 copies of the e-mail that would fit on the drive.
If the prosecution scoured every drive ever made for a random sequence of bits that looked like that e-mail, they'd never find it.
Plus, the expert witness generally has a better and less technical answer. Defense lawyers don't ask technical questions like that: the expert witness will undoubtedly have an answer, and you'll bore the jury to tears, which they hate.