Slashdot Mirror
Software Sorts Electronic Evidence
securitas writes: "The New York Times has a very interesting article about the legal industry using new search software to sort through electronic evidence such as e-mail, documents and recovered files, and the process that they go through to make the evidence usable. It has spawned an industry."
Lawyers discover grep?
The continuing investigations into the Clinton presidency ran into a major problem. Every email sent to, and within, the administration was saved. But not indexed. Apparently there are terabytes of the stuff. On tape. Sorting through it looking for evidence is likely to take years. And then it has to go to the national archives. And that's just the email. Add in all the other digital content they generated, all of which had to be saved, and there's a major problem.
Best Slashdot Co
I worked for one of the state level governments in N.A. and had access, and "da-buck-stops-here" responsibility for the IT side of "Archives". Archives is leglislatively required to hold in permenant storage, "All materials relating to the ongoing business of the government". This caused some real problems:
1) we had a case of an outgoing elected official low level formatting their HDD on the way out the door. Had to be sent out to a special data recovery lab. (they can do some amazing things with scanning electron microscopes on half tracks and such)
2) there are stacks and stacks of 8" floppy disks, in formats like IBM DisplayWriter, and other chunks of physical hardware that haven't been seen by mortal man in 20 yrs.
Finding a chunk of info is damn tricky, but after you find it, you have to find something that can read the punchcard/papertape/magtape/floppydisk/harddisk in question. And due to a querk in how the original act was written (keeping in mind that these things were written back when data was carved on rock slates and format isn't a big consideration) we were required to keep it in its original form.
I feel for someone with my job in 50 yrs. I ran away from govt work after that. It was scary!
One plus side. EMP has a hard time taking out papertape!
On the whole, I find that I prefer Slashdot posts to twitter ones because I don't get limited to 140 chars before
In reality, the biggest difference between grep and so-called "forensics" software is the emphasis on examining the data without modifying it and maintaining the chain of custody and audit trail. In fact, many experienced computer investigators do their jobs with little more than DD, grep, and various other Unix utilities. Most of the digital forensics software out there simply attempts to make this funcionality more accessable to your less tech saavy investigator. (The problems caused by inexperienced/unqualified investigators performing this type of analysis are beyond the scope of this response.)
I am currently the designer and project lead for a cross-platform open source (GPL) digital evidence processing suite. It is intended to bring together the various functionalities required to perform this type of work, and (ideally) operate on whatever platform the investigator desires. Our primary development platform is RedHat 7.1.
There are currently software packages out there that attempt to do this, including EnCase and The Forensic Toolkit in the commercial arena and The Coroner's Toolkit in the open source arena, however they lack the broad filesystem support and/or true ease of use to make them usable by everyone. The other barrier is price as EnCase, for example, costs thousands of dollars per copy.
We're well funded, and have already done a significant amount of work. We have some of our core components functional and plan on starting beta testing and releasing our first code drop later this year. If this field interests you and you'd like more information, or you work in the investigative field and have thoughts on what you'd like to see in such a tool, I'd love to hear from you.
---
One criticism of the NYT article is that it makes it sound like the legal profession just yesterday caught on to digital discovery and forensics. Although there are always some Luddites out there, lawyers who do major commercial or product liability litigation have been using digital techniques for years.
As far as user-friendly interfaces for forensic-ware, and other suggestions by comment-posters for improving the technologies, don't forget that in order to be useful to a lawyer, digital forensic evidence must be admissible in a court of law. Nobody is going to settle a lawsuit based on some damning piece of deleted email recovered from their hard drive, unless you convince them that the jury trying their case is going to see a big blow-up poster of all the bad things they said in it. In order to get that recovered data into evidence (at least in the USA), the lawyer must "lay a foundation" that the evidence has some reliability. An eyewitness to an event, for example, can testify about things she was able to see or hear from her particular location, but her testimony about what might have been happening out of her eye-earshot is not admissible in court. Another way to lay a foundation is through a qualified expert opinion, for example, an accident reconstruction expert who measures the skid marks and applies a scientific method to determine whether the car was speeding before the accident. The point being, even if I as a lawyer could read up on the relationship between skid marks and vehicle speed, make those measurements on my own, and perform the calculations just as accurately, that would not do me a bit of good. I would still have to go out and retain someone with considerable expertise in such matters in order to get the court to admit the results of the calculations into evidence, or I never get to put them on my blow-up poster for the jury. And this is not just a gimme. Especially in federal court, there are specific criteria for the qualifications that an expert must have, and the demonstrated reliability of the expert's method, before the results can be admitted in court.
So for those of you who are devising tomorrow's user-friendly forensics - a warning. No matter how point-and-clicky you make them, my lawyer colleagues and I will likely never touch them. Even though I am technically literate enough to grep anything you can grep, I'll keep on hiring one of you technical experts when I need some digital forensics done, because I need your experience, credentials and signature to convince a court that the results are reliable and not just wishful computer hokey-pokey by a lawyer who wants her client to win. (Also, lawyers don't testify in their own cases, as a rule, for various reasons.) This is especially true with things that *sound* somewhat unreliable, like recovering from low-level formats and such. The more extrapolation and guesswork is involved in the "recovery," the less likely it is to get into evidence.
And if you're developing a search method, or some other new technique for data recovery, keep in mind that in order to qualify yourself and the technique as proper expert testimony, you're likely going to have to disclose quite a bit about how you did it in order to lay the foundation for admissibility. You can just throw those valuable little trade secrets and patentable methods out the window. That's another reason why legal tech forensic shops tend to rely on things like grep and dd rather than innovating - where's the big payoff? Now if you don't care about admissibility, and are just mining the hard drives of your ex-employees (or ex-spouses, or whatever) for business reasons, maybe that's a different story. But most people don't think they're about to get into a lawsuit until it happens, so I wouldn't be so sure.
No, no, no. This is not a sig.