Slashdot Mirror
To Purge Or Not To Purge Your Data
Lucas123 writes "The average company pays from $1 million to $3 million per terabyte of data during legal e-discovery. The average employee generates 10GB of data per year at a cost of $5 per gigabyte to back it up — so a 5,000-worker company will pay out $1.25 million for five years of storage. So while you need to pay attention to retaining data for business and legal requirements, experts say you also need to be keeping less, according to a story on Computerworld. The problem is, most organizations hang on to more data than they need, for much longer than they should. 'Many people would prefer to throw technology at the problem than address it at a business level by making changes in policies and processes.'"
The problem is that it's easier to just archive the cruft stuff than it is to go through it all and figure out what's worth keeping.
http://www.geoffreylandis.com
$250k a year for a 5000 employee company? To put it in perspective, if the average employee at this company is making $60k a year, this company will be paying $1.5 billion in salaries over the same 5 years. To be fair, I think the estimated cost from the article is very much underestimated. But while corporate storage costs more than you'd think, and companies are definately storing a whole bunch of data they don't need, what about the costs of reviewing and purging that data? That is straight up time, whether it's reviewing existing data or spending the time to create guidelines for which data to keep. And time costs money. More than storage.
Whale
10 GB of data per user, sure.
10 GB of user data, no way.
If assuming 300 work days per employee, that would mean that the average employee creates 1.2 kB of data per second.
The only way this could be true is if you count data that isn't user generated, and they count the total data storage for the company and divide it by employees.
If so, users deleting their e-mails won't have much of an effect.
Apps aren't really well designed for this in mind. They don't come at the problem from a "document lifecycle" perspective but instead a "document creation".
This is generally because data has a variable lifespan. Lets take an email as part of a project as an example. As the author I may decide that the email isn't needed after a week so set an expiry of 1 week. But you, as the recipient, may take that email and turn that into several tasks so for you the email is much more important and thus want to keep it for much longer.
Users aren't really going to be good at making these decisions unless some application continually bombards them with "go check the status of these 1000 documents you've got".
It's not so much that you want your company to have a leg to stand on, its that you don't want your legal opposition to get their foot in the door. Innocent until proven guilty remember?
In Soviet Russia meme tires of you!
For example, Financial institutions are required to keep data for longer period for legal purpose as well as traceability (during investigation of fraud or other kind of crimes). The banks worked for had legal requirement of keeping data at 2 places at least 15 km apart, with all kind of protection against fire and intrusion.
A good manufacturing company would keep data for longer period ot only to comply with ISO standards, but to trace manufacturing defects and a good evidence of past history for insurance company against theft/fire and other kind of problems.
We used to keep daily changes of source code of only previous releases, and purge rest of of the releases (we kept the final source code and patches of all previous releases, but purge daily changes).
In a nutshell, it depends upon your type of bussines.
hilarious
The top 500 company I worked for did just the opposite: Destroy all data in case a legal issue comes up.
They called it 'desk cleanout day', and unless you were an official dedicated contact on a particular subject you were to wipe all correspondence of more than a year old.
(There were also other grades of information, but erase after a year was the default).
I don't know what most major companies' policies are regarding backing up emails (just back up the text or back up emails plus attachments) but, as but one example, I'm sure this would be an easy spot for most companies to dramatically reduce the amount of storage space required. Most business communications I see from corporate personnel have various attachments on every email - things like logos, custom backgrounds, etc. Forget getting rid of all the unnecessary attachments - getting rid of the "look at my pretty email that looks like a page from a spiral-bound notebook with my company logo at the bottom" images, and the hundreds and thousands of duplicates of those images, would reduce storage requirements, bandwidth requirements, and probably make corporate communications look more, you know, professional. So many emails are filled with unnecessary garbage and, if that's being backed up, that garbage can get costly.
Then again, I'm biased - I believe email should just be pure text. Perhaps that's a sign that I'm now old...
My last job had some files from the 1890's. The company had moved from New York to New Jersey to Houston in all that time. I can't imagine that material would ever need to be used, or would be called up during a legal investigation. Even if it were, would the authorities penalize a company for files that were that old??? At some point, everything is trashable or museum material.
This company occasionally needed blueprints from the 1930s/1940s (great lakes ships), but none of their ships went back much further than that.
Even those who arrange and design shrubberies are under considerable economic stress at this period in history.
In a world where backup takes money, a law that says to companies "keep every communication backuped" is saying essentially the same thing as "communicate less".
The Wise adapts himself to the world. The Fool adapts the world to himself. Therefore, all progress depends on the Fool.
put everything on one disk drive, unRAIDed. when it fails, problem solved. voila, built in obsolescence
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Unfortunately, writable DVDs are not an acceptable archive medium, and a stack of disks with written labels is not an indexing solution that will scale beyond one person.
If assuming 300 work days per employee, that would mean that the average employee creates 1.2 kB of data per second.
Top posting and absence of editing by Microsoft Outlook users engaged in a brief inter-departmental discussion could easily account for that volume.
Is that what you meant by "isn't user generated"?
I did a back-of-the-envelope calculation on just this question in 2004, and estimated that file deletion was not productive unless we could do it at a rate of at least 17MB per minute (of labor). Four years later the threshold is probably at least 45MB per minute.
Generally, this means that if we can blow away whole disks or huge directories of data, it may pay off. Users going through their files one by one is usually an absolute waste.
"Not an actor, but he plays one on TV."
Any record destruction policy must include a "litigation hold". A litigation hold means that record destruction must stop when litigation is anticipated or pending. But in a complex enterprise, it is tricky to know what litigation the enterprise anticipates. It was the trickiness of litigation hold that led to the demise of Arthur Andersen. The risks associated with litigation hold give enterprises incentive to store lots more records. --Ben http://hack-igations.blogspot.com/2008/07/document-discovery-litigation-hold.html
Benjamin Wright, Dallas, Texas, benjaminwright.us
Congratulations. You're the first person I've seen who understands that.
Accounting understands the need to close one year and open the next. They have processes for what is carried over and how it is identified.
Yet no other department (or application) understands the need to close old data and archive it.
That was a common company wide AT&T policy wipe everything after 60 days. all email to be deleted after 60 days. it was a fireable offense for creating a pst file on your desktop and we did a regular sweep for pst files on corperate pc's on a regular basis.
It really did not stop anyone from keeping info, many managers simply printed out the emails and kept them in files, one IT manager we let go had 3 years of email printed and stored in file cabinets in his office. it was insane.
Do not look at laser with remaining good eye.
Let's say your corp is more than 50% likely to go through "e-discovery" once every 10 years. Each worker will generate 10GB * 10 years = 100GB, backing up all the increasing data pile is (pairing the balancing ends of the accumulation for half the accumulation years) 101GB * 5 = 505GB, at $5:GB is $2525, plus about $2M:TB / 505GB = $1.01M, for a total of $1,012,525 per worker, times at least 0.50 probability is at least $506,262 average predictable cost per employee.
One approach is to keep much less data. But when you keep less data, you have to guess right every time what data you'll need later. If your process discards data that's valuable later (but lost) it better be worth less than the amount you save. That's too hard to know, which is one reason companies keep all the data, and figure it out later.
A better approach is just to cut that $1-3M:TB e-discovery cost. Of course, the best way is to avoid being investigated, but one has less than 100% control over that, especially from inside the IT department. A much better way to do it is to better inventory the data stored as you go along accumulating it, in the terms in which a later e-discovery would search it. Which also can have the benefit of making the info in the data more available in the normal course of business, which can make that data's increased value (and lowered costs of searching it) worth the entire process. The cheaper possible e-discovery would be just a bonus.
What really gets me is how these economics are the true cost of storage. A 1TB drive costs $120, and maybe a better 1TB in a 100% redundant RAID costs $250. But it really costs something like $300,000 over its lifetime (probably replaced every 3 or so years, across the 10 years I analyzed). If IT spent a few hundred hours a year streamlining the navigation of all that data, at a cost of a few dozens of thousands of dollars, divided across all those employees, the entire org's IT operations would be much more economical, when the large cumulative risk of e-discovery costs are factored into the true cost.
--
make install -not war
I've become the e-discovery guy (at least for email) where I work. Our lawyers told me that the latest revision of FRCP (Federal Rules of Civil Procedure) require an entity to keep evidence, even if automatic purging systems are in place.
Rule 37 of FRCP says that if you are ordered to hand over the evidence, and you cannot, then the judge can order that "designated facts be taken as established for purposes of the action, as the prevailing party claims". In other words, if the person suing you claims you sent them an email offering a million dollars to not go to court, and you auto-purge your email (taking away the ability to prove you didn't send the email), the judge has the option of deciding that yes you did make an offer of a million dollars via email. T'would suck to be you.
It even gets a little worse. Although you must keep evidence after being told you are being taken to court, it turns out you need to keep all evidence in case you are taken to court. I'm told that the criteria here is "reasonable expectation that the matter will go to court". It's reasonable (for example) to expect to end up in court if an employee dies while on the job (and it wasn't due to natural causes). The point here is that if a person dies, you'd better keep any email about the situation that lead to death - 60 day auto-purging email expiration practice be damned.
Auto-purging is a fine thing, as long as you have the ability to except items out, in case they become evidence.
"The most sensible request of government we make is not, "Do something!" But "Quit it!"