Slashdot Mirror
You've Got Mail -- Tons Of It
Daniel Goldman writes "The Baltimore Sun has an article about the City of Baltimore's email problem." A snippet: "Millions of old e-mail messages are clogging Baltimore's municipal computers, so the city is going to start automatically deleting any messages older than 90 days.
A common practice in private business, the move raises questions when made by a municipality, which has a responsibility to retain certain public records." Goldman points out "Just think about all the potential law suits; 'if it's not there, they can't subpoena it.'"
Outsource each employees email to GMail. Problem solved.
This might be a practical use of one, determine which emails are valid, and which aren't, like a spam filter. Allow users to flag 25% or so emails as important, and archive those.
got sig?
Either Google or the Internet Archive would be happy to archive that data for the City of Baltimore and keep it available for public reference.
Since they need to delete tons of old messages spam included, but want to save official email, why don't they train a Bayesian Filter to sort through and save as much as possible. Since they can't rely on their employees actually saving each message which was official to their hard drives.
~ there are 10 types of people in this world, those that can read binary and those that can't
figure out what percentage is spam, and sue spammers to recover damages for lost resources.
can't they, like, just buy a big hard drive and stuff?
If the average message is 10kib (10,000 bytes, make the math easier), and compresses down to 3kib (probably even better if you compress a bunch together), then you'd need roughly 30gib to store 10 million of them. Can you even buy hard drives that small any more?
Add some search index, throw a crappy web interface on it, and call it a day. Never delete an email again!
There are always going to be things like replies to an original question and subsequent follow up questions going back and forth, so normally hanging onto the latest/final reply would be sufficient (providing it had the previous history - clearly showed the conclusion).
Now if they were to use this as an excuse to accidently lose records that would be a different matter. This however is where auditors should be playing a role to ensure that they are keeping the right records and discarding the rubbish.
"Baltimore officials, who approved the new e-mail policy at a Board of Estimates meeting last month, say they have no choice but to delete old messages, which are slowing city computers to a crawl. They say the system is so overburdened that creating a daily backup has become impossible; there is so much data that it takes more than 24 hours to copy it."
What?!? What's wrong with an incremental backup? Surely all those millions of messages aren't *changing* every day?!?
Think of all the children that will suffer from this!!!
There are two technical culprits here:
1. On-line storage. There's no reason to keep all of everyone's mail on-line on the server (a la IMAP or proprietary MS Exchange) instead of offline on their PC's (a la POP, most often seen with Eudora for non-techies). With offline storage, the servers don't clog, and you can keep as much mail as you like.
The biggest rap agains off-line storage is that you can't control what people do with their mail or how they store it. My old job had a neat solution for this: Eudora downloaded your mail, but stored it on a file server. Each employee had 100 GB or something very large. It worked great; the SMTP/POP servers were never full, and everyone could keep their email.
2. Ridiculous stupid bullshit HTML rich-text mail crap. Can you tell I have a bias here? Aside from being annoying, HTML mail can take up to ten times the size of plain old text. Some of the HTML generated by common email programs is just terrible; filled with repeating tags for every line, and just wasting an incredible amount of space for absolutely zero benefit. (Outlook is bad, but there are others that are just as bad.)
There's no excuse for not fixing these problems. Someday someone's going to tell a court they had to delete mail for these reasons, and someone else is going to explain exactly why they're wrong. Until then, people who want to delete mail for legal reasons will hide behind false technical reasons.
This has to be the stupidest approach to the problem. Their networks are too slow, so instead, they're going to have each employee go through their old email and save individually important messages to their local hard disk? Not only are they going to tie up employees with this manual effort, they're also going to lose key documents and a key service - the ability to centrally search and reply to requests for information. In the future, each department will have to search their local hard drives for this information.
They've taken a simple problem of old or improperly speced equipment and turned it into a manual labor solution instead. That's an insane waste of time and salary. They should just upgrade their network and storage. If I can build a 4 terabyte RAIDed PC for a few thousand dollars, they can centralize their mailserver and back it up for say a hundred thousand, even with extra redundancy and inefficiencies and admin costs.
By contrast, forcing every current employee to perform a task that would eat up weeks of time per employee per year, in a city of Baltimore's size, will cost tens of millions of dollars.
Dumb, dumb, dumb.
--Pat / zippy@cs.brandeis.edu
If they haven't read it in 90 days, they've already ignored it.
You are in a maze of twisted little posts, all alike.
Backup all e-mails from the last 4½ years into permanent storage, and then from there, get organized. Put spam filters on, force people to sort any important mail or else it gets deleted after, say, two weeks. People always seem to want to "start from scratch". without looking at the situation rationally. Five years of documents, gone overnight. How can anyone not be at least outraged by that?
I'm posting anonymously because this may risk my relationship with my employer.
We see old e-mails as a resource to be harnessed and turned into profit. Thanks to old e-mails we can ensure that no employee leaves with a spotless record since everyone always e-mails something incriminating sooner or later from the company e-mail address.
We also find that the e-mails are great for data repositories; we fill all of our databases with text and when our clients come in, we tell them that those data warehouses contain terabytes of information.
The spam problem is unlikely to go away until people start treating it like the attack on the Internet that it is.
I've noticed an annoying trend lately that e-mail sent to businesses is frequently getting just ignored. Certainly it seems much more frequent this year than in the past. I've wondered if this is simply because so many e-mail boxes are getting filled up as fast as the spammers can send.
I'd suspect that the city of Baltimore wouldn't be having any problems if spam weren't such a problem. If the number of messages they had to deal with dropped by 5 to 20 times (depending on which estimates of current spam levels you believe), they could probably just leave the mail where it is.
This is all something I've been struggling with, being a small business owner doing business on the net. My company of 5 people gets between 4,000 and 20,000 borderline spams per day. By borderline, I mean that we throw away obvious viruses and things which score above a certain score in SpamAssasin (I think it's 9). So, that doesn't count the super spammy messages.
If it weren't for our fairly strict and complicated spam blocker setup, and a very powerful machine, we couldn't get the few hundred messages per day that are of interest to us. Spam is killing e-mail. I'm not sure why more people aren't treating it as an attack, but it's really hard to get anyone's interest to take some action. Canceling accounts doesn't even begin to solve the problem.
In the mean time, the City of Baltimore is suffering...
Sean
to dump it off to tape and then just store the tapes instead of just deleting it. Though they are probably running an Exchange server so offloading data stores wouldn't be the easiest thing to do. If they were using something with a simple mbox store, they could easily just parse it through a date filter and dump the older than 90 day stuff to tape. At least then it could be retrieved at a later date.
Oh, wait, let me guess, they aren't using tape backups...
Don't Ask Questions. I don't know the answers and even if I did I wouldn't tell you.
OMFG, we nearly had a lynch mob attack us when we began deleting mail older than *two years* -- it eventually took the intervention of the CFO and a faked mail system "crash" to make 2-year max retention work, and even then there are people still pissed about it, or who claim that "the client" requires them to retain all correspondence (nope, sorry, we checked the contract).
.PST files, which often max out at 2 gig and can get corrupted way too easily, not to mention being fdisked into eternity by clueless helpdesk people.
90 days seems both unrealistic to implement and way too much reliance on
We don't want someone to be able to request something from backups that the user thinks is gone.
This way it's up to the user to decide if they want their data archived. And the onus is on the user to comply with however long the data is supposed to be kept before being destroyed.
This highlights a fundamental problem with email -- many people pass documents as attachments, or in the body of the email, instead of using email as a sort of metadata describing their works in progress. Documents shouldn't be passed around in email; they should be stored on a network share, where proper controls for mutual exclusion and such can be employed.
'He who has to break a thing to find out what it is, has left the path of wisdom.' -- Gandalf to Saruman
I am not sure if they can use email as official communication? There would be problems with repudiation ("we never received it"), privacy ("someone intercepted it who was not supposed to") and authentication ("it wasn't me who sent it, it was my dog"). Can they use an email in the court then? What would have to be done is to have all the messages signed and encrypted with a public key, and perhaps have some way for the sender to get a receipt back when reciever reads the message.
>A better idea would be to write a script to go through each user's mailboxes every month, export any old emails to text, store the files on a server that uses a journaling filesystem, index the emails, and compress them.
No file system will save you from multiple HDD failures; they should save old (>12 months) data to DVD burners and/or tapes or cheap SATA storage. One can buy 1TB of external SATA space for couple thousand dollars.
>One or two XServe G5s could do the trick quite well.
What do XServe boxes have to do with generic application like email? Besides, they're more expensive than comparable Intel+Linux servers (especially considering the fact that CPU perormance is unimportant for most mail servers).
Though I don't work in the auditors office in my state, here is what they implemented. Any document (digital or not) over 30 days must be made public. Solution, any e-mail over 30 days is deleted. It allows them to not worry about keeping all e-mail till the end-of-time and not worry about making e-mail public. Great solution in that scenario.
Working at a law firm we have to keep everything for 7 years. We have a system in place that takes all mail over 90 days old pulls it out of exchange and move it to the SAN. As a plus it puts a link back into the information store to make it look like the message is still there. User wants a Old message he can still get it himself w/o a IT person having to do dig up a tame, restore the file and the e-mailing it to him (Thus creating MORE mail). The messages are still searchable and it makes retrival when needed a snap.
Mind you, we are only a 700 user shop. But nothing gets deleted. If it gets buy the spam filter it gets saved.
People find it strange that I don't know how to juggle or tap dance.
As far as I've been able to figure out, this arose from a lawsuit against the county where an e-mail retrived from two years previous proved a county commissioner to be taking bribes in a zoning issue.
Rather than fix the corruption, just ensure that it's covered up more efficiently. Gotta love local governments.
No boom today. Boom tomorrow. There's always a boom tomorrow. - Cmdr. Susan Ivanova
Once an actual human person has read and acted on the mail, they should be able to mark it "official business" and/or move the email into an "official business" folder which does get kept as required.
Better procedures and training goes a long way here. These same folks have no problems with snail mail.
I don't know what business you work in, but if they haven't read it in 3 days, they've lost my business.
Just dump the old email to DVD-R and archive it somewhere. If someone wants to subpoena it, burn off copies and wish 'em luck. Even if the city is getting a million pieces of spam a day, at 5kb each after data compression, that's just one DVD-R per day at a buck or so each, peanuts compared to what the city already must spend xeroxing memos for records retention purposes.
A better option would be to archive old messages rather than remove them entirely. From the article it sounds like they are keeping ALL messages active all the time. For example:
"They say the system is so overburdened that creating a daily backup has become impossible; there is so much data that it takes more than 24 hours to copy it."
So, it seems like the solution would be to periodically lop off old messages to offline storage (tape, spare drives, whatever). In the event of a lawsuit the old messages could be reasonably recovered and the cost for such a system would be extremely minimal.
Unlike a legal office where communications are governed by extensive regulation, governments are really only required to keep records of official documents and decisions. The myriad of e-mails leading up to a decision are not generally protected under such an act, nor are snail mail or phone conversations. In fact, the whole idea of there being a digital trail to follow for governmental decision making is really very new. Does it makes sense to change that practice? Do we really think our government officials should be so closely watched that EVERY e-mail/phone conversation/smoke signal should be recorded and exposed to public scrutiny? Talk about making an unattractive job even less inticing.
In responce to the posters question about all those subpoenas: welcome to the world of civil litigation, where the first one to destroy the evidence wins!
Only 120 characters... who can summarize their entire world understanding in 120 characters?!
I'll handle these in reverse order.
Word attachments are acceptable when they are just a means of moving files around, and not the entire content of the email. What is not acceptable is expecting me to load a large word processor just so you can use the company letterhead. In my experience the latter type is far more common. Besides the security implications (macro viruses, etc), I do not have a gui on the computer I read my email. Nor should I need one.
As for HTML email, I'm simply not going to render strange IMG tags. They could lead to goatse, or back to a spammer's site, and now they know my email is active. HTML email generally looks like it was designed by an 8 year old with downs syndrome anyway. Plain text is just more readable for nearly every email. Check out HTML email is STILL evil!!! for more.
Give me Classic Slashdot or give me death!
Any of these so called "important governement documents" shouldn't be stores in an email archive anyway. They should be on a network drive getting backed up.
My point is that a better solution is to put the email storage in the end user's hands. Set file size limits on their accounts and have them move all important mail off of their server mailbox and into a Microsoft PST file...aka Personal folder.
I work for Fortune 50 company, running in an exchange environment and this is the method we use for about 4000 corporate employees. They have 10 MB mailbox limitations that will not allow them to send any email when their account reaches 10 MB. We then shut the accounts off when they reach 50 MB and kick messages back to the sender.
Users who have important email setup Personal folders in Outlook and move messages from their inbox to their PST file. This file is stored locally on laptops (for travel purposes) and on the user's network home drive for desktop PCs.
We run standard incremental backups daily and full backups once a week. The only problem we have with this is that MS's PST files have a 1 gb limitation before they get corrupt so some of the legal and credit employees have three or four personal folders normally sorted by year. So you would have one file for 2004, one for 2003, etc etc. Works for us, has to work for the government.
http://jayceecorder.blogspot.com
We have to spend a lot of time telling people to **NOT** save to local drives. If it is important or confidential, or may be in the future, this should not be saved locally unless you want to loose it or explain to an enquiry why it was found on sale in a car boot sale after a break in. This is what a network is for.
The answer to the problem in the article is quotas. *nix has them, Novell has them and even Windows has them. Our email quota works as follows
Limit 1 - email user once per day marked high importance that they are getting close.
Limit 2 - disable sending and continue with (2k) warning message.
Limit 3 - disable receiving apart from one final message saying that it would all start working again when the user clears some space
When they can't send/receive, they get a dialogue box reminding them when they try and when they can't receive, the sender gets a messge.
This does make for support calls like...
"Why does my computer tell me that the email is full up and I can't send any more?"
"Because your email is full up. You have a message explaining this to you."
"X tried to send me an email and it bounced saying that my mailbox was full up. Why?"
"Because your mailbox is full up."
I'll see your Constitution and raise you a Queen.
The question comes -- should all of it be public?
They are public records. So, yes it should all be public.
Simple, no?
If you're a zombie and you know it, bite your friend!
ILM is the next big thing. Its the logical extension to the ever increasing SAN/NAS Server/Workstation exponentially-increasing-data problem (go google for pretenders to the law).
You can't oversee growing data storage without a parallel increase in administration costs. Instead, the idea is to build automatic archiving into your storage architecture.
In practice this means you build tiers of storage/archive methods. Tier 1 is a high tkt Shark SAN etc, Tier 2 is lower priced SATA RAID and Tier 3 is a DAS Tape Library. Build retention guidelines into the storage management playform (Tivoli etc). Older items are automatically moved to the Tier corresponding to that retention/access policy. Really old items "live" on Tape. Frequently accessed data lives on the high speed boxes near to the users/application. You snapshot updates to a DR replica offsite or burn periodic Tape sets etc. Its a good idea to team this with storage virtualization (virtual LUNS/ Metadata directory servers) and you can add/rotate/modify the storage tiers when necessary without any downtime.
From a user perspective, you click on the link and if applicable, get notified the item is being retrieved from media x (its mostly transparent). Worse case - access times are in the minutes.
Of course, all this comes with a high price. Enterprise Storage systems are not cheap. Recent legislated policy (Sarbanes Oxley etc) enforces the retention of some media (e.g. email). You cannot rely on end users to enforce data retention. This lets you mandate tiers of protection and is highly configurable to support per application monitoring.
Nothing is foolproof. Its still being finessed but if you can afford it - its truly a thing of beauty.
There comes a point where that, too, gets very expensive. At my company (large US healthcare provider, with governmental and private contracts both HMO and PPO), after saying 3, 5, and 7 years, our lawyers have told us we have to archive all email potentially forever that the end user doesn't specifically delete. They may do an end-run around the deletion and archive those, too, but I don't know. Anyway, our email system (Lotus Notes, which is an extreme HOG) eats somewhere between 100GB - 1TB/week. I was told it was well over 1TB, but I don't believe them. This is of course due to older Notes versions inability to store attachments in public directories and simply sending a copy to each and every recipient (and the stupidity of no size limits on internal email). There is a point to how many drives you can add to a SAN, and then you have to get a whole extra chassis, which is where the expensive part comes in. To keep buying new SAN units every 6 months or so, as well as the harddrives to put in them (plus the maintenance contracts, 24/7 support, etc) could easily add up to $1million/year or more. Which is definitely more costly than 10 average low-mid level administrator's salaries.
The 10 secretaries in question were only using 1 GB each per year. 10GB per year in total. If your company is as large as you imply, the amount of work hours involved in sorting though old emails will be larger than that. Each person (or their PA) would need to do their own. That's a lot of hours.
At one company that I worked for, they got the brilliant idea to delete all email older than 30 days. They also didn't want employees to make backups of their personal mailboxes. They intentionally wanted all traces of old email to disappear. While I'm sure that it made the lawyers happy, it caused a lot of grief for the people actually doing work for the customer. Many design decisions, bug reports and other important things were only documented in email messages. This is supposed to be the age of the paperless office, right? When you are involved in a multi-year project, you often need to refer to old messages. It also had the effect of making old policy memos disappear, whose existence had proved to be very inconvenient to management on several notable occasions.
Mea navis aericumbens anguillis abundat
When I heard my city were outsourcing their garbage collection services, I imagined office blocks of staff in India sifting through online hex editors looking for spare memory blocks to delete.
Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
Right. Our company originally tried to instate size limits when we went to Notes (only 3 years ago), but then the lawyers said we need to keep everything anyway (HIPAA requirements). So even with the exorbitant expense of the system, it is probably still cheaper to keep expanding every couple months rather than pay people to sit there and sort through their own email. Anything from an external party must be kept, and anything remotely regarding a customer must be kept as well. It's a huge pain, and they took the easy way out by archiving every single email. But neither option is very cost effective. There are four people that I know of in my department alone that have email boxes (extensively categorized with dozens or up to hundreds of folders) with up to 20GB each. It's crazy. But even without the ever looming threat of a lawsuit, they claim that they have been able to disprove what other people were badmouthing them about by being able to produce an email from that person stating the exact opposite a year or two previously. I've witnessed it once, and it is pretty funny watching somebody turn beet red in a room with 25 supervisor's and above.
Probably too late in the thread for this to catch much attention...
However, I work for a local Government office, close to Charlotte, NC. It is our stated policy to remove all e-mail older than two weeks except for e-mail that is crucial to job performance. This is less to save space then it is to keep the news media from finding dirt. We really don't care that our e-mail is public record. We really have nothing to hide. However, the local newspaper (in Charlotte) is constantly asking for _ALL_ of the County Manager's e-mail. They aren't looking for anything specific, they are just on a fishing trip, trying to see what trouble they can stir up. They rely on the Freedom of Information act to, hopefully, generate some news, instead of doing some real investigative work. *sigh*
Anonymity enabled for self-protection. Wouldn't want the powers that be see my e-mail...
I hate it when people associate taxpayers + government with customers + business. The two relationships are very different.
There are no laws I know of that tell me I have to pay Company X for products. If I don't want any products from Company X, I won't buy anything from them. I'm not going to be breaking any laws because of it. However, if I don't pay my taxes I'll get hounded to death with the possibly of being tossed in jail.
See the difference?
A Penny for my thoughts? Here's my two cents. I got ripped off!
I don't know what business you work in, but if they haven't read it in 3 days, they've lost my business.
Let me guess.... you're emigrating a lot, yes? Otherwise you might have to have "business" with the government. Good luck getting a reply in three days there.
Kjella
Live today, because you never know what tomorrow brings