Slashdot Mirror
Charter Accidentally Wipes 14K Email Accounts
dacut writes with the sad news that Charter Communications, which provides cable and Internet access to 2.6 million customers, accidentally and irretrievably wiped out 14,000 active email accounts while trying to clear out unused accounts. They're providing a $50 credit to each affected customer, which seems a paltry sum for anyone who was less than diligent about backing up their email — though those who relied on Charter's webmail interface had no easy way to accomplish backups. From the article: "There is no way to retrieve the messages, photos and other attachments that were erased from inboxes and archive folders across the country on Monday, said Anita Lamont, a spokeswoman for the suburban St. Louis-based company. 'We really are sincerely sorry for having had this happen and do apologize to all those folks who were affected by the error,' Lamont said Thursday when the company announced the gaffe."
Gmail also offers IMAP now... Just sync up once in a while and you have a complete snapshot of your mailbox - folders and everything.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
For the size of enterprise that Charter is, this is a non-trivial requirement. Having architected, administered, implemented, repaired, and re-engineered backup solutions for many enterprise environments (some in the petabytes-range each), I can tell you that "very little effort" doesn't come anywhere close. I've also worked on the implementation of a mail environment (very much like what Charter has) for a cable modem ISP, so I'm very familiar with the kinds of challenges these environments face.
The backup architecture required to efficiently and safely protect this kind of environment would cost easily several hundred thousand dollars and several full time employees to manage.
Before anyone jumps in with "just buy a bunch of cheap IDE hard drives and rsync, tar, etc...", please don't forget that we're talking about a major server farm, probably in several locations, consisting of likely hundreds (if not thousands) of servers and mail stores.
More than likely, Charter made the business decision that (as other posters have pointed out) email is a volatile storage medium and their internal checks and balances (RAID, etc) were sufficient for protecting against loss. Obviously, they made a mistake and miscalculation. At the end of the day, however, I suspect they'll implement more checks and balances to protect against human error, but I'd be really surprised if IBM/SUN/etc got a big order for a tape library/upgrade. I just can't imagine a company like Charter spending the money (hardware, consumables, people) to back up "Forward this to 10 people in the next 10 minutes and Bill Gates will give you a hand job" messages.
-- "Other than that, how was the play Mrs. Lincoln?"
Sounds like you need better software. I use offlineimap and it syncs a couple of family Gmail accounts in seconds.
As opposed to, say, the $700,000k they just paid out.
--Rob
Towards the Singularity.
Yes, I'm responding to my own post...
Just because I can, I did a couple of bar-napkin type calculations to see what it would take to protect this environment.
I have no idea what Charter's cable modem subscriber base is, so I took some wild ass guesses. According to Charter's website, they have around 5.7 million customers. Assuming that a 12.5% of them are cable modem subscribers, and each of those accounts has an average of two mailboxes, that gives us just under 1.5 million mailboxes to protect. Further assuming an average mailbox size of 50MB (not unreasonable, given the similar environments I've seen), that's somewhere in the neighborhood of 71 terabytes of data - just for email. That's not counting the supporting infrastructure (authentication, transport, etc).
So to protect 71 terabytes of data, we need somewhere to put that. Tape is most likely. Let's assume LTO3 (probably the most commonly deployed tape technology today in the open systems world), so we've got a raw capacity of 400GB per tape (don't believe the compression specs, I rarely see more than 600GB in the wild). Assuming daily backups kept one week, and weekly backups kept for a month, we'd need about 1780 tapes for the month's rotation. At $40/tape, that's $71,200 in media. Figure 10% per year to replace failed media, and we've got a first year consumable cost of $78,320.
Now, to get the data onto our ~$80k worth of tapes.
Let's figure a 12 hour backup window. (We'll assume that this backup infrastructure will be used to protect some other assets in the other 12 hours) To move 71 terabytes of data in 12 hours, we'll need about 28 LTO3 tape drives (I'll spare you the calculations used to get there - but suffice it to say that I included reasonable overhead and observed real-world performance). At $3k a pop (for quality, supportable, maintainable drives), that's $85k in drives. A tape library to contain said drives will be somewhere in the $100k-$150k range depending on options (redundant robotics, etc). The SAN infrastructure required to connect these drives should be in the $30-$40k range.
So just tape hardware, lets call it $250k.
Additionally, we need backup servers to handle all this data. No, cheap 1U Intel boxes aren't going to cut it. You're going to need some serious iron to drive 28 LTO3 tape drives at full capacity. Off the top of my head, I'd say you're looking at 2-3 mid-sized Unix servers (IBM System p, etc) loaded down with 4GB fibre adapters. Easily another $150k.
Right now, we're just under $500k, and we haven't even started talking about software licensing (Tivoli Storage Manager, Veritas Netbackup, ComVault, etc), infrastructure for the systems being backed up (dedicated Ethernet, or depending on volume, dedicated fibre), miscellaneous supporting infrastructure (power, UPS, air conditioning, etc), and so on.
Once you've got all that, who's going to manage that? Probably a senior backup administrator/architect (90-120k yearly), a mid-range systems administrator (60-90k yearly), and one or two operators (media handling, etc, 30-50k yearly). So that's $250k or so in salaries to manage this beast, figure a benefit load of 60%, and we're at $400k to employ these people.
Initial hardware investment : $480k
Yearly consumables : $8k
Yearly media storage : $60k (no idea - completely made this one up - anyone with knowledge of Iron Mountain, etc, want to comment?)
Yearly salaries to manage: $400k.
Completely ignoring data center costs (AC, power, etc) and software, let's call it an up front investment of $1mil, and a yearly ongoing cost of $500k to support.
14,000 customers at $50 service credit (not real cash) = $700,000
So as long as they only do this once every two years or so, they're in the black.
Sorry, but that's how businesses think. (And yes, there's the cost to customer satisfaction, lost customers, etc, but growth will easily outstrip those losses)
Damn, posting on slashdot feels way too much like real work.
-- "Other than that, how was the play Mrs. Lincoln?"
I'm not exactly sure what you mean by "entitled to expect."
Gmail is a free (to the user) service. The user might expect it to be reliable, but they are entitled to nothing. If it all goes away, there's no recourse. Also, in light of all the free competing e-mail services these days, most ISP's offer e-mail as a courtesy and will not guarantee it's integrity. It's probabbly carefully worded in every AUP. To practice a business where you are legally responsible for terabytes of customer data is scary can of worms... let Citibank deal with that.
I work in the ISP business and can tell you that the overhead for e-mail is greater then any other service provided. Every other piece of vital data here is peanuts compared to the size of our customer e-mail storage arrays. We back it up as often as we can. It's an absolutely enormous amount of data that changes every second. We do the best we can but, Hell no, I don't want to be liable for it. I would hope you don't expect me to. Even though I know it's being rsyncd off site twice a day I still advise customers every chance I get. Download your messages and archive the important ones periodically.
As the old saying goes, the only person you can rely on is yourself.