Slashdot Mirror
Charter Accidentally Wipes 14K Email Accounts
dacut writes with the sad news that Charter Communications, which provides cable and Internet access to 2.6 million customers, accidentally and irretrievably wiped out 14,000 active email accounts while trying to clear out unused accounts. They're providing a $50 credit to each affected customer, which seems a paltry sum for anyone who was less than diligent about backing up their email — though those who relied on Charter's webmail interface had no easy way to accomplish backups. From the article: "There is no way to retrieve the messages, photos and other attachments that were erased from inboxes and archive folders across the country on Monday, said Anita Lamont, a spokeswoman for the suburban St. Louis-based company. 'We really are sincerely sorry for having had this happen and do apologize to all those folks who were affected by the error,' Lamont said Thursday when the company announced the gaffe."
You just know this must be related to the story IT: You Used Perl to Write WHAT?! from earlier this morning...
Trolling is a art,
Did they send an email to notify people of the $50 rebate? My inbox is empty...
I've always told users, that email is not a storage medium. It's a volatile one.
Yes, they should have had backups now days, but none the less, if you want it saved, don't leave it in your inbox.
I've had folks complain that the trash automatically was cleaned out every three days. WTF?
There are no loopholes. It's either legal or it's not.
I know these kinds of things aren't supposed to happen, but sometimes they do. The worst part for the company itself is not the backlash they receive...it's the fact that nothing they do and nothing they say will fix it.
It's one thing if you have angry customers over something you have control over. It's another thing entirely if your customers are angry at you AND there isn't a single solitary thing you can do. That said, I hope that they are more careful in the future...
Living With a Nerd
Unlike Charter (who probably uses something not all that different from an mbox file), Google has a global, highly redundant data store that is easier to insert information into than it is to delete from. Even when an email is "deleted" from the GMail interface, there's no guarantee that the data in GoogleFS is actually gone. Google themselves have stated that it may take months (or even years) before the data is purged from the system.
:-/
Which is part of the reason why I actually trust Google with my email. I wouldn't mind them providing a proper backup mechanism (no, POP3 isn't a worthwhile mechanism for me), but it simply isn't as necessary as some hosting providers.
That being said, this entire mess could have been avoided if someone took a tape backup before purging data from the system...
Javascript + Nintendo DSi = DSiCade
Crap! UNDELETE UNDELETE!!!!!
Charter Manager: You sure that these are the correct accounts to nuke.
Charter Employee: Yessiree ! 'Click'
Charter Employee: Oh shit.
Charter Manager: What ?
Charter Employee: 'Surfs over to Monster.com' Oh nothing. Nothing at all.
Guns are for wimps... Use a crossbow.. this way you can pin them to their chair when you go postal.
Back in the olden days when everyone POP'd their email and disk quotas on the mail server were in the 5-10 MB range, most ISPs didn't bother to back up email for very long because it was expensive and mostly pointless. These days, however, with everyone pushing huge disk quotas and webmail interfaces, the ISPs must be aware that most people will be keeping their email on the server for long periods of time. If this service were free, I might be able to excuse some shoddy backup practices, but in the case of an ISP your mail service is part of the overall service that you're paying for.
So, either Charter doesn't back up email very well, or their process to "clear out old accounts" involves actually deleting all of the backups of those accounts as well. I already addressed the issue with the former scenario, but if it's the latter, I'd have to say that's a pretty nasty practice too. Any time you clear out old and "unused" data, you have to assume that you're likely to accidentally hit some false positives, which is one of the reasons we have backups in the first place.
Somewhere deep in the bowels of a server room at 2:14 am...
clickety clickety (SIGH) clickety (beep)
clickety clickety (beep)
clickety (beep)
clickety clickety (beep)
click- OHHHH SH**! F***!
stuff |
Gmail also offers IMAP now... Just sync up once in a while and you have a complete snapshot of your mailbox - folders and everything.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
...I actually trust Google with my email. I wouldn't mind them providing a proper backup mechanism...
There's always Google paper...
For the size of enterprise that Charter is, this is a non-trivial requirement. Having architected, administered, implemented, repaired, and re-engineered backup solutions for many enterprise environments (some in the petabytes-range each), I can tell you that "very little effort" doesn't come anywhere close. I've also worked on the implementation of a mail environment (very much like what Charter has) for a cable modem ISP, so I'm very familiar with the kinds of challenges these environments face.
The backup architecture required to efficiently and safely protect this kind of environment would cost easily several hundred thousand dollars and several full time employees to manage.
Before anyone jumps in with "just buy a bunch of cheap IDE hard drives and rsync, tar, etc...", please don't forget that we're talking about a major server farm, probably in several locations, consisting of likely hundreds (if not thousands) of servers and mail stores.
More than likely, Charter made the business decision that (as other posters have pointed out) email is a volatile storage medium and their internal checks and balances (RAID, etc) were sufficient for protecting against loss. Obviously, they made a mistake and miscalculation. At the end of the day, however, I suspect they'll implement more checks and balances to protect against human error, but I'd be really surprised if IBM/SUN/etc got a big order for a tape library/upgrade. I just can't imagine a company like Charter spending the money (hardware, consumables, people) to back up "Forward this to 10 people in the next 10 minutes and Bill Gates will give you a hand job" messages.
-- "Other than that, how was the play Mrs. Lincoln?"
Back in the dot-com days I worked for a local ISP, and established my online identity over the years. The company died, and I lost my long-established email address. Lesson learned - I obtained my own domain name and webhosting, just on shared servers, mind you, and now I have a portable identity that I have control over. Webhost screwing up? I've had it happen a few times now. I just point the domain elsewhere. I have unlimited POP, IMAP, and even webmail. Multiple spam controls that I can fiddle with. And I don't have to worry about Google, Yahoo, etc fiddling with anything either.
It isn't hard, either. My 63-yo father is now doing the same thing, as he switched ISPs for the first time now that he can get DSL out on the farm, and he isn't the most technical guy.
I know many people are saying that Charter should have backed up the email, but I used to work for AOL, and I know they don't back up any of the email, other than having redundant servers in multiple locations. By not sending your email to tape or other media, they can't be hauled into court and forced to give it up. Once it's gone, it's gone.
That said, it's standard practice when deleting an account to mark the data as deleted, so that it looks like it's gone to the user, but it's actually pending deletion later. Then, when someone complains or pays their bill, you can restore what was "deleted." After a predetermined amount of time, if you don't complain, a cleanup script deletes it permenently.
All ideas^H^H^H^H^Hprocesses in this post are Patent Pending. (as well as the process of patenting all postings)
Sounds like you need better software. I use offlineimap and it syncs a couple of family Gmail accounts in seconds.
As opposed to, say, the $700,000k they just paid out.
--Rob
Towards the Singularity.
Yes, I'm responding to my own post...
Just because I can, I did a couple of bar-napkin type calculations to see what it would take to protect this environment.
I have no idea what Charter's cable modem subscriber base is, so I took some wild ass guesses. According to Charter's website, they have around 5.7 million customers. Assuming that a 12.5% of them are cable modem subscribers, and each of those accounts has an average of two mailboxes, that gives us just under 1.5 million mailboxes to protect. Further assuming an average mailbox size of 50MB (not unreasonable, given the similar environments I've seen), that's somewhere in the neighborhood of 71 terabytes of data - just for email. That's not counting the supporting infrastructure (authentication, transport, etc).
So to protect 71 terabytes of data, we need somewhere to put that. Tape is most likely. Let's assume LTO3 (probably the most commonly deployed tape technology today in the open systems world), so we've got a raw capacity of 400GB per tape (don't believe the compression specs, I rarely see more than 600GB in the wild). Assuming daily backups kept one week, and weekly backups kept for a month, we'd need about 1780 tapes for the month's rotation. At $40/tape, that's $71,200 in media. Figure 10% per year to replace failed media, and we've got a first year consumable cost of $78,320.
Now, to get the data onto our ~$80k worth of tapes.
Let's figure a 12 hour backup window. (We'll assume that this backup infrastructure will be used to protect some other assets in the other 12 hours) To move 71 terabytes of data in 12 hours, we'll need about 28 LTO3 tape drives (I'll spare you the calculations used to get there - but suffice it to say that I included reasonable overhead and observed real-world performance). At $3k a pop (for quality, supportable, maintainable drives), that's $85k in drives. A tape library to contain said drives will be somewhere in the $100k-$150k range depending on options (redundant robotics, etc). The SAN infrastructure required to connect these drives should be in the $30-$40k range.
So just tape hardware, lets call it $250k.
Additionally, we need backup servers to handle all this data. No, cheap 1U Intel boxes aren't going to cut it. You're going to need some serious iron to drive 28 LTO3 tape drives at full capacity. Off the top of my head, I'd say you're looking at 2-3 mid-sized Unix servers (IBM System p, etc) loaded down with 4GB fibre adapters. Easily another $150k.
Right now, we're just under $500k, and we haven't even started talking about software licensing (Tivoli Storage Manager, Veritas Netbackup, ComVault, etc), infrastructure for the systems being backed up (dedicated Ethernet, or depending on volume, dedicated fibre), miscellaneous supporting infrastructure (power, UPS, air conditioning, etc), and so on.
Once you've got all that, who's going to manage that? Probably a senior backup administrator/architect (90-120k yearly), a mid-range systems administrator (60-90k yearly), and one or two operators (media handling, etc, 30-50k yearly). So that's $250k or so in salaries to manage this beast, figure a benefit load of 60%, and we're at $400k to employ these people.
Initial hardware investment : $480k
Yearly consumables : $8k
Yearly media storage : $60k (no idea - completely made this one up - anyone with knowledge of Iron Mountain, etc, want to comment?)
Yearly salaries to manage: $400k.
Completely ignoring data center costs (AC, power, etc) and software, let's call it an up front investment of $1mil, and a yearly ongoing cost of $500k to support.
14,000 customers at $50 service credit (not real cash) = $700,000
So as long as they only do this once every two years or so, they're in the black.
Sorry, but that's how businesses think. (And yes, there's the cost to customer satisfaction, lost customers, etc, but growth will easily outstrip those losses)
Damn, posting on slashdot feels way too much like real work.
-- "Other than that, how was the play Mrs. Lincoln?"
Your statement about people complaining about the trash(deleted email) being emptied brought back a fond memory of mine. I was early in my IT Admin career (was a programmer for over 10 prior to that), I was working for a major insurance company administrating their CC:Mail network. We had issues of people never emptying their "trash" folder and it was taking a long time to do anything for everyone on the system. We sent out a memo to the entire company telling them that in a week we would start emptying the trash folder nightly around midnight before the backups and other maintenance begins. very few people bothered to pay attention to this memo.
Here is where it get...funny. A high-muckety-muck (eg: pointy haired VP) called the help desk screaming for his trash folder to be restored as it was emptied without his permission and had important files in there. He wanted to see the guilty parties in his office post-haste. I was part of the team, so I had to go to his office. while waiting outside his door for him to let our team in, I grabbed a trashcan and some vertical file folders and paper from his secretary. (can you see where I'm going yet?) We were let in, and he proceeded to rip us up one side and down the other. Our manager brought a copy of the memo, which he promptly threw away while continuing to yell.
I asked to speak (everyone else was quietly taking the heat) and proceeded to put the trash can on the desk, put the vertical file folders in the trashcan and put paper in each folder. While I was explaining this analogy to him, I asked if every day his office trashcan was empty when he came into the office. He said yes. I basically gave him the analogy that the trash folder in cc:mail was the same as his office trashcan - whatever went in during the day was retrievable, but at midnight, the office cleaning crew would come in and empty his trashcan and we also would be emptying his electronic trash folder the same way. He blinked, and understood what we were doing, pulled the memo out of the trash and reread it. we were dismissed with no apology back to our offices.
Note that I was a contractor - I couldn't be fired...just sent away back to my firm to go on to the next assignment. I ended up being there for 2.5 years. I still laugh at that situation even today.
You have got to be kidding me. I was going to moderate this as idiocy, but I decided to elaborate with a response instead.
One might be able to reach a person at Charter, but a helpful person? Not on your life. You speak about behemoth corporations, and Charter embodies the worst of corporate bureaucracy. They are total idiots, the left hand doesn't talk to the right hand, and their prices are unreasonable. And yes, I dumped them as soon as I could so I don't have to deal with them any more. But not once did I deal with a helpful person.
And deleting 14,000 email accounts just shows the heights of stupidity this company has achieved.
I'm Peggy.
A year or two ago, our internet access stopped working. Over the course of three weeks, I spent 12 hours on the phone with Charter support. During those calls I was told: they had deleted my MAC address from the database, that they had no record of me ever being a Charter customer (despite the cable boxes in my living room), that they had no record of me having internet access, and that I was stealing cable.
Finally, I got fed up, and called AT&T, got DSL and Dish. Once everything was working, I packed up all of Charter's equipment, brought it down to a local office, and told them to shut me off. This was a Saturday.
On Monday, my Dish stops working. I call up the Dish people; they come out around 10 days later to tell me that Charter had disconnected my satellite and had put a terminator on the line.
Fuck Charter. Fuck them in the neck with a donkey cock. They are the worst of the worst of companies (believe it or not, this is the short version - the long version is worse). If they actually had to compete, instead of having a monopoly, they would have been out of business years ago - and I would have bought some of their assets so I could set them on fire.
Fuck Charter.