Slashdot Mirror
Charter Accidentally Wipes 14K Email Accounts
dacut writes with the sad news that Charter Communications, which provides cable and Internet access to 2.6 million customers, accidentally and irretrievably wiped out 14,000 active email accounts while trying to clear out unused accounts. They're providing a $50 credit to each affected customer, which seems a paltry sum for anyone who was less than diligent about backing up their email — though those who relied on Charter's webmail interface had no easy way to accomplish backups. From the article: "There is no way to retrieve the messages, photos and other attachments that were erased from inboxes and archive folders across the country on Monday, said Anita Lamont, a spokeswoman for the suburban St. Louis-based company. 'We really are sincerely sorry for having had this happen and do apologize to all those folks who were affected by the error,' Lamont said Thursday when the company announced the gaffe."
You just know this must be related to the story IT: You Used Perl to Write WHAT?! from earlier this morning...
Trolling is a art,
I am one of those people who uses Gmail as a backup betting it's more reliable than my hard drive.
Did they send an email to notify people of the $50 rebate? My inbox is empty...
I've always told users, that email is not a storage medium. It's a volatile one.
Yes, they should have had backups now days, but none the less, if you want it saved, don't leave it in your inbox.
I've had folks complain that the trash automatically was cleaned out every three days. WTF?
There are no loopholes. It's either legal or it's not.
I know these kinds of things aren't supposed to happen, but sometimes they do. The worst part for the company itself is not the backlash they receive...it's the fact that nothing they do and nothing they say will fix it.
It's one thing if you have angry customers over something you have control over. It's another thing entirely if your customers are angry at you AND there isn't a single solitary thing you can do. That said, I hope that they are more careful in the future...
Living With a Nerd
They didn't make backups beforehand? What kinda incompetent sysadmins do they have over there anyway?
In Xanadu did Kubla Khan
A stately pleasure dome decree
Crap! UNDELETE UNDELETE!!!!!
Charter Manager: You sure that these are the correct accounts to nuke.
Charter Employee: Yessiree ! 'Click'
Charter Employee: Oh shit.
Charter Manager: What ?
Charter Employee: 'Surfs over to Monster.com' Oh nothing. Nothing at all.
Guns are for wimps... Use a crossbow.. this way you can pin them to their chair when you go postal.
Back in the olden days when everyone POP'd their email and disk quotas on the mail server were in the 5-10 MB range, most ISPs didn't bother to back up email for very long because it was expensive and mostly pointless. These days, however, with everyone pushing huge disk quotas and webmail interfaces, the ISPs must be aware that most people will be keeping their email on the server for long periods of time. If this service were free, I might be able to excuse some shoddy backup practices, but in the case of an ISP your mail service is part of the overall service that you're paying for.
So, either Charter doesn't back up email very well, or their process to "clear out old accounts" involves actually deleting all of the backups of those accounts as well. I already addressed the issue with the former scenario, but if it's the latter, I'd have to say that's a pretty nasty practice too. Any time you clear out old and "unused" data, you have to assume that you're likely to accidentally hit some false positives, which is one of the reasons we have backups in the first place.
Somewhere deep in the bowels of a server room at 2:14 am...
clickety clickety (SIGH) clickety (beep)
clickety clickety (beep)
clickety (beep)
clickety clickety (beep)
click- OHHHH SH**! F***!
stuff |
This, once again, highlights the trouble of using "remotely hosted applications" - you are not in control of your data.
/I/ lose it it's on /my/ head.
I always POP my email down to my own local computer.
At least if
A work that expires before its copyright never enters the public domain and thus enjoys eternal copyright protection.
For the size of enterprise that Charter is, this is a non-trivial requirement. Having architected, administered, implemented, repaired, and re-engineered backup solutions for many enterprise environments (some in the petabytes-range each), I can tell you that "very little effort" doesn't come anywhere close. I've also worked on the implementation of a mail environment (very much like what Charter has) for a cable modem ISP, so I'm very familiar with the kinds of challenges these environments face.
The backup architecture required to efficiently and safely protect this kind of environment would cost easily several hundred thousand dollars and several full time employees to manage.
Before anyone jumps in with "just buy a bunch of cheap IDE hard drives and rsync, tar, etc...", please don't forget that we're talking about a major server farm, probably in several locations, consisting of likely hundreds (if not thousands) of servers and mail stores.
More than likely, Charter made the business decision that (as other posters have pointed out) email is a volatile storage medium and their internal checks and balances (RAID, etc) were sufficient for protecting against loss. Obviously, they made a mistake and miscalculation. At the end of the day, however, I suspect they'll implement more checks and balances to protect against human error, but I'd be really surprised if IBM/SUN/etc got a big order for a tape library/upgrade. I just can't imagine a company like Charter spending the money (hardware, consumables, people) to back up "Forward this to 10 people in the next 10 minutes and Bill Gates will give you a hand job" messages.
-- "Other than that, how was the play Mrs. Lincoln?"
Back in the dot-com days I worked for a local ISP, and established my online identity over the years. The company died, and I lost my long-established email address. Lesson learned - I obtained my own domain name and webhosting, just on shared servers, mind you, and now I have a portable identity that I have control over. Webhost screwing up? I've had it happen a few times now. I just point the domain elsewhere. I have unlimited POP, IMAP, and even webmail. Multiple spam controls that I can fiddle with. And I don't have to worry about Google, Yahoo, etc fiddling with anything either.
It isn't hard, either. My 63-yo father is now doing the same thing, as he switched ISPs for the first time now that he can get DSL out on the farm, and he isn't the most technical guy.
I know many people are saying that Charter should have backed up the email, but I used to work for AOL, and I know they don't back up any of the email, other than having redundant servers in multiple locations. By not sending your email to tape or other media, they can't be hauled into court and forced to give it up. Once it's gone, it's gone.
That said, it's standard practice when deleting an account to mark the data as deleted, so that it looks like it's gone to the user, but it's actually pending deletion later. Then, when someone complains or pays their bill, you can restore what was "deleted." After a predetermined amount of time, if you don't complain, a cleanup script deletes it permenently.
All ideas^H^H^H^H^Hprocesses in this post are Patent Pending. (as well as the process of patenting all postings)
People still leave messages on server? Worse, they rely on it still being there? Man, I must be getting old, I thought we were past this, but apparently web mail has brought back a few of the 'net's child diseases.
/ Per
..should have had backups... should have been more careful... of course.
But what's done is done and props to them for a bullshit-free apology.
Most people are prepared to cut you some slack when you screw up as long as you admit your mistake.
- recognise what it was that you did wrong
- claim responsibility for your actions
- apologise
- state clearly what you learned and what actions you will take to prevent a recurrence
Or you could take to legal advice / bush administration route
- flatly refuse to acknowledge that anything bad actually happened
- talk about how 'the other guys' screw up all the time
- start an internal investigation and refuse to comment on the issue while it is under investigation
- eventually admit that 'mistakes were made' but no, you can't think of any specific examples right now and it was all someone else's fault and you there's no way you could have known it would happen.
As opposed to, say, the $700,000k they just paid out.
--Rob
Towards the Singularity.
Note the use of the passive voice, which is commonly done to avoid taking responsibility. It seems like even when they're trying to apologize, spin-doctors can't turn off their instinct of avoiding responsibility for mistakes.
the customers could always write to the NSA and ask for their backups.
I've never had to back up email for 50 million people, but I've been responsible for a system with 50 thousand people. We didn't backup our email, didn't even come close to having the resources to do so, and it clearly stated in the SLA that we didn't do backups, and if your email got lost, tough shit. Our customers got what they paid for, since the email was free.
Mostly likely their asses were covered by their service agreement. I am pretty sure that Yahoo's policy for lost email is "tough shit" as well.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
The statement is still valid: your email server is not and cannot be a reliable long term archive. You are foolish if you leave VALUABLE information on it that you don't ever want to lose or have compromised:
1. Most access is plain text and subject to snooping/hijacking (passwd/userid/content)
2. Email is the most abused internet protocol (my opinion) by zombots, spammers, and virus/trojan propagators. ISPs do a lot to counter spam and threatening content but sometimes they get hosed. Or your home machine gets compromised and the ISP will do things to clean up.
3. Grooming accounts for stale accounts, unauthorized accounts, and stale/large data is a reality on most messaging systems. "Ooops"s happen. Usually stated as "sxxt happens"
Whatever your feelings of outrage are, common sense says put your important stuff somewhere close at hand and under your own control.
My 2 cents FWIW
i can kinda sympathize with charter (shudder, omg did i actually type that?!). ok, i can feel for their admins.
for eight months, i worked for a small-town isp with dsl and dialup customers. we had old equipment and no budget for upgrades. we had an autoloader that would occasionally snap tapes, old drive arrays that would fail with no replacement parts on hand ("whuddya mean, we got harddrives right there" "those are ide, i need scsi3"). backupexec would report completed jobs but find no restorable data. our dhcp scope was too small to serve all our customers at once (meaning I would hunt down inactive leases and free them for people trying to connect at that moment).
i did get a new tapedrive after six months of empty promises from my managers (and two catastrophic domain controller failures). i left them a year ago, and they still have my job posted on their site. (don't bother asking for the link, you do not want to work there.)
I worked for Charter back in 03 and 04. I noticed a trend before I decided to stop working there. They are self serving morons. They even wanted their tech support to try and upsell someone. "Yeah, um my internet is down" "Okay we can fix that, while you are waiting for the repair guy want to watch HBO for *such and such* extra a month?". I kindly told my bosses at Charter to shove it up their butt. They hired me to fix issues, and I was damn good at it, not to bring in more money. By the way, 3 months after I quit (and I saw hints of this too), they moved the tech team from St. Louis to Louisiana or some crap. Charter has been going down hill since the inet bubble burst back in 02 and they have been Enron'esq since then as well.
Yes, I'm responding to my own post...
Just because I can, I did a couple of bar-napkin type calculations to see what it would take to protect this environment.
I have no idea what Charter's cable modem subscriber base is, so I took some wild ass guesses. According to Charter's website, they have around 5.7 million customers. Assuming that a 12.5% of them are cable modem subscribers, and each of those accounts has an average of two mailboxes, that gives us just under 1.5 million mailboxes to protect. Further assuming an average mailbox size of 50MB (not unreasonable, given the similar environments I've seen), that's somewhere in the neighborhood of 71 terabytes of data - just for email. That's not counting the supporting infrastructure (authentication, transport, etc).
So to protect 71 terabytes of data, we need somewhere to put that. Tape is most likely. Let's assume LTO3 (probably the most commonly deployed tape technology today in the open systems world), so we've got a raw capacity of 400GB per tape (don't believe the compression specs, I rarely see more than 600GB in the wild). Assuming daily backups kept one week, and weekly backups kept for a month, we'd need about 1780 tapes for the month's rotation. At $40/tape, that's $71,200 in media. Figure 10% per year to replace failed media, and we've got a first year consumable cost of $78,320.
Now, to get the data onto our ~$80k worth of tapes.
Let's figure a 12 hour backup window. (We'll assume that this backup infrastructure will be used to protect some other assets in the other 12 hours) To move 71 terabytes of data in 12 hours, we'll need about 28 LTO3 tape drives (I'll spare you the calculations used to get there - but suffice it to say that I included reasonable overhead and observed real-world performance). At $3k a pop (for quality, supportable, maintainable drives), that's $85k in drives. A tape library to contain said drives will be somewhere in the $100k-$150k range depending on options (redundant robotics, etc). The SAN infrastructure required to connect these drives should be in the $30-$40k range.
So just tape hardware, lets call it $250k.
Additionally, we need backup servers to handle all this data. No, cheap 1U Intel boxes aren't going to cut it. You're going to need some serious iron to drive 28 LTO3 tape drives at full capacity. Off the top of my head, I'd say you're looking at 2-3 mid-sized Unix servers (IBM System p, etc) loaded down with 4GB fibre adapters. Easily another $150k.
Right now, we're just under $500k, and we haven't even started talking about software licensing (Tivoli Storage Manager, Veritas Netbackup, ComVault, etc), infrastructure for the systems being backed up (dedicated Ethernet, or depending on volume, dedicated fibre), miscellaneous supporting infrastructure (power, UPS, air conditioning, etc), and so on.
Once you've got all that, who's going to manage that? Probably a senior backup administrator/architect (90-120k yearly), a mid-range systems administrator (60-90k yearly), and one or two operators (media handling, etc, 30-50k yearly). So that's $250k or so in salaries to manage this beast, figure a benefit load of 60%, and we're at $400k to employ these people.
Initial hardware investment : $480k
Yearly consumables : $8k
Yearly media storage : $60k (no idea - completely made this one up - anyone with knowledge of Iron Mountain, etc, want to comment?)
Yearly salaries to manage: $400k.
Completely ignoring data center costs (AC, power, etc) and software, let's call it an up front investment of $1mil, and a yearly ongoing cost of $500k to support.
14,000 customers at $50 service credit (not real cash) = $700,000
So as long as they only do this once every two years or so, they're in the black.
Sorry, but that's how businesses think. (And yes, there's the cost to customer satisfaction, lost customers, etc, but growth will easily outstrip those losses)
Damn, posting on slashdot feels way too much like real work.
-- "Other than that, how was the play Mrs. Lincoln?"
Your statement about people complaining about the trash(deleted email) being emptied brought back a fond memory of mine. I was early in my IT Admin career (was a programmer for over 10 prior to that), I was working for a major insurance company administrating their CC:Mail network. We had issues of people never emptying their "trash" folder and it was taking a long time to do anything for everyone on the system. We sent out a memo to the entire company telling them that in a week we would start emptying the trash folder nightly around midnight before the backups and other maintenance begins. very few people bothered to pay attention to this memo.
Here is where it get...funny. A high-muckety-muck (eg: pointy haired VP) called the help desk screaming for his trash folder to be restored as it was emptied without his permission and had important files in there. He wanted to see the guilty parties in his office post-haste. I was part of the team, so I had to go to his office. while waiting outside his door for him to let our team in, I grabbed a trashcan and some vertical file folders and paper from his secretary. (can you see where I'm going yet?) We were let in, and he proceeded to rip us up one side and down the other. Our manager brought a copy of the memo, which he promptly threw away while continuing to yell.
I asked to speak (everyone else was quietly taking the heat) and proceeded to put the trash can on the desk, put the vertical file folders in the trashcan and put paper in each folder. While I was explaining this analogy to him, I asked if every day his office trashcan was empty when he came into the office. He said yes. I basically gave him the analogy that the trash folder in cc:mail was the same as his office trashcan - whatever went in during the day was retrievable, but at midnight, the office cleaning crew would come in and empty his trashcan and we also would be emptying his electronic trash folder the same way. He blinked, and understood what we were doing, pulled the memo out of the trash and reread it. we were dismissed with no apology back to our offices.
Note that I was a contractor - I couldn't be fired...just sent away back to my firm to go on to the next assignment. I ended up being there for 2.5 years. I still laugh at that situation even today.
Keep it simple, cupcake.
... assuming an average mailbox size of 50MB$240/TB / 1000GB/TB / 1000GB/MB * 50MB = 1.2 pennies
Of course, there are some additional costs involved, but it doesn't cost a heck of a lot per user to back up the email. The more customers, the more cost, but also the more revenue.I don't get upset or worried when I hear a sysadmin shouting and screaming. It's usually the result of some user doing something stupid but limited in scope.
I was sharing an office with the lab's sysadmin. One day, while I was happily programming away, I heard the quiet utterance from my office-mate: "Oh, shit." Shivers ran down my spine and I started to panic. I knew immediately that all hell was about to break loose.
Truly a frightening phrase to hear from your sysadmin.
My employer does a lot of state and local government systems installation and support contracts. All the email systems we install must have archive mechanisms that capture copies of all emails that are sent and received and that the end-users cannot access or modify. Emails sent or received by government employees are often considered public records, and typically the state has a set of regulatory statutes that govern how long each classification of email must be retained, some classes must be kept forever.
Ever wonder why so many state and local government email system run on Lotus Notes/Domino? It's because Lotus has a built-in feature called "mail journaling" that automatically does the archiving. In addition, Lotus has a standard clustering capability in its design that allows you to replicate the entire servers and their contents effortlessly across multiple machines. When I first had to learn Lotus, I thought it was going to kill me, but the more time I spend with it, the more I realize it is an incredibly powerful and capable messaging and application/database platform. But it has a super-weird learning curve to it that most people never can seem to "get it", hence the widespread fear and loathing towards Lotus Notes.
Just because I can, I did a couple of bar-napkin type calculations to see what it would take to protect this environment.
Dude, that wasn't a "bar-napkin type calculation"... You used the whole tablecloth!
Life is short; think quickly.
*THAT*'s why they won't let me back in that bar...
-- "Other than that, how was the play Mrs. Lincoln?"
NetApps are commodity. ZFS is free. Bigger storage iron is a competitive marketplace with thin margins. Who on earth is doing production storage without modern data management facilities?
ian