Slashdot Mirror
Charter Accidentally Wipes 14K Email Accounts
dacut writes with the sad news that Charter Communications, which provides cable and Internet access to 2.6 million customers, accidentally and irretrievably wiped out 14,000 active email accounts while trying to clear out unused accounts. They're providing a $50 credit to each affected customer, which seems a paltry sum for anyone who was less than diligent about backing up their email — though those who relied on Charter's webmail interface had no easy way to accomplish backups. From the article: "There is no way to retrieve the messages, photos and other attachments that were erased from inboxes and archive folders across the country on Monday, said Anita Lamont, a spokeswoman for the suburban St. Louis-based company. 'We really are sincerely sorry for having had this happen and do apologize to all those folks who were affected by the error,' Lamont said Thursday when the company announced the gaffe."
You just know this must be related to the story IT: You Used Perl to Write WHAT?! from earlier this morning...
Trolling is a art,
I am one of those people who uses Gmail as a backup betting it's more reliable than my hard drive.
That sounds like Charter Communications.
how much do you think the result could be?
Did they send an email to notify people of the $50 rebate? My inbox is empty...
It seems like it would be bound to happen if the company doesn't backup its client's data automatically.
while(1) attack(People.Sandy);
OK, one more time...
test it, test it, test it
and over, and over, and over
again and again and again
and back-up and back-up and back-up...
feel the burn!
Yet Socrates himself is particularly missed.
A lovely little thinker but a bugger when he's pissed.
I've always told users, that email is not a storage medium. It's a volatile one.
Yes, they should have had backups now days, but none the less, if you want it saved, don't leave it in your inbox.
I've had folks complain that the trash automatically was cleaned out every three days. WTF?
There are no loopholes. It's either legal or it's not.
I know these kinds of things aren't supposed to happen, but sometimes they do. The worst part for the company itself is not the backlash they receive...it's the fact that nothing they do and nothing they say will fix it.
It's one thing if you have angry customers over something you have control over. It's another thing entirely if your customers are angry at you AND there isn't a single solitary thing you can do. That said, I hope that they are more careful in the future...
Living With a Nerd
This is why I don't trust a foreign server for a sole electronic copy of a critical document. For the most important documents, I keep a multiple hard-copies, including one in my freezer. Don't tell anyone, our secret /.
Hi, I Boris. Hear fix bear, yes?
They didn't make backups beforehand? What kinda incompetent sysadmins do they have over there anyway?
In Xanadu did Kubla Khan
A stately pleasure dome decree
Crap! UNDELETE UNDELETE!!!!!
Charter Manager: You sure that these are the correct accounts to nuke.
Charter Employee: Yessiree ! 'Click'
Charter Employee: Oh shit.
Charter Manager: What ?
Charter Employee: 'Surfs over to Monster.com' Oh nothing. Nothing at all.
Guns are for wimps... Use a crossbow.. this way you can pin them to their chair when you go postal.
Sure, users should always back up their email. But so should the providers. It takes very little effort to schedule staggered daily/weekly/etc backups. Basically all you need is to have some flunky to carry tapes and disks around.
I use Thunderbird for my charter.net email accounts. I don't understand why more people don't do the same.
Speedy thing goes in; speedy thing comes out.
Back in the olden days when everyone POP'd their email and disk quotas on the mail server were in the 5-10 MB range, most ISPs didn't bother to back up email for very long because it was expensive and mostly pointless. These days, however, with everyone pushing huge disk quotas and webmail interfaces, the ISPs must be aware that most people will be keeping their email on the server for long periods of time. If this service were free, I might be able to excuse some shoddy backup practices, but in the case of an ISP your mail service is part of the overall service that you're paying for.
So, either Charter doesn't back up email very well, or their process to "clear out old accounts" involves actually deleting all of the backups of those accounts as well. I already addressed the issue with the former scenario, but if it's the latter, I'd have to say that's a pretty nasty practice too. Any time you clear out old and "unused" data, you have to assume that you're likely to accidentally hit some false positives, which is one of the reasons we have backups in the first place.
These people CLEARLY qualify for jobs working on the White House archive team... Just following the example of our dear leaders, I suspect.
Sig Registration Form 34c_766(a) submitted to Ministry of Signature Management. Approval pending.
Somewhere deep in the bowels of a server room at 2:14 am...
clickety clickety (SIGH) clickety (beep)
clickety clickety (beep)
clickety (beep)
clickety clickety (beep)
click- OHHHH SH**! F***!
stuff |
I just find it incredible that they don't have their own backups of this data.
How did they not think to backup the data before going deleting various accounts. Do they not legally have to keep records of things now-a-days incase they get their data subpoenaed?
This, once again, highlights the trouble of using "remotely hosted applications" - you are not in control of your data.
/I/ lose it it's on /my/ head.
I always POP my email down to my own local computer.
At least if
A work that expires before its copyright never enters the public domain and thus enjoys eternal copyright protection.
At a minimum, the compensation should be
.PST, Outlook Express, and competing mail-readers on Windows and non-Windows platforms would go a long way to restoring goodwill.
* anyone can get out of any long-term contracts without penalty plus reimbursement for setting up service with another provider
* everyone gets a refund for the last 3 months' service
Charter should also look into tools for web-mail-based users to download their mail in bulk without having to use Outlook. Buttons that say "download selected messages," "download selected folders," "download everything," etc. along with a choice of popular file-formats such as
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
for (int i = 0; i < accts.Length; i++)
:)
if (accts[i].IsInactive) accts.RemoveAt(i);
Anyone spot the bug?
For one thing, they involve a certain sort of lock-in. For another, the ASP never do the jobs as well as dedicated email or hosting companies.
Get your own domain name, so that you can switch providers (hosting and email) if you need to. Most people here know this, but I deal regularly with lots of people who even run their businesses with email addresses at verizon.net or charter.net or comcast or even AoL. I've been preaching this for a long time. I will certainly use this case as a frightening example.
Personally, I'm a fan of fastmail.fm which is the best IMAP provider I know. Years ago they did have reliability problems with downtime in the past, but their back-ups were rock solid. And they are very open with users about what kinds of redundancy, back-up and disaster recovery systems they have in place. And, of course, they will (for a fee) host the MX for your domain, so you aren't locked into them.
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
All of us who are programmers have accidentally deleted important information at least once or twice. This kind of error can be forgiven, since we all make mistakes. But there is NO excuse for Charter to not have had a backup of those email accounts before erasing them. Even the most novice programmer, NA or DBA knows that when there is the potential for fiddling with important data, you make sure there is a backup of that data somewhere. I'd be shocked if there were not a class action lawsuit in the works. I know the loss of all my email would be worth a lot more than $50 to me.
Proverbs 21:19
I look at friends with several years of e-mail sitting on Yahoo, or Hotmail, or Gmail, and always think that they're rather foolish. Aside from the chance that their entire filing system could go poof at any time, or that the company holding it could go bankrupt, those interfaces just aren't intended for archiving and managing large volumes of e-mail.
And of course, if your Internet connection goes down you're cut off from everything.
Still, I can't believe that the ISP doesn't have a backup somewhere. Charter may be looking at a lawsuit in this.
Three Squirrels
Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
go and get a gmail account if you don't already have one... use fowarding options... make sure you are forwarding to your isp's email address. If that's not big enough, send it elsewhere to another huge, free email account. If you are paranoid, do this 3-4 times so that you have multiple email addresses all receiving the same boatload of emails.
Back in the dot-com days I worked for a local ISP, and established my online identity over the years. The company died, and I lost my long-established email address. Lesson learned - I obtained my own domain name and webhosting, just on shared servers, mind you, and now I have a portable identity that I have control over. Webhost screwing up? I've had it happen a few times now. I just point the domain elsewhere. I have unlimited POP, IMAP, and even webmail. Multiple spam controls that I can fiddle with. And I don't have to worry about Google, Yahoo, etc fiddling with anything either.
It isn't hard, either. My 63-yo father is now doing the same thing, as he switched ISPs for the first time now that he can get DSL out on the farm, and he isn't the most technical guy.
Remember the story about the disgruntled dba who was going to wipe the company's whole database but his "logic bomb" failed? People were saying "Oh, it wouldn't be all that bad, they'd just restore from backup and lose a day or two of work." Yeah, that's assuming the company did things the way they're supposed to. It always seems like companies do things half-assed. This is just another example.
Kwisatz Haderach
Sell the spice to CHOAM
This Mahdi took Shaddam's Throne
I know many people are saying that Charter should have backed up the email, but I used to work for AOL, and I know they don't back up any of the email, other than having redundant servers in multiple locations. By not sending your email to tape or other media, they can't be hauled into court and forced to give it up. Once it's gone, it's gone.
That said, it's standard practice when deleting an account to mark the data as deleted, so that it looks like it's gone to the user, but it's actually pending deletion later. Then, when someone complains or pays their bill, you can restore what was "deleted." After a predetermined amount of time, if you don't complain, a cleanup script deletes it permenently.
All ideas^H^H^H^H^Hprocesses in this post are Patent Pending. (as well as the process of patenting all postings)
Oh, say does that Star-Spangled Banner entwine / The myrtle of Venus with Bacchus's vine?
The email most safe from spam is the one that doesn't exist.
What about those of us who will cancel their Charter service? How does a rebate help us?
I had already planned to cancel next Monday when my DSL gets hooked up.
IWARS.
People, in general, disappoint me. Politicians even more so.
It had been working for at least 3 years after I quit using them. It had still been in semi-active use up until six months ago. Tried it today and it no longer works. I was surprised it lasted so long.
They are the crappiest ISP in the broadband market I have ever had the displeasure of having to deal with. I am a Charter Customer, but I didn't loose mail because I run my own server in violation of the TOS/AUP. This is hands down the number one reason why it should be allowed.
Power Corrupts,Absolute Power Corrupts Absolutely, leaving one person(group)in charge is absolutely corrupt.
I had the unfortunate experience of being a neglected customer of Charter Communications for over three years during college. During that time I experienced regular television and internet outages. I or my rommates would call to ask about the problem and would get a different explanation/excuse from each person we talked to. The really annoying part was that we were regularly treated like we had no right to complain, and these things were to be expected. Really? I'm paying you over one hundred dollars a month for services that you can't provide to me when I need them? If we complained enough they would knock a little money of the monthly bill for us, but it shouldn't have had to work like that.
Some of my previous rommates have switched to SBC DSL and DishNet, which recently started offering good deals in the area, and they've apparently been much happier with it. I've since moved away from the area to a Comcast area, and have so far been ok with it.
Charter's method of operation sudo purgeoldemailaccounts contents of purgeoldemailaccounts: rm -rf /
People still leave messages on server? Worse, they rely on it still being there? Man, I must be getting old, I thought we were past this, but apparently web mail has brought back a few of the 'net's child diseases.
/ Per
It is not Charter's job to determine what is and isn't important for me to keep. They should have prepared and if they didn't (which they clearly have not), then shame on them.
I think it's 100% reasonable to expect your e-mail (which is paid for as part of your service) be kept safe and not deleted. I do not think that is asking too much -- if your provider includes that service as part of their product offering.
If Charter had said "hey guys, we'll give you free e-mail but we aren't going to support it", then that's another story. But that is not the case here. E-mail is about as fundamental to "teh intarnet" as any service aside from the WWW. Customers should expect their e-mail to be safe (which means backups!!!)
..should have had backups... should have been more careful... of course.
But what's done is done and props to them for a bullshit-free apology.
Most people are prepared to cut you some slack when you screw up as long as you admit your mistake.
- recognise what it was that you did wrong
- claim responsibility for your actions
- apologise
- state clearly what you learned and what actions you will take to prevent a recurrence
Or you could take to legal advice / bush administration route
- flatly refuse to acknowledge that anything bad actually happened
- talk about how 'the other guys' screw up all the time
- start an internal investigation and refuse to comment on the issue while it is under investigation
- eventually admit that 'mistakes were made' but no, you can't think of any specific examples right now and it was all someone else's fault and you there's no way you could have known it would happen.
Part II:
**Status 'F' = Fired His Ass
"He Who Dares Wins"
Note the use of the passive voice, which is commonly done to avoid taking responsibility. It seems like even when they're trying to apologize, spin-doctors can't turn off their instinct of avoiding responsibility for mistakes.
the customers could always write to the NSA and ask for their backups.
that's a paddle'n.
I've never had to back up email for 50 million people, but I've been responsible for a system with 50 thousand people. We didn't backup our email, didn't even come close to having the resources to do so, and it clearly stated in the SLA that we didn't do backups, and if your email got lost, tough shit. Our customers got what they paid for, since the email was free.
Mostly likely their asses were covered by their service agreement. I am pretty sure that Yahoo's policy for lost email is "tough shit" as well.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
The statement is still valid: your email server is not and cannot be a reliable long term archive. You are foolish if you leave VALUABLE information on it that you don't ever want to lose or have compromised:
1. Most access is plain text and subject to snooping/hijacking (passwd/userid/content)
2. Email is the most abused internet protocol (my opinion) by zombots, spammers, and virus/trojan propagators. ISPs do a lot to counter spam and threatening content but sometimes they get hosed. Or your home machine gets compromised and the ISP will do things to clean up.
3. Grooming accounts for stale accounts, unauthorized accounts, and stale/large data is a reality on most messaging systems. "Ooops"s happen. Usually stated as "sxxt happens"
Whatever your feelings of outrage are, common sense says put your important stuff somewhere close at hand and under your own control.
My 2 cents FWIW
i can kinda sympathize with charter (shudder, omg did i actually type that?!). ok, i can feel for their admins.
for eight months, i worked for a small-town isp with dsl and dialup customers. we had old equipment and no budget for upgrades. we had an autoloader that would occasionally snap tapes, old drive arrays that would fail with no replacement parts on hand ("whuddya mean, we got harddrives right there" "those are ide, i need scsi3"). backupexec would report completed jobs but find no restorable data. our dhcp scope was too small to serve all our customers at once (meaning I would hunt down inactive leases and free them for people trying to connect at that moment).
i did get a new tapedrive after six months of empty promises from my managers (and two catastrophic domain controller failures). i left them a year ago, and they still have my job posted on their site. (don't bother asking for the link, you do not want to work there.)
...all 14,000 accounts belonged to members of the US government's executive branch.
How 'bout instead of giving everyone $50 when something easily preventable--on the order of duh^10, if I remember high school math well enough--they invest in some fucking backup systems for when catastrophes happen?
Mikey-San
Karma: +Eleventy billion (mostly affected by watching Celebrity Jeopardy)
I worked for Charter back in 03 and 04. I noticed a trend before I decided to stop working there. They are self serving morons. They even wanted their tech support to try and upsell someone. "Yeah, um my internet is down" "Okay we can fix that, while you are waiting for the repair guy want to watch HBO for *such and such* extra a month?". I kindly told my bosses at Charter to shove it up their butt. They hired me to fix issues, and I was damn good at it, not to bring in more money. By the way, 3 months after I quit (and I saw hints of this too), they moved the tech team from St. Louis to Louisiana or some crap. Charter has been going down hill since the inet bubble burst back in 02 and they have been Enron'esq since then as well.
use gmail's POP collector to harvest mail from Charter to Gmail, then use Thunderbird to read gmail
email will be backed up on: Charter's servers, for as long as they can manage. Gmails servers, til judgment day, and thunderbird's mbx storage file, until you have a HDD fail or similar bad luck.
I am often asked to 'set people up on the net' and that is how I always configure their ISP email accounts.. it gives a little bit of extra resilience and also, free virus/spam filtering
I have been a user for about 10 years. This ends Feb 2014. The site's been ruined. I'm off. Dice, FU
That is most good. Most good.
-Shoe
Your statement about people complaining about the trash(deleted email) being emptied brought back a fond memory of mine. I was early in my IT Admin career (was a programmer for over 10 prior to that), I was working for a major insurance company administrating their CC:Mail network. We had issues of people never emptying their "trash" folder and it was taking a long time to do anything for everyone on the system. We sent out a memo to the entire company telling them that in a week we would start emptying the trash folder nightly around midnight before the backups and other maintenance begins. very few people bothered to pay attention to this memo.
Here is where it get...funny. A high-muckety-muck (eg: pointy haired VP) called the help desk screaming for his trash folder to be restored as it was emptied without his permission and had important files in there. He wanted to see the guilty parties in his office post-haste. I was part of the team, so I had to go to his office. while waiting outside his door for him to let our team in, I grabbed a trashcan and some vertical file folders and paper from his secretary. (can you see where I'm going yet?) We were let in, and he proceeded to rip us up one side and down the other. Our manager brought a copy of the memo, which he promptly threw away while continuing to yell.
I asked to speak (everyone else was quietly taking the heat) and proceeded to put the trash can on the desk, put the vertical file folders in the trashcan and put paper in each folder. While I was explaining this analogy to him, I asked if every day his office trashcan was empty when he came into the office. He said yes. I basically gave him the analogy that the trash folder in cc:mail was the same as his office trashcan - whatever went in during the day was retrievable, but at midnight, the office cleaning crew would come in and empty his trashcan and we also would be emptying his electronic trash folder the same way. He blinked, and understood what we were doing, pulled the memo out of the trash and reread it. we were dismissed with no apology back to our offices.
Note that I was a contractor - I couldn't be fired...just sent away back to my firm to go on to the next assignment. I ended up being there for 2.5 years. I still laugh at that situation even today.
I really don't see this as that big of a deal. It was just email and I'm willing to bet 90% or more of it was spam anyway.
It would be a big deal however if they accidently deleted paid web hosting accounts and the users had not the brains to backup their web site data.
Kickass Cheap Web Hosting
There was a time when service providers would clean your mailbox and CHARGE you $50 for it...
If all the ISPs do this, what will the FBI and the NSA read to keep themselves busy???
Agreed. All of their local phone lines are connected to machines that tell you to please wait and disconnect you after a few minutes. It is absolutely impossible to even get someone who speaks English on the phone... After tracking down the office, they admitted that yes, there is not a single phone connected to any of their local numbers.
I'm glad they seem to be representing themselves more accurately lately. Hopefully it will become more obvious how atrocious their service is.
I don't get upset or worried when I hear a sysadmin shouting and screaming. It's usually the result of some user doing something stupid but limited in scope.
I was sharing an office with the lab's sysadmin. One day, while I was happily programming away, I heard the quiet utterance from my office-mate: "Oh, shit." Shivers ran down my spine and I started to panic. I knew immediately that all hell was about to break loose.
Truly a frightening phrase to hear from your sysadmin.
Is there anyone out there who actually uses their ISP-provided email account for anything important? What do you plan to do when you move, or switch ISPs?
All my Bellsouth.net e-mail account contains are lots of marketing messages from BellSouth.
You keep using that word. Even if you spelled it correctly, I do not think it means what you think it means.
To anyone who's had the pleasure of dealing with Charter for more than a few months, this isn't exactly astounding. Their customer service is awful, their sales department lies through their teeth, their billing department proves itself to be continually inept, and their network has numerous single-points-of-failure (i.e., a poorly placed server for the cable modem configurations).
The plus side is, I had 10m/1m cable long before anyone using Comcast did.
Now, to see if I can get $50 out of them. Not that we use the email accounts they offer, but we do have some.
I was a Charter customer for a while. They has been trying to sell their subscribers to somebody else for several years now. They only invest the absolute minimum in their current customers. Charter internet is low priority "value added service" they only have because they are required to in some jurisdictions where they are the monopoly cable provider. Pretty much everything at Charter is subcontracted to lowest bidder locals.
After a year of intermittent internet service and fuzzy cable channels I jumped ship for DSL. I get a number of HD channels for free over the air. Netflix and iTunes takes care of the rest.
My employer does a lot of state and local government systems installation and support contracts. All the email systems we install must have archive mechanisms that capture copies of all emails that are sent and received and that the end-users cannot access or modify. Emails sent or received by government employees are often considered public records, and typically the state has a set of regulatory statutes that govern how long each classification of email must be retained, some classes must be kept forever.
Ever wonder why so many state and local government email system run on Lotus Notes/Domino? It's because Lotus has a built-in feature called "mail journaling" that automatically does the archiving. In addition, Lotus has a standard clustering capability in its design that allows you to replicate the entire servers and their contents effortlessly across multiple machines. When I first had to learn Lotus, I thought it was going to kill me, but the more time I spend with it, the more I realize it is an incredibly powerful and capable messaging and application/database platform. But it has a super-weird learning curve to it that most people never can seem to "get it", hence the widespread fear and loathing towards Lotus Notes.
When I moved apartments last month, despite the locations being only a handful of blocks away, it took 3 hours on the phone and four service visits to install my internet connection.
First Service Rep: Local apt. rep couldn't figure out why the modem wouldn't synch and didn't know who to call; left confused and muttered something about having to schedule a service appointment.
2nd Service Rep (4 days later): Tech missed appointment because he went to the old address.
1rst Phone Call to Service Department to describe problem: Dyslexic teenager who keeps switching numbers on my phone and address tells me it is not their problem and I have to call Linksys; absolutely clueless and runs no tests whatsoever.
2nd Phone Call to Service Department: Actually get someone who knows what they are doing and states they would have to check the line and signal levels at the apartment box.
3rd Service Rep (4 days later): Shows up, tries to check line in apartment box but cannot because lock on the box is jammed shut and he doesn't have the tools to open it. Leaves and says he will schedule someone to look at it the next day.
3rd - 5th Phone calls: Reps never show or call back, phone calls to service state they are really busy.
6th Phone call: actually get a scheduled appointment.
4th Service Rep (4-5 days later): Checks levels, but also calls the problem in only to find out the account was set up wrong -- "I see this all the time when people move from one node to the next." Fixes problem in five minutes. All previous phone calls / talks say my modem is provisioned correctly and working fine.
No internet access for 2 weeks: not so bad.
Amount of wasted time resolving the problem: infuriating.
When charters email servers went poof.... and it took them 2.5 months or so to get new ones ( and they had NO backup servers)
I should have made them pay for the college classes i missed, since the college would not change email addresses... only use the one you registered with.
To err is human; effective mayhem requires the root password!
What do you want to bet this was an offshored person who did this? Or a junior developer who was given too big a task. This type of thing--which will kill Charter's ISP business, I would think--is precisely why it is stupid for companies in general--not just this episode--to scrimp on IT resources. You get what you pay for...
The funny thing about that is, my email account with them is still active, with all of my emails.
While that in itself isn't very funny, what is, is I haven't been a Charter customer in about 3 years. I just keep getting my mail forwarded to my gmail account.
Their service hasn't exactly been the most reliable, so I've now pointed all my webmail accounts to forward to gmail. When I read this I was first smug, then realized that as soon as I get all uppity, gmail will hose me. Just goes to show that you should always back up to tape; no wait, floppy; hang on, cdrom ... dvd? usb extra drive.. paper????
B) Eliminate all the stupid users. This is frowned upon by society.
"Computer experts advise backing up all important e-mail."
Yeah, you stupid ass users. Why didn't you cut and paste every single message from the web interface into text files? What, do you expect these billion-dollar corporations to shell out for backing up YOUR MESSAGES their own drive? And since when do they have to give you warning or recourse before wiping THEIR DRIVES? Get a brain morans.
NetApps are commodity. ZFS is free. Bigger storage iron is a competitive marketplace with thin margins. Who on earth is doing production storage without modern data management facilities?
ian
I know these kinds of things aren't supposed to happen, but sometimes they do. The worst part for the company itself is not the backlash they receive...it's the fact that nothing they do and nothing they say will fix it.
It's one thing if you have angry customers over something you have control over. It's another thing entirely if your customers are angry at you AND there isn't a single solitary thing you can do. That said, I hope that they are more careful in the future...
Well, I hope the next time I get on an airplane, the pilot checks to see if the ailerons are working before he takes off. It would be "unfortunate" if we fell out of the sky.
Well, if we're doing silly analogies: I don't stuff cash into my banker's pillowcase; I find a safe place for it, which may be a bank. Email inboxes are potentially volatile on-line storage. A pillowcase, not a bank.
This poor excuse of an ISP did the exact same thing to me several years back. I dropped them immediately and will NEVER go back. FYI, i never saw $50...never saw a dime...just some crappy apology saying i wasn't the only one affected. F charter.
Why waste your time explaining to Slashdaughers how things are done in the real world?
great........that's the only sysadmin who trusts hard disks without backups
coooooool.......
Unlike Charter (who probably uses something not all that different from an mbox file)
Actually I'd think Charter would use whatever file format Outlook uses seeing as how Microsoft cofounder Paul Allen has a controlling interest in and is the CEO of Charter.
FalconShould there be a Law?
I administrate hosted email services. These guys are morons or worse. Period. There is absolutely no reason on earth that this should have happened. It represents a massive failure on many levels.
It's a "small" service being provided for "free" to give added value to their overall service. It's important to people, and it is a reasonable assumption that this ISP could safeguard their data. For sure, they could have been a lot safer with their own domain and IMAP supporting email service, but it DOES NOT negate or marginalize just how bad this ISP screwed up. This ISP had all the motivation it needed to provide safeguards for this data. It's not like this is new or anything or prohibitively expensive. Any business today offering services on the net would be insane to have any single point of failure in their systems. Stark Raving Madness. Looney Tunes.
Many posters can point out that email is not a small service and that it does take a lot of resources to do correctly, but we have a lot of experience doing it and storage is cheap.
My systems do not have any single point of failure anywhere. I could deliberately destroy whole domains and hundreds of users or basically torch a whole cabinet, and I could still bring everything back up on separate systems within a few hours.
I am not spending that much money doing it either. A couple of iSCSI racks with RAID 6 and snapshots takes care of the data, which is shared between multiple virtual email servers. It is not even limited to location. These iSCSI racks are synced between both locations. So one location can act as a fail over for the other. Furthermore, the IMAP accounts have a separate backup of each and every email (both inbound and outbound), contact, etc. copied instantly. They are split when coming and going out to separate backup servers that CANNOT be deleted or modified. That is their purpose. So we have no single point of failure for the current email account data, and a permanent backup of all data maintained separately with no single point of failure either. When accounts are deleted, all of their information, across all systems is backed up and synced between iSCSI racks. After a few billing cycles, it will end up getting deleted. So there is just no way possible to manually run any script that will completely remove all the mailbox and account information irretrievably. We approached the situation from the start with the attitude that failure was unavoidable, and that we assumed a system would fail at some point. By providing for it and creating multiple points of failure, we reached a point where 2 separate locations would have to be razed to the ground the stop the service and lose all the data.
I don't think this is groundbreaking and is rather obvious. Once it is setup it runs rather smoothly. These guys must have been saving a ton of time and money running it on the edge like this. I'm just so blown away, since I don't think I am that freakin smart or anything and even I could figure out how to not have one of these nightmares.
I'm truly shocked. I always had the impression that companies this big hired reasonably smart people in their IT departments and implemented intelligent solutions. My gut feeling is that these guys are the exception and not the rule.
If people would actually read the TOS they'd see that many ISP protect themselves against cases like this by adding something like "we try everything we can to backup your data and keep it safe, but if something goes wrong we can't guarantee for backups. It's you responsibility to keep backups of your data." Or often it contains something like "You should check your mail everyday, have to check it at least once a week and all mails older than X may be deleted at any time". Most ISP's have something like this in their TOS, in fact - I don't know any that don't. So: Keep your own Backups! It's really simple, you could just create a second e-mail account somewhere with ANOTHER company than your current e-mail provider -like GMail- and then just forward all your mails there as a backup, or just use IMAP to regularly copy the contents to an IMAP account on another host.
IMAPsize is a Windows utility for IMAP account maintenance and it has a function to do incremental backups of an IMAP account. Just a few clicks and the process can be left to do its job. Then you can burn it to a CD and keep it in a safe place.
So if a similar mistake happens at google with your completely free service...
Do you really think that they'll be willing to find your data, or even able to find it.
Charter is obviously angling to become the new email provider for White House staff. This shows that they have the moxy to do the job!