Slashdot Mirror
British ISP Bombards Users With Deleted Emails
judgecorp writes "For three days, customers at British ISP Sky have been receiving a flood of old and deleted messages. The problem started when the company switched its email provider from Google to Yahoo. As it began to move accounts from one provider to another, it became obvious that the new provider could not tell which emails in the old system had been sent or deleted. Some users had up to 8000 old messages. The incident has been going for three days, as users are migrated. Sky is apparently unable to fix the problem — its best advice been to suggest users delete the old messages."
You might want to read the summary again. Google already had the data. They shipped it to Yahoo, and Yahoo bungled it up.
I guess they used the Facebook definition of deleting contents...
if they had only let the good yahoo engineers work from home, none of this fark-up would have happened...
(snark)
--
"It is now safe to switch off your computer."
OK, manager of a data centre and an ISP here. This happens very easily if you move from one IMAP/POP3 provider to another (even if just changing software on your mail server, sometimes even with a major version update). There are two issues normally:
For IMAP users, the way the IMAP server stores it's flags for Seen, Deleted etc. may not be recognised by the new software.
For POP3 users, whether or not an email has been downloaded is tracked by the client, based upon the UID for the message. If this UID is changed (different servers use different systems) the client will decide the message are new - where users decide to leave messages on the mail server (rather than deleting them after retrieval), this is a common problem.
Neither of these cases is necessarily the fault of Sky, sometimes it's just not possible to reliably import this information between mail servers, and in the case of POP3 users, it's just down to the fact that POP3 is not designed for leaving read messages on the server for multiple clients to pick up.
Actually, I'm pretty certain I know exactly what happened, because I just handled a major migration to google and dealt with an issue like this. It's due to the way google uses labels instead of folders, and how they (mostly-transparently) expose them as folder via imap (though this is one of the few non-transparent side effects).
In google, when you "delete" a message via imap, it doesn't get deleted. Instead, google just removes the label. That message still exist with all of the other labels, and it also exists in "All Mail" (which is exposed via IMAP through the "[Gmail]/All Mail" folder). So, if you have new mail come it, it is by default in your INBOX and your "[Gmail]/All Mail" folders. When you then delete it from the INBOX via imap, it's still in All Mail.
The way to deal with this is to move the message into "[Gmail]/Trash" instead of deleting it. That will truely delete it. However, since that wasn't done all along, those "deleted" messages are now orphaned in "[Gmail]/All Mail". There is a potential way to resolve even this problem, but it depends on how the account has been used. If users have logged into Gmail directly and taken advantage of the "Archive" feature to remove a message from the inbox (without truely deleting it) then all bets are off. There is no way to differentiate intentionally saved messages from deleted-via-imap messages. However, if it has only been accessed via imap (and users haven't intentionally been trying to take advantage of the All Mail folder), then you can do the following via a script:
Go through every message in "[Gmail]/All Mail".
For each message, try to find that same message in another folder.
If you don't find it in another folder, then that message only exists in "[Gmail]/All Mail". You can then move it to "[Gmail]/Trash" to get rid of it.
Searching for messages 1 at a time is a bit slow, so you can optimize this by first building a list of all messages in other folders. If you just retrieve a few headers from every message, it's actually fairly fast. The "Message-ID" field is usually sufficient for this, but there may be messages here and there that don't have that header, so you'll have to have other headers to fall back on.
oh bullshit. It *is* the fault of Sky because they chose to change what they were doing without working around the issues their customers were going to see.
Had they actually *tested* this, in advance, would we be discussing the various flaws in each mail systems' implementations, or their real/imagined problems?
No. We would not even know it happened.
Don't blame Google's use of IMAP flags and folders, blame Yahoo!'s apparent lack of planning. Or Sky. Or whoever. Plenty of blame at the receiving end.
If you're moving mail from some system into yours, the responsibility is yours to make it right.
And I've done this. Wait till ya hose up the passwords, my friend. Fun times.
deleting the extra space after periods so i can stay relevant, yeah.
Recently, I saw a large organization convert email systems. They are split into handful of business units that had their own IT staff and email systems. They made the announcement and told us how it would go. All your email, calendar, and contacts would be converted and everyone's mailbox size would be 25G. As the migration got closer, one business unit's IT kept making frustrating announcements while the other units went as smooth as expected. (I don't know the source of that units internal issues, but this is what was seen from the outside)
Announcement: Our mailboxes will be 2G in size instead of the 25G that everyone else gets.
Announcement: You will have to move your contacts by hand, we will take care of your calendar and email.
Announcement: You can't move personal email groups because the addresses are not saved in any known email standard.
Announcement: We tried to migrate a calendar but could not get it to work. We won't be able to migrate calendars.
Announcement: The automated email migration is taking longer than we expected, bu the vendor is working with us.
Announcement: The automated migration is taking too long, so we wont be able to do it. You will have to archive your own email.
Announcement: If you need someone to migrate your stuff, we can have a contractor do it for $200.00 for each mailbox.
Announcement: We recommend everyone purchase a license of Acrobat Pro and PDF your emails. Or email old messages to your self after the switch.
Announcement: Our existing system archives email after 1 year, you have to open each archived message once before you PDF it or it will PDF an empty email.
Announcement: Here is a trick to open up to 10 archived emails at once.
Announcement: We must remove every copy of the old email client (due to licensing) so all your old personal archives in that system will not longer be accessible.
And all the central mail groups needed to be recreated by the admin team with no conversion from the old lists. And the new groups could not contain outside email accounts. There were a hand full of mistakes that showed that someone had to do this by hand (scripts or automation would have create different mistakes).
I kind of feel bad for that IT team. Because if that was the flow of announcements going out, you can only imagine what they were dealing with inside the team.
Surely they could just opt to sync every folder except the "[Gmail]/All Mail" folder. Doing that, and syncing [Gmail]/Trash to the Yahoo! deleted mail equivalent would sort it all out.
I think you're right though. Sounds like the people handling the migration just aren't very familiar with the Google IMAP interface.
Python coder | PyQt Applications | Writer
Neither of these cases is necessarily the fault of Sky, sometimes it's just not possible to reliably import this information between mail servers, and in the case of POP3 users, it's just down to the fact that POP3 is not designed for leaving read messages on the server for multiple clients to pick up.
What a ridiculous thing to say. Sky is one of the UK's biggest ISPs, and they're moving their email provider from probably the biggest webmail provider in the world (Google) to probably the second largest (Yahoo). Are you really telling me that it was beyond the abilities of any of these three parties to a) do a test migration to check for serious errors, and b) make sure their migration methodology/toolset was compatible with the two technologies at hand before making the move?
To let such a huge and obvious public-facing error go live to so many people smacks of absolutely shocking incompetence.
This raises several questions:
Yahoo still exists?
Who on earth would move them from Google to Yahoo instead of the other way around?
Did anybody expect something different?
Who got paid for this?
If the service "offers POP3 support", which presumably they do, then it is the service supplier's problem. Not the user's fault for using one of the two services on offer which happens to be the less sophisticated one.
And in any case, I've not read anything in this thread or TFA to imply that POP3 users are affected while IMAP ones aren't. The GGP points out a way in which the problem would occur for users of both technologies. If I use an IMAP client to "delete" an email on one day, there is zero good reason for it to reappear in my inbox a year later. That is a system error pure and simple.