Slashdot Mirror
British ISP Bombards Users With Deleted Emails
judgecorp writes "For three days, customers at British ISP Sky have been receiving a flood of old and deleted messages. The problem started when the company switched its email provider from Google to Yahoo. As it began to move accounts from one provider to another, it became obvious that the new provider could not tell which emails in the old system had been sent or deleted. Some users had up to 8000 old messages. The incident has been going for three days, as users are migrated. Sky is apparently unable to fix the problem — its best advice been to suggest users delete the old messages."
You might want to read the summary again. Google already had the data. They shipped it to Yahoo, and Yahoo bungled it up.
OK, manager of a data centre and an ISP here. This happens very easily if you move from one IMAP/POP3 provider to another (even if just changing software on your mail server, sometimes even with a major version update). There are two issues normally:
For IMAP users, the way the IMAP server stores it's flags for Seen, Deleted etc. may not be recognised by the new software.
For POP3 users, whether or not an email has been downloaded is tracked by the client, based upon the UID for the message. If this UID is changed (different servers use different systems) the client will decide the message are new - where users decide to leave messages on the mail server (rather than deleting them after retrieval), this is a common problem.
Neither of these cases is necessarily the fault of Sky, sometimes it's just not possible to reliably import this information between mail servers, and in the case of POP3 users, it's just down to the fact that POP3 is not designed for leaving read messages on the server for multiple clients to pick up.
Actually, I'm pretty certain I know exactly what happened, because I just handled a major migration to google and dealt with an issue like this. It's due to the way google uses labels instead of folders, and how they (mostly-transparently) expose them as folder via imap (though this is one of the few non-transparent side effects).
In google, when you "delete" a message via imap, it doesn't get deleted. Instead, google just removes the label. That message still exist with all of the other labels, and it also exists in "All Mail" (which is exposed via IMAP through the "[Gmail]/All Mail" folder). So, if you have new mail come it, it is by default in your INBOX and your "[Gmail]/All Mail" folders. When you then delete it from the INBOX via imap, it's still in All Mail.
The way to deal with this is to move the message into "[Gmail]/Trash" instead of deleting it. That will truely delete it. However, since that wasn't done all along, those "deleted" messages are now orphaned in "[Gmail]/All Mail". There is a potential way to resolve even this problem, but it depends on how the account has been used. If users have logged into Gmail directly and taken advantage of the "Archive" feature to remove a message from the inbox (without truely deleting it) then all bets are off. There is no way to differentiate intentionally saved messages from deleted-via-imap messages. However, if it has only been accessed via imap (and users haven't intentionally been trying to take advantage of the All Mail folder), then you can do the following via a script:
Go through every message in "[Gmail]/All Mail".
For each message, try to find that same message in another folder.
If you don't find it in another folder, then that message only exists in "[Gmail]/All Mail". You can then move it to "[Gmail]/Trash" to get rid of it.
Searching for messages 1 at a time is a bit slow, so you can optimize this by first building a list of all messages in other folders. If you just retrieve a few headers from every message, it's actually fairly fast. The "Message-ID" field is usually sufficient for this, but there may be messages here and there that don't have that header, so you'll have to have other headers to fall back on.
oh bullshit. It *is* the fault of Sky because they chose to change what they were doing without working around the issues their customers were going to see.