Slashdot Mirror


Mac OS 10.9's Mail App — Infinity Times Your Spam

An anonymous reader writes "Email service FastMail.fm has an blog post about an interesting bug they're dealing with related to the new Mail.app in Mac OS 10.9 Mavericks. After finding a user who had 71 messages in his Junk Mail folder that were somehow responsible for over a million entries in the index file, they decided to investigate. 'This morning I checked again, there were nearly a million messages again, so I enabled telemetry on the account ... [Mail.app] copying all the email from the Junk Folder back into the Junk Folder again!. This is legal IMAP, so our server proceeds to create a new copy of each message in the folder. It then expunges the old copies of the messages, but it's happening so often that the current UID on that folder is up to over 3 million. It was just over 2 million a few days ago when I first emailed the user to alert them to the situation, so it's grown by another million since. The only way I can think this escaped QA was that they used a server which (like gmail) automatically suppresses duplicates for all their testing, because this is a massively bad problem.' The actual emails added up to about 2MB of actual disk usage, but the bug generated an additional 2GB of data on top of that."

34 of 158 comments (clear)

  1. Apple Build Quality by johnsie · · Score: 2, Funny

    This must be the Apple build quality that people keep telling me about.

    1. Re: Apple Build Quality by ericloewe · · Score: 3, Informative

      Well, software that generates a thousand copies of junk (seriously, the spam folder of all things...) isn't very typical.

    2. Re: Apple Build Quality by Anonymous Coward · · Score: 2, Interesting

      Ships? For the past 3 releases Mail.app will randomly choose not to display the body of a message. I have to quit and reopen Mail.app just to read my e-mail sometimes.

      I thought it was just me and something screwy with my account, but the other day I asked a coworker and immediately heads began popping up from cubicles everywhere within earshot, with people admitting they've had precisely the same problem.

      Apple's focus on iOS and cute little phone apps has, for whatever reasons, caused defect rates in their core desktop code to serious balloon. Maybe it's because they've greatly expanded their workforce, and there's only so many decent engineers in the valley. I dunno....

      I used to admire the high quality of Apple software. Not anymore. And I was never an Apple fanboy. My relationship with Apple was purely practical. They shipped an excellent laptop with a command-line, and I spend 90% of my time staring at terminal windows and logged into various Linux, OpenBSD, and other unix servers writing low-level C code. I still like their laptop, and damnit the command-line stuff still works well enough, but in the other 10% of my time I spend using Apple software, quality has surely dropped precipitously.

      I once criticized in Slashdot post Apple's recent flagging support for POSIX. An actual Apple engineer replied, but rather than see refutations in his post, to my eyes I just saw confirmations and excuses. While Apple is too busy to support the latest extensions and addendums to POSIX, the teeny-tiny OpenBSD and NetBSD teams have been furiously becoming more compliant, including useful extensions of their own.

      Apple isn't doomed, but the glory days of OS X are gone. iOS will destroy OS X. But all you app developers writing the 10,000th calorie counter... I'm sure you're rejoicing in all those endless tweaks to Objective-C.

    3. Re: Apple Build Quality by BitZtream · · Score: 5, Insightful

      If you think the 'stuff that ships with my food production systems is FLAWLESS' you are an idiot who doesn't understand the way the world works.

      I've worked in food production and medical, nothing is flawless, you have to be an idiot to make such retarded statements.

      You mitigate the risk, try to double/triple/quadrupal check for problems and build in fail-SAFE systems, but you are not flawless.

      Your statement sounds more like an arrogant cluebie beginner who's going to get a nice spanking when reality finally hits.

      Nothing is flawless, to imply you've created something flawless shows your ignorance.

      --
      Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
    4. Re: Apple Build Quality by Skuld-Chan · · Score: 2

      How long has imap been around now? The RFC was last updated in 2003 - thats only 10 years to get it right.

    5. Re: Apple Build Quality by AmiMoJo · · Score: 2

      10 PRINT "HELLO WORLD"
      20 GOTO 10

      Flawless.

      --
      const int one = 65536; (Silvermoon, Texture.cs)
      SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
    6. Re: Apple Build Quality by EvanED · · Score: 2

      Flawless software exists. NASA can do it...

      But not always, even when they try. The LEM had a couple of bugs that made the Apollo 11 landing even more dicey than it would have otherwise been. Mars Pathfinder had a priority inversion bug that caused software resets. And of course the Mars Climate Observer was lost because of its famous metric vs imperial unit mixup.

      And those are just the ones that I know about.

    7. Re: Apple Build Quality by mattack2 · · Score: 2

      Definitely write a bug report at bugreport.apple.com with relevant crash logs in ~/Library/Logs/DiagnosticReports and /Library/Logs/DiagnosticReports.

    8. Re: Apple Build Quality by mattack2 · · Score: 2

      So you wrote a bug right?

      bugreport.apple.com

  2. Re:I guess I have to ask by Anonymous Coward · · Score: 2, Funny

    Why doesn't fastmail also use servers that suppress duplicates?

    The guy who approved it sent the approval via email on a Friday evening ... from his Mac. Since the recipient received millions of copies over the weekend he just figured it was spam.

  3. One infinity drive. by MRe_nl · · Score: 4, Funny

    Not just an address anymore.

    --
    "Kill 'em all and let Root sort 'em out"
    1. Re:One infinity drive. by Anonymous Coward · · Score: 2, Funny

      If you're making a joke about the address of Apple's corporate HQ, you got it wrong:

      1 INFINITE LOOP
      CUPERTINO CA 95014-2083

    2. Re:One infinity drive. by ArcadeMan · · Score: 2

      Maybe he just needs to drink a cup of very hot tea.

  4. All hail Apple's new storage technology! by chrism238 · · Score: 2, Funny

    "The actual emails added up to about 2MB of actual disk usage," So the 1,2, or 3 million emails occupied just 2MB of storage? Wow, Apple should be widely lauded for being able to store each email, including its header, in just one byte!

    1. Re:All hail Apple's new storage technology! by Anonymous Coward · · Score: 3, Informative

      From the summary: "The actual emails added up to about 2MB of actual disk usage, but the bug generated an additional 2GB of data on top of that." Which I assume means that it was really only 2MB of emails, but the duplication (ie millions of emails) used up 2GB.

    2. Re:All hail Apple's new storage technology! by tftp · · Score: 2

      Wow, Apple should be widely lauded for being able to store each email, including its header, in just one byte!

      Even that is awfully wasteful. To store spam wisely you use a counter. If the size of the counter is 64 bits, each individual spam message occupies 64/2^64 bits, or 3.5e-18 bits.

      Naturally, all spam will look alike to you, but doesn't it already?

    3. Re:All hail Apple's new storage technology! by Bronster · · Score: 4, Funny

      We de-duplicate on COPY, so there was only one copy of each email on disk. We don't de-dulplicate metadata though, because it's usually so small, and generally in the cache file of a different folder, where de-duplication isn't possible.

    4. Re:All hail Apple's new storage technology! by immaterial · · Score: 2

      This is one of the bugs I reported months ago that got marked 'closed.' :/

      Enable the "all mail" folder to be visible in IMAP, and remove [Gmail] as your IMAP path prefix. That fixed it for me.

    5. Re:All hail Apple's new storage technology! by Bronster · · Score: 3, Informative

      Sorry, I'm not quite sure where you're getting your information about what Mail.App does. I'm getting mine from the server telemetry logs where the client first identifies itself as:

      "name" "Mac OS X Mail" "version" "7.0 (1816)"

      And then proceeds to issue a COPY command:

      UID COPY 3360991:3361069 "INBOX.Junk Mail"

      See the "COPY" in there. I am the author of the blog post, and I think my credentials in this particular case trump yours, even if you're the author of Mail.App.

  5. Re:I guess I have to ask by homey+of+my+owney · · Score: 2

    It's certainly not the only bug in 10.9 Mail.

    Watch the Mail Activity section when receiving or sending mail. I have no idea what it's counting.

    When I receive two emails and it says receiving 415 and 416 of 416 I kinda get concerned.

    When sending one, sending 6 of 6... again, what is it counting?

    And set up a smart folder that is all Unread mail. Set it up and watch that it's not very smart... Like it really can't tract unread mail at all...

  6. Re:Buggiest Mail by 93+Escort+Wagon · · Score: 2

    Actually part of the problem is they made Mail.app work better with Gmail, but all the hacks that used to be necessary really screw things up. Apple should've posted a FAQ about the changes rather than quietly make them.

    One big change... you need to enable "All Mail" in IMAP now, since the latest Mail.app wants that as the Archive folder (which makes sense). But everyone has it disabled in IMAP since, up until now, it was problematic to do otherwise.

    I found https://tidbits.com/article/14219 to be helpful.

    --
    #DeleteChrome
  7. Whar is wrong with programmers? by cpotoso · · Score: 2, Insightful

    I mean, it is a MAIL program, not a revolutionary new product. The protocols have been out there for years (esp. IMAP). Why is it still buggy? Even worse: why is it buggier than the previous version? If it worked before THERE IS NO F*ING EXCUSE FOR IT NOT TO WORK NOW. Very very very lame.

    1. Re:Whar is wrong with programmers? by immaterial · · Score: 5, Interesting

      I filed some bugs on Mavericks' Mail right after the first developer preview came out (all ended up being marked as duplicates, so others were having the same issue). All were closed the week before the GM was released. And all are still present in the GM; they're MailGmail specific. However, enabling "All Mail" and removing [Gmail] from my IMAP path prefix made everything work.

      Clearly, whoever rewrote Mail to "better" support Gmail decided that as long as it worked okay with just the right settings, any deviation from that wasn't a bug but just user error. Despite the fact that those settings were both perfectly valid and *incredibly* common.

      I think moving OS X to a yearly release schedule results in them pushing things out too fast. It's bad enough with iOS, and OS X is a more complex beast.

    2. Re:Whar is wrong with programmers? by dbIII · · Score: 2

      Whar is wrong with programmers?

      There's a long list of answers to that - but the real question here is what is wrong with the testers that let something stupid from a rookie or slapdash programmer out into the wild. The answer to that is probably that they are at home looking though the job ads because proper testing was not considered important enough.
      Email problems are fun when they happen to somebody else because of how much they can snowball into ridiculously huge numbers of messages. I've worked with an idiot that decided he wanted to get an email notification every time there was a problem with a mail server - you get one guess as to how the thing ran out of disk, memory, crashed and why he's not allowed on production systems any more. That's right, every error message generated a few more.

  8. Re:Buggiest Mail by arkhan_jg · · Score: 2

    Even enabling All Mail doesn't do the trick - from that tidbits article: (which has been doing the rounds quite a bit)

    That is, I can read, move, delete, reply to, or otherwise operate on messages in my Inbox on the Gmail Web site, on my iPhone or iPad, or in another IMAP client, and they all sync up perfectly with each other — but even after several hours, my Inbox in the Mavericks version of Mail doesn’t reflect those changes. It seems not to matter how frequently I tell Mail to check for new messages. I also tried quitting and restarting Mail, rebuilding the Inbox, and forcing a synchronization — several times — but my Inbox stubbornly refused to reflect reality. Occasionally I’ll glance at Mail after having ignored it for hours and notice that the Inbox is closer to being up to date than it used to be, but I can’t figure out when, why, or how this happens. This is the behavior that makes me truly crazy — if I have to keep Gmail open in a Web browser to make sure I’m getting all my messages, I might as well not be running Mail at all.

    My boss is a) an Apple fan, and b) a Mail fan. I've had to instruct him and a couple of other senior management not to go to Mavericks for the time being. Because we use Google Apps, and having mail notifications delayed for hours is going to be a problem. Switching to a decent email client would of course solve the problem, but he loves Mail to death, and he'd rather switch the whole company to another mail provider than give it up (seriously - he suggested it because of this). Not that having Mail cause problems is anything new; my personal favourite is the way Mail does embedded attachments, causing most other mail clients to struggle to handle his messages - usually, they end up with half an email, the attachment, and a second (and sometimes 3rd and 4th) set of attachments with the rest of the email message piecemeal. And then he complains that people can't read his bloody mail.

    Showing that it's not just Gmail getting f***** up the IMAP by Maverick Mail will be quite useful to argue the real problem, as usual, is Mail.

    --
    Remember kids, it's all fun and games until someone commits wholesale galactic genocide.
  9. Re:I guess I have to ask by immaterial · · Score: 4, Informative

    One of the reasons they noticed the issue is they don't actually delete expunged messages for a week (the blog post says for backup purposes). The Mail bug, for whatever reason, duplicates the junk mail and immediately cleans up after itself by expunging the originals. If the server were actually deleting them it wouldn't be such a critical issue (but an issue nonetheless).

    It's also worth noting though that so far, there is only a single report of this, despite the author implying they have a huge number of users. Most likely this isn't something that happens on the average Mail install; it could be that Mail is hitting some error condition on this user's specific account and that is causing the bug to manifest.

  10. Re:Apple done fucked up good by silverdr · · Score: 4, Informative

    [...] I mention that because Apple now seems to be my Microsoft. iOS 7 is ugly as fuck.

    My fucks are always beautiful. Or at least pretty (when I am more desperate).

    --
    Now, mod me down freely. My karma can't get any worse...
  11. Re:Apple done fucked up good by silverdr · · Score: 3, Interesting

    OS X has been going downhill (the autosave/versioning sucks for how I use software) and now with 10.9 mail.app regressions and iWorks losing features. I'm not upgrading to iOS 7. I'm not sure if I'll upgrade to 10.9 I need to buy a new computer in a couple months so I may switch to OpenIndiana. Maybe Linux for steam box, we'll see.

    The last uphill version was 10.5. This current 10.9 is in big part back-pedalling the visuals of 10.7/8 without removing the functional crap they introduced. I decided not to go beyond 10.6 the moment I saw "Edge Resize" in 10.7 :-(( So... no, thanks - even being free (as beer) doesn't make it more appealing..

    --
    Now, mod me down freely. My karma can't get any worse...
  12. iMail has a history of infinate recursion by maas15 · · Score: 4, Interesting

    This isn't the first infinate recursion iMail bug. Around five years ago I worked for a webhost at which we had customers complaining about there being nothing in their INBOX. When we checked, we'd find a giant tree of INBOX folders - for some reason iMail would create a new subirectory called INBOX every time it logged in, and then make the *new* INBOX folder the default INBOX. All the mail would still be delivered to the original inbox...

  13. As someone once said... by feranick · · Score: 4, Funny

    ... it's not a bug. You're holding it wrong.

  14. It's a defense against the NSA by dutchwhizzman · · Score: 4, Funny

    By generating so much metadata, the NSA will overflow and your real messages' metadata will be overwritten!!!1!.

    --
    I was promised a flying car. Where is my flying car?
  15. Re:Apple done fucked up good by Simon+Brooke · · Score: 2

    Out of curiosity I've just tried, with every terminal application I have (xterm, lxterm, uxterm, gnome-termonal, konsole...). I can't do it. Perhaps I'm incompetent? Or perhaps it's a bug in some specific window manager? Or perhaps Guy Harris is special?

    --
    I'm old enough to remember when discussions on Slashdot were well informed.
  16. Blame Google. by bensyverson · · Score: 2

    Gmail uses a highly non-standard implementation of IMAP, which never worked quite right in the previous versions of Mail.app. Mail is expecting a more or less standards-compliant IMAP server. Well, people complained, so Apple modified Mail.app to special case Gmail. Unfortunately the fix looks a bit buggy at the moment.

  17. Re:Apple done fucked up good by houstonbofh · · Score: 3, Insightful

    iOS 7 is ugly as fuck.

    Millions of people love it.

    But you, personally, hate it.

    So... it must suck.

    Millions of people love the Kardashians, Honey BooBoo and Jersey Shore. Your point is irrelevant.