Slashdot Mirror
EarthLink Is Losing a Lot of Email
LandGator writes "Robert X. Cringely, doyen compu-columnist for PBS, reports on a hidden e-mail problem at Earthlink: They're losing up to 9 messages out of 10, found as a result of a friend's testing." From the article: "He sent messages from other accounts to his Earthlink address, to his aliased Blackberry address, and to his Gmail account. For every 10 messages sent, 1-2 arrived in his Earthlink mailbox, 1-2 (not necessarily the SAME 1-2) on his Blackberry, and all 10 arrived with Gmail. Swimming upstream through Earthlink customer support, my buddy finally found a technical contact who freely acknowledged the problem. Since June, he was told, Earthlink's mail system has been so overloaded that some users have been missing up to 90 percent of their incoming e-mail. It isn't bounced back to senders; it just disappears. And Earthlink hasn't mentioned the problem to these affected customers unless they complain."
However, it was via email.
Is earthlink hosting slashdot?!
"It is possible to commit no errors and still lose. That is not a weakness. That is life." -Peak Performance
Less spam. Enjoy.
This ain't no upwardly mobile freeway This is the road to hell
E-mail should never be lost! We had an issue where I work with e-mail BACKING UP for a few months while we implimented new mail servers... but no mail was ever lost.. it either got bounced back (not usually) or would arrive several hours after it was sent. To actually LOOSE e-mail indicates that Earthlink is ACCEPTING the mail and then DUMPING it!!!! When our servers were overloaded, we just rejected the connection, until the mail server could handle more mail.. and then we accepted it.
I checked their mailservers (what the MX record reports anyway), and they have a very generic ESMTP banner, not really apparent which MTA they use. I want to know which MTA can lose mail because of overload. So I can avoid it like the plague. I do know for a fact that Sendmail and Postfix send a 4xx error if mail cannot be spooled for delivery (for whatever reason), allowing the sending MX to retry at a later time. There is absolutely NO excuse for a mailserver dropping mail like that.
This isn't a free webmail account, this is something customers pay for. Some people could have lost a lot of business. And what if someone has been searching for a job for the last 6 months and their monster.com etc contact info has only this email address?
My luck, the one email this chick got from me, was the one with me telling her off for not answering the other 9.
Wanna fight ? Bend over, stick your head up your ass, and fight for air.
2) I didn't know Cringely was still around.
If it hadn't been for the reference to GMail, I'd be wondering if this story had been sitting in the queue since 1998. Now, off to buy some LNUX shares, and one of those Tommy Hilfiger straps to hang my keys around my neck!
What I'm listening to now on Pandora...
I run an online retail business, and non-tech savy customers using earthlink don't get a lot of our email.
Biggest problem is that Earthlink uses a white-list spam blocking setup that sends back a time-limited challenge to the sender ("Please go to this link and fill in this form so that this user can receive your mail").
We get these challenges when our automated system sends messages to customers
- Roach
I used to work for a hosting company (a pretty shitey one at that). They refused to update their mail systems so all customers had email issues. They used an old customized version of SendMail on the Linux box's. The problem, the queues were getting full and either locking the mail up or raising the machine's processes up so it could no longer do anything. The solution, delete the mail off the server and kill all processes. This was a temporary solution that turned into a permanent one because "It worked." The problem was, the mail was gone when it happened so all the tech support guys had to hear the poo poo mad customers.
I would wager thats whats going on here and they don't want to admit it. There is some admin there (or it could be company policy) that see's alot of mail getting queued up but not being delivered but instead of fixing the problem he just deletes the mail and everything's fine.
Freedom is a state of mind. A mind is a state of being. Stay the fuck out of my mind and my being. - Corporate Avenger
This sort of thing is the reason I host my own e-mail. At least this way I usually know when it's broken, and I have the opportunity to fix it.
Technically that is against the ToS for regular Earthlink accounts.
Secondly they like to block a lot of traffic on email-esque ports.
Either way... As a former employee, I'm not surprised.
"I am the king of the Romans, and am superior to rules of grammar!"
-Sigismund, Holy Roman Emperor (1368-1437)
Generally when you're stating statistics like "up to 9 out of 10", that means that in at least one your test runs, whatever it was you were testing hit 9 out of 10 times, in this case the fact that Earthlink lost e-mails. You say "up to 9 out 10," because in other runs it may have been 1 out of 10, 3 out of 10, or 7 out of 10. You're trying to show that how bad it gets. When stating the statistic for useful statistical purposes, however, one should definitely also give averages, like: "overall Earthlink lost 50%". Now, in this case, Cringely actually states that his friend tried several times to send messages a block of 10 messages to Earthlink account, to an aliased Blackberry account, and to his Gmail account. Each time, Only 1-2 made it to the Earthlink and to the Blackberry (aliased off the Earthlink account), and all 10 made it to Gmail.
My blog
Last month or so, yahoo started bouncing email from earthlink. 100% of the time. Calls to support eventually indicated it was a known problem (didn't admit it until pressed), and then indicated multi-day wait for it to be fixed. It was easier to fix on my wife's side; reroute her mail through my hosting server. Though the advice from our 13 year old son was probably the best: "why aren't you using gmail?"
Have you seen the commercials? They've off-shored half their jobs to magic-fairytale land. They've probably got some under trained ogre for an email admin who stands around the water cooler all day chatting with the fairies. Sure, it's a lot cheaper when you can pay your staff in pixie dust, but you end up getting bitten by poor customer service (I had to call them the other day, and the customer service rep had such a thick elven accent I could barely understand him). Outsourcing just doesn't work...
Shameless plug for my photos on Flickr
Lets remember way back when this happened all the time shall we.....and we used to say to the users.....
"there is no guarantee of email delivery" (and optionally "Get over it")
Remember this folks, no where in the RFC's is there anything that states email will get delivered....
Just because all us sys-admins do such a great job, most of the time it does get there, people forget the dark ages of the internet when this would happen all the time.
OK 90% email loss is really really bad, and it use to be more like 5% loss (at worst), but people need to remember email isn't guaranteed.
To be losing mail, Earthlink servers must be accepting mail and then throwing it away, or at the very least, not continuing to forward it to the destination, which is just as bad. This goes completely against how the system is supposed to work. If they can't handle the load, there's a specific set of return codes to give (RFC821, section 4.2): I understand your perspective -- email is a loosely connected system, with lots of points of failure. However, in the vast majority of cases, a failure at one point will cause either delays or errors, not dropped mail.
I [may] disapprove of what you say, but I will defend to the death your right to say it.
Essentially true.
Though I wouldn't use the word "inept".
Try putting a couple hundred domains and 10k users on it and your threat surface for spammers goes up exponentially from a small server with a few domains and a couple hundred users.
Ours gets tens of thousands of bogus connection attempts from spammers per hour. How many are you getting? 50? That's not including the stuff that does get into the filters to be processed by the rules.
Until you have run a big box with lots of users on it, you have no freaking idea what we deal with on a daily basis. And it has gotten MUCH worse in the past 18 months.