EarthLink Is Losing a Lot of Email

LandGator writes "Robert X. Cringely, doyen compu-columnist for PBS, reports on a hidden e-mail problem at Earthlink: They're losing up to 9 messages out of 10, found as a result of a friend's testing." From the article: "He sent messages from other accounts to his Earthlink address, to his aliased Blackberry address, and to his Gmail account. For every 10 messages sent, 1-2 arrived in his Earthlink mailbox, 1-2 (not necessarily the SAME 1-2) on his Blackberry, and all 10 arrived with Gmail. Swimming upstream through Earthlink customer support, my buddy finally found a technical contact who freely acknowledged the problem. Since June, he was told, Earthlink's mail system has been so overloaded that some users have been missing up to 90 percent of their incoming e-mail. It isn't bounced back to senders; it just disappears. And Earthlink hasn't mentioned the problem to these affected customers unless they complain."

Comments

They DID notify customers

[Anonymous Coward]

However, it was via email.

Re:They DID notify customers

[chazd]

I am currently woking for the United States Antarctica Program in Antartica. My email is a vital link between Antarctica nad my life off the ice. Earthlink has seriously dropped the ball. I have had so many problems with their webmail it has made me almost scream. I now have to tell people if they really have to get ahold of me they have to use my yahoo account. I sincerely hope that earthlink gets the feedback through lost revenue because they have not shown any interest in fixing the problems. I have strong thoughts for any company that stands between me and my family.

"Nothing for you to see here..."

[Atlantis-Rising]

Is earthlink hosting slashdot?!

Lucky for Earthlink Customers!

[Timtimes]

Less spam. Enjoy.

DIY

[tverbeek]

This sort of thing is the reason I host my own e-mail. At least this way I usually know when it's broken, and I have the opportunity to fix it.

Re:DIY

[vertinox]

This sort of thing is the reason I host my own e-mail. At least this way I usually know when it's broken, and I have the opportunity to fix it.

Technically that is against the ToS for regular Earthlink accounts.

Secondly they like to block a lot of traffic on email-esque ports.

Either way... As a former employee, I'm not surprised.

Re:DIY

[jandrese]

I'm surprised you can do this. Most ISPs block port 25 traffic that isn't sent to or from their own badly overloaded email servers. The more breathless among us might even claim that spam is outright killing email as a communication tool.

Re:DIY

[Ciarang]

Maybe I'm missing something, but the Exchange thing struck me as an extremely odd assumption to make.

A couple of comments though - I admin a few Exchange Servers. For the most part, they take literally zero effort to maintain. They're very well behaved. Perhaps I'm doing something wrong?

On the other hand, it's definitely unnecessarily complex and bloated for 'home use'. Someone wanting to run a DIY mail server on a Windows box could do a lot worse than to take a look at the very clean, compact, GPL'd Hmailserver - www.hmailserver.com

Re:DIY

[towerdave]

Hmmmm, I have been running SBS2003 at home for a few years now with no problems, including Exchange. No MCSE here, although I'd love to be able to get one now that they aren't dumbed down. I'm not saying it's the perfect tool for the job, but it works fine for me.

I used to be in Tech Support for MindSpring, and remember the Netcom mail fiasco. This sounds like it could be worse. Glad I'm not there anymore.

TowerDave

Re:DIY

[cayenne8]

"Technically that is against the ToS for regular Earthlink accounts."

So...get a business acct. I had one with Cox..was great. Static IP...no caps on uploads or downloads...could run all the servers I wanted, and a low level SLA. The one time I had trouble, I called the support line..rather than put me on hold, I left a msg. with my problem. In about 3 min..I had a tech call ME back.

It was only about $70/mo....I'd highly recommend it.

Re:DIY

[jafiwam]

Essentially true.

Though I wouldn't use the word "inept".

Try putting a couple hundred domains and 10k users on it and your threat surface for spammers goes up exponentially from a small server with a few domains and a couple hundred users.

Ours gets tens of thousands of bogus connection attempts from spammers per hour. How many are you getting? 50? That's not including the stuff that does get into the filters to be processed by the rules.

Until you have run a big box with lots of users on it, you have no freaking idea what we deal with on a daily basis. And it has gotten MUCH worse in the past 18 months.

Re:DIY

[techno-vampire]

As a former employee, I'm not surprised.

Same here. Ever since they closed the Pasadena Call Center and dumped about 2/3 of their most experienced employees the quality of service has been dropping. It used to be that you had to understand what was going on in order to work there. Now, all you need is the ability to read scripts. You don't even need to be able to tell when the scripts don't apply, or when to ask somebody that knows what they're doing for help. Just read the script and don't care if it works or not.

I must say that this is just typical of the new EarthLink mis-management. Do anything that increases the short-term bottom line and to hell with the long-term effects. EarthLink has fallen into the hands of the MBAs and doesn't have long to last.

Re:DIY

[walt-sjc]

If you don't want shit service it's going to cost you. The old saying "you get what you pay for" applies. If you don't need to run servers, have a static IP, or a better TOS, then don't pay for them. If you do, then it's not free. Obviously the market supports the extra cost.

But back on topic here, earthlink has had email problems for YEARS. The only difference now is that the problem is worse than it was and more people are noticing.

Lost e-mail? WHAT THE HECK?

[matth]

E-mail should never be lost! We had an issue where I work with e-mail BACKING UP for a few months while we implimented new mail servers... but no mail was ever lost.. it either got bounced back (not usually) or would arrive several hours after it was sent. To actually LOOSE e-mail indicates that Earthlink is ACCEPTING the mail and then DUMPING it!!!! When our servers were overloaded, we just rejected the connection, until the mail server could handle more mail.. and then we accepted it.

Re:Lost e-mail? WHAT THE HECK?

[the linux geek]

Exactly. At the very least it should be saving the email and sending it later, not just ignoring it. It is not that complicated; it's cetainly technically feasible. Of course, this is Earthlink we're talking about. I used to have these guys as my ISP before cable became widely available, and the system for connecting was a mess (If you tried to use Outlook instead of Outlook Express, for instance, it would require uninstalling/reinstalling their client software.)

Re:Lost e-mail? WHAT THE HECK?

[matth]

Actually we handle close to 50,000 e-mails per minute.... I happen to work for a largeish ISP in Pennsylvania :)

Re:Lost e-mail? WHAT THE HECK?

[gregmark]

E-mail should never be lost! ... To actually LOOSE e-mail indicates that Earthlink is ACCEPTING the mail and then DUMPING it!!!!

This is absolutely correct, so any policy checks that occur during the SMTP handshake (who are you? where are you coming from? who do you want to send to? how much data do you have? Oh, do you now? REJECT). However, anti-spam and anti-virus checks happen after the message is accepted. If the result of the check is X, and policy rules say drop mail on the floor when X, then bye-bye e-mail and sorry Bond, the government will not ackowledge its involvement.
Otherwise, the only way to loose mail is to shutdown a machine with a heavy queue and throw out the disk. SMTP is impervious to network badness. My money is on an SMTP policy run amok.

Re:Lost e-mail? WHAT THE HECK?

[parkrrrr]

"Loose" is clearly spelled with two Os. "Lose," on the other hand, is spelled with one.

Re:Lost e-mail? WHAT THE HECK?

[scatter_gather]

The extra "o" is for emphasis. The more "o"s, the loster it is. Glad I could clear that up for ya.

Re:Lost e-mail? WHAT THE HECK?

[silas_moeckel]

If your antispam and antivirus are running after you have accepted the email your system in broken. This is all companies playing fast and loose with the RFC's you should never accept mail that you are not going to deliver. This really is not that hard to run spam and virus checks as milters (or whatever your email application does) or place front ends that do so.

Re:Lost e-mail? WHAT THE HECK?

[SanityInAnarchy]

However, anti-spam and anti-virus checks happen after the message is accepted. If the result of the check is X, and policy rules say drop mail on the floor when X, then

Then it's a horrible policy.

Every single email service I've signed up for that does spam filtering has a "spam" or "bulk mail" or "junk mail" folder. I implemented my own when I deployed my own personal server.

And virus scans should be able to remove the virus and tag the message as "WHOOPS THIS HAS A VIRUS", but shouuld not drop them on the floor.

If this is causing undue stress, you could implement policies in the handshake. "Oh, I've gotten 100 emails from you in the past hour, and 90 of them were spam/viruses. REJECT." Or put it in some sort of tarpit.

An email service losing email is somewhat like anyone losing data these days. There is absolutely no situation where email should completely disappear into an email server and never come out.

Re:Lost e-mail? WHAT THE HECK?

[Anonymous Coward]

It's a troll because mail wasn't delayed for a few months (as you and the retarded troll seem to think), but there were a few months where, during heavy times, mail was delayed for an hour or two.

When you move out of your parent's basement, you might realize that it is perfectly reasonable for the turn-up of a new mail service to take several months of planning, execution, and testing to ensure that no mail is actually lost when you are dealing with 50k messages an hour.

Re:Lost e-mail? WHAT THE HECK?

[parkrrrr]

One might note that, but one would be wrong. The "no apostrophes for plurals" rule doesn't go on vacation just because it's a single letter.

There are style guides that disagree on this point, granted, but the ones that disagree with me are self-evidently wrong.

What MTA do they use?

[CoolVibe]

I checked their mailservers (what the MX record reports anyway), and they have a very generic ESMTP banner, not really apparent which MTA they use. I want to know which MTA can lose mail because of overload. So I can avoid it like the plague. I do know for a fact that Sendmail and Postfix send a 4xx error if mail cannot be spooled for delivery (for whatever reason), allowing the sending MX to retry at a later time. There is absolutely NO excuse for a mailserver dropping mail like that.

Re:What MTA do they use?

[yourlord]

Earthlink uses an SMTP server they developed in house because they felt the existing servers out there would not scale to the level they needed and were not secure enough.

I used to work there back in the day.

Re:What MTA do they use?

[Anonymous Coward]

At the time Earthlink wrote their MTA, they were 100% right about the scalability thing.

These days, good old postfix is more than sufficient. Or qmail, if you can stand it -- it works for Yahoo, who deals with more than a little bit of email.

Wonder if they can be sued

[kalirion]

This isn't a free webmail account, this is something customers pay for. Some people could have lost a lot of business. And what if someone has been searching for a job for the last 6 months and their monster.com etc contact info has only this email address?

Re:Wonder if they can be sued

[olyar]

From their policies section

3. THE SERVICE Depending on the type of Service that you choose, the Service may include internet access, software, hardware, email, webspace...

...and further down...

THE SERVICES ARE PROVIDED ON AN "AS IS" AND "AS AVAILABLE" BASIS. EARTHLINK DOES NOT WARRANT THAT THE SERVICES WILL BE UNINTERRUPTED, ERROR-FREE OR FREE OF VIRUSES, OR OTHER HARMFUL COMPONENTS. So while it sounds like its horrible customer service, it doesn't sound like they're in any danger of being sued.

Re:Wonder if they can be sued

[Thraxen]

Well.. let's think about this. They have had the problem since JUNE. That's a damn long time to be losing up to 90% of a customer's emails. On top of that, instead of notifying all their customers of the issue, they only tell them if they call to ask. So how is a customer supposed to know that some of their e-mail is missing? It's hard to know about something you never had. The only way you would likely figure it out is if e-mail you expected to receive were repeatedly not making it through. Even then, the first couple of times you would likely figure that the person/company that was supposed to be sending the e-mail simply never did.

The example to which you responded is a perfectly valid concern and is an example of the type of e-mail a customer would never know is missing. They really deserve some sort of litigation. A simple notice sent out to all your customers back in June would have allowed their customers to switch any important accounts tied to that address before much was lost. But now, SIX MONTHS LATER, a lot of people have likely missed out on a lot of important messages.

She loves me not

[Joebert]

My luck, the one email this chick got from me, was the one with me telling her off for not answering the other 9.

Is this 1998?

[Otter]

1) I didn't know EarthLink was still around.

2) I didn't know Cringely was still around.

If it hadn't been for the reference to GMail, I'd be wondering if this story had been sitting in the queue since 1998. Now, off to buy some LNUX shares, and one of those Tommy Hilfiger straps to hang my keys around my neck!

White list spam block with challenge

[BrianRoach]

I run an online retail business, and non-tech savy customers using earthlink don't get a lot of our email.

Biggest problem is that Earthlink uses a white-list spam blocking setup that sends back a time-limited challenge to the sender ("Please go to this link and fill in this form so that this user can receive your mail").

We get these challenges when our automated system sends messages to customers ... but there's simply no practical way to respond to them all within the time limit (during business hours it's not practical - it's impossible at 2am while you're sleeping). And unfortunately it seems that a lot of their members simply don't understand what it means when you tell them they have to add you to their list or they won't get your email.

- Roach

Re:White list spam block with challenge

[JavaManJim]

Hey Brian,

Your problem could be your users or it could be you.

USER LAZINESS Suspect email to the user sits in Earthlink's SuspectEmail page for a default of 14 days. Earthlink users can easily go that page to move email and add you to their address book. However checking a web page takes energy and users might not do that very diligently.

MULTIPLE EMAIL ADDRESSES One problem with some businesses email is they constantly change their internal email address. For example NRA (National Rifle Association) always winds up in my Earthlink suspect email. Below are two NRA examples.
NRA_Endorsed_Insurance_Programs.UM.A.2.4767@www.nr anews.org
NRA_Endorsed_Insurance_Programs.UM.A.2.4687@www.nr anews.org

I use Earthlink for email. Earthlink spamblocker is practically my home page because I check it all the time.

Thanks,
Jim Burke

When asked about the email problems...

[clickety6]

... an Earthlink representative said that he had received no notice of any email problems. Next question please.

The problem, of course, is...

[captnjameskirk]

...their tubes are clogged!

I bet I know whats happening...

[aliendisaster]

I used to work for a hosting company (a pretty shitey one at that). They refused to update their mail systems so all customers had email issues. They used an old customized version of SendMail on the Linux box's. The problem, the queues were getting full and either locking the mail up or raising the machine's processes up so it could no longer do anything. The solution, delete the mail off the server and kill all processes. This was a temporary solution that turned into a permanent one because "It worked." The problem was, the mail was gone when it happened so all the tech support guys had to hear the poo poo mad customers.

I would wager thats whats going on here and they don't want to admit it. There is some admin there (or it could be company policy) that see's alot of mail getting queued up but not being delivered but instead of fixing the problem he just deletes the mail and everything's fine.

They must have hired the BOFH

[TheWoozle]

and he just piped the e-mail to /dev/null. All that e-mail was taking up too much disk space, anyways.

Fix the Blog Post Link

[RockoW]

The url for the blog entry is: http://kiloseven.blogspot.com/2006/12/got-earthlin k-got-mail-no-they-lost-it.html

Re:Says a lot..

[morgan_greywolf]

Generally when you're stating statistics like "up to 9 out of 10", that means that in at least one your test runs, whatever it was you were testing hit 9 out of 10 times, in this case the fact that Earthlink lost e-mails. You say "up to 9 out 10," because in other runs it may have been 1 out of 10, 3 out of 10, or 7 out of 10. You're trying to show that how bad it gets. When stating the statistic for useful statistical purposes, however, one should definitely also give averages, like: "overall Earthlink lost 50%". Now, in this case, Cringely actually states that his friend tried several times to send messages a block of 10 messages to Earthlink account, to an aliased Blackberry account, and to his Gmail account. Each time, Only 1-2 made it to the Earthlink and to the Blackberry (aliased off the Earthlink account), and all 10 made it to Gmail.

Re:Says a lot..

[Southpaw018]

Parent is +4 insightful? Wtf? The quote specifically says the average was 8-9 lost. Parent is implying that maybe only 1 was lost for the most part, or something similar.

C'mon, can people not even be bothered to read the article SUMMARY any more?

Not only losing incoming email

[pmuellr]

Last month or so, yahoo started bouncing email from earthlink. 100% of the time. Calls to support eventually indicated it was a known problem (didn't admit it until pressed), and then indicated multi-day wait for it to be fixed. It was easier to fix on my wife's side; reroute her mail through my hosting server. Though the advice from our 13 year old son was probably the best: "why aren't you using gmail?"

Re:Says a lot..

[Mr. Underbridge]

I turn green up to 9 out of 10 times I take a shower

Can I watch?

Here's the root of the problem

[vmxeo]

Have you seen the commercials? They've off-shored half their jobs to magic-fairytale land. They've probably got some under trained ogre for an email admin who stands around the water cooler all day chatting with the fairies. Sure, it's a lot cheaper when you can pay your staff in pixie dust, but you end up getting bitten by poor customer service (I had to call them the other day, and the customer service rep had such a thick elven accent I could barely understand him). Outsourcing just doesn't work...

No guaranteed email delivery

[martin]

Lets remember way back when this happened all the time shall we.....and we used to say to the users.....

"there is no guarantee of email delivery" (and optionally "Get over it")

Remember this folks, no where in the RFC's is there anything that states email will get delivered....

Just because all us sys-admins do such a great job, most of the time it does get there, people forget the dark ages of the internet when this would happen all the time.

OK 90% email loss is really really bad, and it use to be more like 5% loss (at worst), but people need to remember email isn't guaranteed.

Re:No guaranteed email delivery

[soccerisgod]

Just because all us sys-admins do such a great job, most of the time it does get there, people forget the dark ages of the internet when this would happen all the time.

Back then, you may have had an excuse. Today, the excuse that the RFC doesn't specify email gets delivered should get you fired for being a failure who doesn't give a shit. Just my $0.02.

Re:No guaranteed email delivery

[TheJasper]

Remember this folks, no where in the RFC's is there anything that states email will get delivered.... This is what I thought when I read the headline (having used that excuse in the past ;). So I immediately looked up RFC2821 - Simple Mail Transfer Protocol

6.1 Reliable Delivery and Replies by Email

      When the receiver-SMTP accepts a piece of mail (by sending a "250 OK"
      message in response to DATA), it is accepting responsibility for
      delivering or relaying the message. It must take this responsibility
      seriously. It MUST NOT lose the message for frivolous reasons, such
      as because the host later crashes or because of a predictable
      resource shortage.

      If there is a delivery failure after acceptance of a message, the
      receiver-SMTP MUST formulate and mail a notification message.

naturally there is alot more, including cases where it is acceptable not to send a notification, but I don't think any apply here.

So basically, SMTP is defined as a reliable protocol which guarantees delivery or notification of failure. The days of unreliable e-mail no longer apply.

Re:Says a lot..

[hey!]

Usually the phrase "up to" is used to make the data sound like it says more than it does. It literally means "possibly as much as but no more than", but people read it as "on average". Using "up to" to make a claim that sounds broader than it really is is, as you imply, dishonest.

In this case what they have is an estimate of the average. But it's not based on much data or systematic testing. By saying "up to" they are actually encouraging the reader to interpret their results narrowlly. They could have said "9 out of 10 times", but by saying "up to 9 out of ten", they imply (a) they have no evidence that the system performs worse than this and (b) the system may at times perform better than this.

Overall, I think this is an example of honesty.

Re:Powweb sometimes does not forward email.

[wordsnyc]

pair.com. Ten years, no problems.

Bullshit

[ZombieSquirrel]

Something stinks here. This article does not have a lot to stand on. "A friend's testing"?! How scientific is that? And anyway, Earthlink is not exactly a fly-by-night operation. Don't you think more people would have noticed if 9 out of 10 of their emails were disapearing since June!? No way. This is crap. I have two earthlink accounts and I haven't noticed anything. Maybe his "friend" is just an idiot. Maybe Cringely is just an idiot I have nothing for or against Earthlink, I just hate bad information.

So? No guaranteed US postal mail either ....

[King_TJ]

But if they started losing 9 out of every 10 letters you mailed, how long would you keep using them?

Re:Semantic Man!

[value_added]

Technically, we do not know that the email was *lost*, nor can we ever know.

Seeing a black sheep in the field does not prove that every sheep is black, nor that there is at least one black sheep. All you can prove is that there is at least one sheep that is black on at least one side.

Donald Rumsfeld, is that you?

As An Earthlink Customer

[mpapet]

I have no problems. Plenty of spam and the good stuff comes through for me and my wife.

Believe it or not, they've been very good. The one issue I've had they resolved quickly once I was escalated above first-level tech support.

Maybe it's a location-specific issue?

I can qualify another post that talked about the new email sender verification thing. I get it sending mail from the web email interface. But none of my friends or the emails I send myself from work require sender verification.

I don't know what the motivation behind these complaints may be. It certainly isn't bothering me or my wife. Maybe it will..

(shrugs)

Re:As An Earthlink Customer

[N7DR]

Ditto and likewise. No problem here at all, and I would definitely notice it (and have noticed it in the past, within less than an hour of a problem occurring). So maybe Earthlink is losing a ton of e-mail on some accounts, but it seems to be working fine for others. I really can't believe that the problem can be widespread. People may be sheep, but surely if Earthlink were losing any noticeable fraction of most people's e-mail, they would have suffered a mass defection.

Re:As An Earthlink Customer

[Enzo the Baker]

From the article:

The two groups affected are those who get their mail with an Earthlink-hosted domain and those with aliased e-mail addresses like my friend's Blackberry.

So it probably doesn't affect the majority of their users who are not doing anything fancy.

Re:Semantic Man!

[DoorFrame]

There are known emails and unknown emails, and those divide into the known unknown emails and the unknown unknown emails. THOSE are the ones you've got to watch out for.

SMTP Digression

[Spaceman40]

I think you have an unrealistic expectation of the quality of service you can expect from email.

I wrote an SMTP server a while back (to check out Ruby's network libraries), and while going through the RFCs I found that there are expectations of quality that include delivery of something. It all comes down to the protocol: if a server accepts a message, it takes responsibility for the message's delivery. A server should reject the message if it cannot deliver (causing the delivery server to either try again later or tell the user there's a problem).

To be losing mail, Earthlink servers must be accepting mail and then throwing it away, or at the very least, not continuing to forward it to the destination, which is just as bad. This goes completely against how the system is supposed to work. If they can't handle the load, there's a specific set of return codes to give (RFC821, section 4.2):

450 Requested mail action not taken: mailbox unavailable [E.g., mailbox busy]
550 Requested action not taken: mailbox unavailable [E.g., mailbox not found, no access]
451 Requested action aborted: error in processing
452 Requested action not taken: insufficient system storage
552 Requested mail action aborted: exceeded storage allocation
553 Requested action not taken: mailbox name not allowed [E.g., mailbox syntax incorrect]
554 Transaction failed

I understand your perspective -- email is a loosely connected system, with lots of points of failure. However, in the vast majority of cases, a failure at one point will cause either delays or errors, not dropped mail.

More Info via Earthling

[anearthling]

I write Earthling, EarthLink's blog. I spoke to Stephen Currie from our email team this morning and published some more information from him on the issue over on Earthling.

Mail problems at Earthlink - Insiders view

[Anonymous Coward]

As a recent "former Earthlink employee", I worked directly on the mail systems. Some of the problems with email are related to the lack of adoption of common anti-spam practices. For example, on reader pointed out the white-list blocking with challenge response emails, those challenge emails are numerous that the IP's that generate are often times blacklisted by large email sites like Yahoo! and AOL. At the time I worked there Yahoo! was suggesting we use domain keys on our inbound mail to verify that it truely originated from Yahoo!, instead of blindly sending millions of Challenge-Response email to their users that would be labled as spam on their end. Another piece of the 'missing-mail' puzzle is the inbound anti-spam methods employed. One piece, is a script that counts the number of connections from remote MTA's, and if they reach a certain threshold, instructs all the MX'ers to null route them. This particular system seems to work well to combat spam, but when it fails, the entire inbound mail farm is brought to it's knees. This may account for some non-scientific research to claim 90% mail loss, but these emails are typically temp-fails. But, I've moved on to bigger and better shops, and they may have changed this since I last saw it.