Slashdot Mirror
Massive Email Crash Hits Canadian ISP Shaw
rueger writes "One of Canada's biggest cable/Internet providers has their customers in an outrage. '... after an interruption of Shaw's email services Thursday led to millions of emails being deleted ... About 70 per cent of Shaw's email customers were affected when the company was troubleshooting an unrelated email delay problem and an attempted solution caused incoming emails to be deleted ... Emails were deleted for a 10-hour period between 7:45 a.m. and 6:15 p.m. Thursday, although customers did not learn about the problem until Friday, and only then by calling customer service or accessing an online forum for Shaw Internet subscribers.' To top it off, when Shaw did send out notices about this, they looked so much like every day phishing spam that many people deleted them unread."
Who?
Are one of the reasons I don't use ISP hosted email. Main reason is portability.
Except when the failing account is that forwarder account.
There will be more mail tomorrow.
The details are that the messages were never delivered in the first place, your setup would not protect against such a problem.
The right to protest the State is more sacred than the State.
isn't a holding-bay?
Shaw is probably the least abusive of the canadian major telecom companies. I've been a shaw customer for 14 years and this is the only incedent I've had other than lines being blown down in a storm. My wife's email was effected but mine was not. This is a normal (and rare) human error... most of the actual abuse telecom companies dish out is abusive contracts and misleading advertizing like 3-year cellphone contracts and "Optik TV and Internet" ... which is actually satellite and DSL, not FTTH.
Like, ones that make a backup before messing with critical data? As an elementary precaution known to anybody halfway competent in IT?
This just demonstrates a massive, massive management screwup, as they allowed unqualified personnel to work on their systems. Save a buck, loose a million.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
From TFA : "The mistake was an “isolated event,” Lakshman said, and promised a detailed review, which would include a discussion about compensation."
Except it isn't. Few years ago I had a business's domain email hosted with Shaw (was included with the internet service and they provided IMAP), and they lost all of it. They wouldn't return my calls about it, and on the third time I called in a week or so later I was told it would not be recoverable, that there is no backup for their business email service at all, instead they would credit the account ~5 days of internet service. I was floored, but was too busy to get into it with them and I had our email backed up so I just moved on with a email hosting provider I felt more confident about.
Shaw's internet service has been decent, but I wouldn't trust them as anything more then a data pipe.
Yeah, messing up is normal. Failing mark a snapshot before becoming with a million emails is incompetence. With a snapshot, human error might have resulted in losing three minutes worth of emails.
"To err is human, to fuck up the whole system requires root."
Also, gmail exists.
Haven't we all fantasized about just deleting the goddamn queue and going home?
I would say Distributel is less abusive, similar prices, runs over Shaw lines but internet is unlimited - no overage fees. Teksavvy used to be good too, not sure if they still are.
Website Just Down For Me? Find out
BTW: Gmail provides IMAP and POP access, which is a stumbling block for those who want a desktop email client. I'm not sure about Yahoo or Hotmail.
I'm sorry, I don't follow your logic. How is providing the option for POP and IMAP -- in addition to webmail -- considered a "stumbling block"?
How do you know?
Making mistakes is human. This is why a competent professional acknowledges that he will make a mistake sooner or later, and designs his activities so that mistakes in execution won't have catastrophic consequences. These guys failed to properly do this.
...if your email is not in at least two physically separate places, you are at risk of losing all of it, forever.
It's weird Shaw can't restore from a backup - the article is a bit weird on the exact details about what happened and just ends with "the emails were not backed up".
If your online mail provider does not allow you to access or export your data to your own PC (via IMAP, POP, or whatever) then you should switch to one that does - and start backing up your own email if you want to be more confident that it's going to survive catastrophes.
The details are that the messages were never delivered in the first place, your setup would not protect against such a problem.
That's true.
How bad this is depends on the system -- in this case it sounds like Shaw was doing "accept, then drop" which is the worst case because no one is notified of the failure. If however Shaw rejected spam rather than accept it, the sending mail system would notify the sender that the message was not delivered. It should be noted that this latter solution also does not cause backscatter because it doesn't generate a bounce. [For a bounce to occur, the message first needs to be accepted, but then for some reason cannot be delivered.]
I work for TELUS, and can assure you that optik tv is NOT a satellite service. It is an IPTV service being delivered over either ADSL 2+ or VDSL connections (and sometimes fibre in new areas).
Satellite tv is offered in areas where we don't have the broadband infrastructure to support standard Optik TV. However we refer to that as TELUS satellite TV and not Optik tv.
I don't believe TELUS ever claimed optik was FTTH. The optik name refers to the fact that it is served by our new fibre network. The "last mile" is still copper in most neighbourhoods though (past the DSLAM essentially).
Minor part of your post i know, but thought i would help clarify since you seemed to be confused on what optik is.
Cheers.
http://www.telus.com/content/tv/sat/
http://www.telus.com/content/tv/optik/index.jsp
http://en.wikipedia.org/wiki/Telus_TV
-----
This post reflects my own views and do not necessarily represent the views of my employer.
Damn, isn't there anybody here but me who has been locked out of their gmail account for about 2 weeks now? I have not changed a thing in my fetchmailrc or mailfilterrc's, and have been sucking my gmail account dry at 3 minute intervals with fetchmail for damned near 5 years.
2 weeks ago, both fetchmail and mailfilter started reporting password failures. It worked about 30 minutes a day for 5 or 6 days, but has not worked since the last week of February.
I call them up, get some yahoo whose command of English sucks dead toads through soda straws, he leaves to go get someone who speaks English, but the next guy isn't a hell of a lot better, and he finally speaks clear enough that he is telling me the account is blocked because my machine is compromised. I object, its a linux box, behind a router running DD-WRT. Doesn't make squat to him, my machine is compromised.
Seeing as how everything that comes in here has to run the clamav gauntlet, and that this is a linux machine which has not had java enabled anywhere near firefox in months, currently at V-19.0.2, AND that its behind a router running DD-WRT, AND neither chkrootkit nor rkhunter can find anything to complain about, I seriously doubt it has been compromised.
I had been gradually weaning my mailing list activities, moving them to other servers precisely because of their no dups policy, so that was all the impetus I needed to just move all my subs. I still scan them on schedule just in case they actually get someone who reads english wondering why a fetchmail instance is failing to login, telling fetchmail the password is toast when its the same pw I've been using for years, and its long enough John didn't get it in 6 hours of grinding on it when I last checked with john the ripper.
Until that happens, screw gmail, and the camel that rode in on them.
Cheers, Gene
"To top it off, when Shaw did send out notices about this, they looked so much like every day phishing spam that many people deleted them unread."
Erm. No they didn't? I'm looking at one right now and it doesn't look remotely like 'every day phishing spam'. It doesn't offer me anything, threaten me with anything, or ask me to click on anything. It doesn't include any links except to a forum thread, which the text doesn't make any special effort to make you click on. It didn't trigger my mental 'phishing detector' in the slightest.
I got the email notification late Saturday, two days after the event happened, I guess. That's not a horrible delay. I also saw a bunch of delayed mails come through around that time - 10 or so - and they notified me of the sender and subject line of three mails that were lost, so looks like they managed to recover quite a lot.
I dunno, I guess I'm not TOTALLY OUTRAGED at this. As another commenter said, you know, admins screw up sometimes. Lord knows I have. The fact that they're at least able to identify the subject lines of all the lost mails makes a big difference; you could get any really vital ones re-sent.
MS Exchange lowered the bar. Yes I know it's supposed to do a dozen other things but it's MTA was crap for years and still seems to generate a lot of panic on sysadmin mailing lists.
While it's utterly trivial to alias everything incoming (or even outgoing) to another host that's another bit of infrastructure and often seen as an unnecessary expense. Their backups will be system files, whatever is in the mail spool at any given day is beneath their care factor and anything that arrives after the last backup is gone anyway.
Remember this folks before considering outsourcing, it's not their email so they don't care about it as much as you do. While you may want to keep stuff in two places they are not going to bother to go to the extra expense unless it's to their advantage.
Fibre To Aardvarks?
You're a temporary arrangement of matter sliding towards oblivion in a cold, uncaring universe
You have to log in with the gmail interface and answer a captcha. Then your account's back on.
Shutting down free speech with violence isn't fighting fascism. It IS fascism!
The emails were received and accepted, but then deleted. There is nothing your MTA is going to do about this.
"Hot lesbian witches! It's fucking genius!"
But, if anything happens, It's my own fault. I don't have to trust my ISP to do anything but provide the pipe.
I would never trust exchange as a relay. Everywhere I worked where I had the power to do so, Exchange did not sit in the DMZ, and relayed through proper unix mail servers. I prefer sendmail, because I am familiar with it and know how to properly extend and secure it. Use postfix if you prefer, but again, I'd never trust a Microsoft Exchange mail relay.
I've also been on Shaw since pretty much continuously since the first month they offered service, and I live in one of the first cities activated. Yes, about 14 years. Haven't used their email or web services since the first year.
I had one incident with Shaw which was very annoying. An OpenBSD firewall just suddenly stopped working with no change on my part. If just the firewall accessed the internet, it worked normally. But as soon a NAT client relayed traffic through the firewall concurrently, responses from the Shaw head-end would cease to arrive, for about 30 seconds. If I had ping running, I'd seen 30 ping packets go out, then all of sudden 30 pink packet responses, then maybe a few other packets, then another 30 second hiatus. Who knows, maybe I had something unorthodox in my pf configuration about handling all the background arp chatter. I had only ever aspired to "works for me" with my pf configuration.
The Shaw technician determined that the problem was customer premise equipment by showing that routing my service into a Mac with no firewall present worked just fine. The network trace showing their head-end buffering 30 seconds worth of ping responses and then blurping them back in a packet noogie didn't strike him as a hinting toward an anomaly with their own administration.
This had happened once before for a week or so, and then suddenly cleared itself up with no intervention on my part. The next time it was permanent.
I didn't feel like fighting with them or with messing with my firewall configuration, so I ordered Telus as a backup, and that worked perfectly with my firewall without changing anything. While I had both services, I observed that Shaw is fundamentally superior. You don't see this in data rates (not often) but you do see this whenever you're surfing the web with a big download running in background. Telus gets very chunky. I was banned by a family member from downloading anything on our Telus connection during a remote session to the office. Shaw has tiny lurches, too, but you almost don't notice them. This whole problem, likely having something to do with buffer bloat, has become progressively worse (not better) over the last 14 years, with the biggest uptick in chunkiness right around the time Netflix became popular.
Telus is also a vastly more irritating company to deal with. Don't even get me started, I could go for a week.
Shaw is no angel, but over the term of my experience, they've about as enlightened and as reliable as any ISP on the planet. There's no such thing as an ISP that never pisses anyone off.
However, by some miracle of economics, I'm now paying more for essentially the same service than I was 14 years ago. I was a heavy user then. Good grief, I downloaded 100 megabyte service patches over dial-up the year before Shaw offered broadband. Now that 100MB patch is 700MB ISO, so there has been usage inflation, yet hardly outstripping technological progress. Somehow in the telecoms industry, economies of scale run contrary to every other field of economic endeavor.
The Shaw email outage is a brutal error, but I wouldn't trade Shaw for 90% of the other ISPs out there, not without a gun to my head. This is easier for me to say having the wits to set myself apart from ISP email services 13 years ago. This also made it easier to switch pipes when I did experience my Shaw difficulties.
Note that my PF problems went away when I rebuilt my ruleset from scratch on a FreeBSD server that replaced my old OpenBSD firewall, when time permitted me to mess with this.
I'm sure it was a case where something unusual in my configuration triggered a bug in how the service was configured on their side. Shaw is not the kind of ISP that digs into anomalies even if you shove the packet trace right in their face. Maybe after this email thing boils over, they'll get religion on pursuing those small anomalies people were noticing a week before one final fault routed all their received email into the giant bit bucket.