Slashdot Mirror
Gmail Spam Filter Changes Bite Linus Torvalds
An anonymous reader points out The Register's story that recent changes to the spam filters that Google uses to pare down junk in gmail evidently are a bit overzealous. Linus Torvalds, who famously likes to manage by email, and whose email flow includes a lot of mailing lists, isn't happy with it.
Ironically perhaps, it was only last week that the Gmail team blogged that its spam filter's rate of false positives is down to less than 0.05 per cent.
In his post, Torvalds said his own experience belies that claim, and that around 30 per cent of the mail in his spam box turned out not to be spam.
"It's actually at the point where I'm noticing missing messages in the email conversations I see, because Gmail has been marking emails in the middle of the conversation as spam. Things that people replied to and that contained patches and problem descriptions," Torvalds wrote.
Or if the other comment's got hit by spam filters
Individual that differs more than 6-sigma from the population's mean has trouble with automated tools designed for the average person.
Gmail's spam filter is why email is still useful.
Well, Linus doesn't *HAVE* to use gmail. There are other email providers.
Welcome to the 21st century, Leeeenooooooos!
It is pretty much a given that the FP rate would go up as filters become more ubiquitous. This is how the spammers are winning the spam battles when people place too much faith in filters.
As I have said before, spam is an economic problem. We won't solve the problem with filters, or with any kind of punishment (legal or otherwise); we need to look at this rationally as an economic problem.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
...have his filters set up properly. Every mailing list has been grappling with this problem for the last year or so and gmail users have had to adapt to properly defined filters that prevent marking list messages as spam.
I've noted an increment of spam on my Gmail account.
From one email every few week to a couple per day.
Patches and problem descriptions for a kernel are gibberish to the lay person, and a SPAM filter is more stupid than a lay person. I hope.
Individual that differs more than 6-sigma from the population's mean has trouble with automated tools designed for the average person.
Exactly. I use Gmail and I honestly haven't had a false positive (flagged as spam when it isn't) in over two years. I still get the occasional false negative (spam that isn't flagged) at a frequency of a few per week. It's good enough that I don't even bother to routinely check my spam filter. It also is pretty good on the training - once you've spent a little time telling it what is spam and what isn't for you in my experience it is pretty good after that. Frankly if you have to check your spam filter often it isn't a very good spam filter.
I suspect Linus has rather unusual email requirements. Perhaps Gmail isn't the ideal solution for him. Very few tools are perfect for everyone. I'm a little surprised he's having that much trouble but stranger things have happened.
more at 10
Apparently he can't afford to purchase decent email service. Maybe someday he'll create something important and then he can get off the crap freemail.
It's mostly old people. They see something "in writing" and they think it's true. That's why they get bombarded by the "FWD: FWD: FWD: FWD: FWD: FWD: ..." crap and forward it to other old people. And when someone sends them some "offer" via spam, they think it's true. And many think that the government regulates the internet. And those are the folks who have all their marbles. There are quite a few elderly people who just don't have their brains working right.
When I was working a consumer hotline a couple of years ago, the folks complaining about being ripped off by spam were always old people. They were also the ones got ripped off the most by those infomercial products that clean colons and lose fat with just a pill.
Every sane spam filter returns three possible results:
When you have three categories it will reduce the FP rate very hugely. And the most important fact is that a spam filter should never throw away spam. It can be illegal to do so, or at some time it is going to be illegal. You should keep all your spam for documentation, for this reason. Also, you can initialize (learn) your filters very quickly when migrating the system. Spam is a valuable resource.
You can tell this is old news given that it was this past weekend the Linus posted an update on Google Plus stating that false positive rates were back down to normal for him.
https://plus.google.com/+LinusTorvalds/posts/dJdkRxUCRmK
I'm surprised to see this wasn't submitted by Bennet.
That's what you get for unleashing your half-baked Git solution on the world. I've used it every day for two years and it is just a horrid mess. How many times I've had to do a "reset --hard" because a branch just shits itself for no good reason.
And what if I want to bring up two separate branches side-by-side to do some copying? Can't fucking do it in Git.
Linus, bless you for Linux. But curse you for Git. Git is Shit.
He posted about it in G+, a googler noticed and offered to look into it. One day later The Register is feeding off the echoes and the story is slashdotted.
"Much better now.
Of the 100+ messages caught as spam over-night, only two were false positives (and I reported them). My email is getting back to normal."
https://plus.google.com/+LinusTorvalds/posts/dJdkRxUCRmK
On the next day, Linus wrote "My email is getting back to normal."
https://plus.google.com/+LinusTorvalds/posts/dJdkRxUCRmK
Maybe someday he'll create something important and then he can get off the crap freemail.
Yup. Given his past success with both Linux kernel and with GIT distributed source management, I too think that out of anyone Linus Torvalds might be the only guy able to effectively solve the SPAM problem.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
I'm sure NSA has a copy. All you need is to fill out a FIOA request and interrogate Michael Hayden until he admits it
>80 column hard wrapped e-mail is not a sign of intelligent
>life
Somebody should tell Linus about this great new operating system I run at home. I have sendmail running on my machine, and it lets me control my spam filters and everything.
It's called "Linux". I highly recommend it.
Slow news days huh?
I for one am extremely shocked that the above post ('use other providers') be flagged as funny.
Torvalds is the last person I'd imagine registering an email address @ Google.
Wise as he may have been, sorry, but to me he's a moron now just because of this.
I just hope I won't evolve his way when getting older.
Herve S.
From his original post, there is a clear date he claims the FP rate to have gone up... so this isn't a blanket Gmail FP rate issue, but rather a Gmail or spam blacklist incident, which is quite different from what the summary would suggest. As of right now:
http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3aLKML.ORG&run=toolpage
lkml.org Added to UCEPROTECTL2
Uceprotectl2 Automatically Delists Entries
This blacklist does not offer any form of manual request to delist. Your IP Address will either automatically expire from listing after a given timeframe, or after time expires from the last receipt of spam into their spamtraps from your IP Address.
Uceprotectl2 Accepts Payments Or Donations
This blacklist does support a manual request to remove, delist, or expedite your IP Address from their database upon Payment or Donation of fees to their organization. Please note the following; 1) MxToolBox does not in any way advocate the paying of removal from any blacklists. 2) Removal requests that are submitted without addressing the core problem will likely result in your IP Address being relisted in the database which can cause subsequent problems and extended listing periods without release.
More information about UCEPROTECTL2 can be found at their website: http://www.uceprotect.net/
Reason for listing - Net 146.185.176.0/21 is UCEPROTECT-Level2 listed because 36 abusers are hosted by RCN-ASN - Reality Check Network Corp./AS46652 there. See: http://www.uceprotect.net/rblc...
UCEPROTECTL2 seems a bit shady, but I am not blacklist expert.
Also as a side note, any spam filter that attempts context evaluation has a tendency to mark emails with code or special character formatting as spam. Even emails with links. So for someone like Linus to have a higher blanket spam FP rate is also not surprising.
The best gmail feature is the "never treat as spam" filter.
I just checked my spam folder again. I reserved a U-Haul, and the email confirming that reservation went to spam. One of the few false positives I get, but there are others.
What's interesting is all the political fund-raising emails. Only the conservative ones end up in Spam (Campaign for Liberty, Conservative Senate committee, Scott Walker, Rand Paul, Jeb Bush, Mark Mix, etc.). That's fine - I don't really want to see those emails anyway. Yet, I get lots of the same emails from Hillary Clinton, Obama, Harry Reid, common good VA, Virginia Dems, Webb 2016, etc, and NONE of those left-leaning groups or politicians get sent to Spam. 0. That seems a little odd - I didn't do any filter training based on that. So does Google's spam filter have a political leaning?
"Somebody has to do something. It's just incredibly pathetic it has to be us."
--- Jerry Garcia
Spam is getting annoying the other way around - ie. my servers / emails get marked much more easily than they used to, even though they are legit.
I receive a gazillion notifications and reports from my servers: my servers get marked as spam.
I use a remote server to send a low volume of notification messages to paying customers : my servers get marked as spam.
So yeah, we beat spam, sort of.
Moreover, any time I access gmail via ipv6 (I have dual-stack) my messages are marked as spam and I receive terrible warnings about someone else trying to break into my e-mail, and that despite using the same system and browser and the IPv6 whois records pointing to the very same ISP and town as the IPv4 ones. (The funny thing is that they don't tell me that, but that someone is trying to log into my email using "Linux" (as if that wasn't the case everytime).
It seems that despite their pretensions of achieving consciousness and singularity with their AI and stuff, their day-to-day operations are based on 'trusty' cargo-cult more typical of some neighbourhood network run by highschool dropouts.
If you don't check your spam folder than you miss mail. If you do check your spam filter than it's a pointless feature as your wasting time checking it anyway. How many people can afford to miss an important message? Grandma died? Your boss wants you to be in at 5AM or your fired for some emergency? Your kid got hit by a bus on the way home from school? A huge $100,000 deal caught-up by spam?
Yea.. I have a spam filter. It's called myself. I also can't stand OTHER people's filters. If I send you an email and your using some overzealous filter and my mail doesn't get through to you your on your own. At best I might alert you to the fact, but that's about it. I want something from you or you from me and you have an overzealous spam filter I'm going to assume your ignoring me and being rude. Because that's what your doing when you use such overzealous spam filters. I also refuse to "click that stupid link that says "Please click the link if you want your mail to go through". Sorry. No. If you want my mail you'll have to do the work yourself.
Hell, our server was blacklisted because we were on the same IP range as another server that was running a Tor relay. Not a Tor exit node, but a Tor relay. AND even if it had been an exit node it wouldn't have made any f'ing difference. The default exit policy restricts mail so you won't see spam coming from an exit node so its utterly pointless to refuse mail from it. If you do get spam from it chances are its cause that exit is infected, and had nothing to do with being an exit. Blocking all exits or relays is stupid and pointless. It has NO positive effect on reducing spam.
So he doesn't run his own email server but uses gmail? Wow
I have the same experience. For many months now, Gmail has been overzealous in marking stuff as spam. Stuff like daily emails from servers I manage with log digests. Emails about pending security package upgrades. Even when I specifically say that a certain subject string (e.g. "logwatch") is to be excluded, Gmail ignores that rule. It has been very frustrating trying to exclude stuff via filters in Gmail.
2bits.com, Inc: Drupal, WordPress, and LAMP performance tuning.
I had the same problem. I had to make a new email rule that would just send everything to the inbox and let me deal with it from there. Googles spam filter settings blow ass. Why can't I jut white list email addresses from the spam folder. Also, Google if you cannot get this right, please let me just turn off your shitty spam filter instead of having to hack my way around it.
automatically reject email failing its SPF or DKIM checks. If it's forged, by definition it's spam.
What is probably causing Linus's problem is the open subscription nature of the LKML. I'm guessing that a lot of people are subscribing, then flagging messages as spam rather than deleting them. Once a certain threshold is hit, that and similar messages are then flagged as spam.
Was helping a guy having trouble posting to our LUG list the other day.
He had DSN's being delivered to his GMail spam folder. I thought, "golly, how does Gmail figure those could be spam?" Nobody is going to sneak a Viagra ad through underneath a 550 report.
Of course his other problem was he was using Comcast as an outbound relay. Their new relay retries a message once every second five times and then gives up forever. Totally breaks greylisting, or even temporary outages. Didn't even try my backup MX.
But I'm running RFC-complaint secure servers, so those two big bozos can go pound sand. Eventually they'll fix their game. If a LUG member, of all people, can't be bothered to spend $2 a month for quality email hosting (assuming they just don't want to do their own), I'm not going to cry about it.
Free services are only guaranteed to be worth just as much as you pay for them. This falls squarely into the "not my circus, not my monkeys" category.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
I said, "NT", dammit
First up lkml.org is a third party site that hosts Linux kernel mailing list archives on a website. Regular Linux kernel mail isn't actually sent from it (I believe that's done by vger) so we're looking up the email reputation for the wrong IP...
Secondly UCEPROTECT is a very aggressive blacklist which states upfront they will block people who they believe are in the vicinity of people who the judge to be sending them spam. It's not the be and end all though and on one server I looked some time ago it's effectiveness was surpassed by other blacklists (here's someone else's old DNS blacklist comparison for 2014). In general I prefer more conservative tools like senderbase when trying to work out an IPs mail reputation.
For what it's worth I've also seen GMail incorrectly mark mails sent to the fio mailing list (which is also managed by vger) as spam and in that case it was purely down to mail being proxied through the list which was a place that didn't match the sender's DMARC records. Most of the time GMail was getting the marking of spam right though (even for mailing list mails)...
But nobody could every remember the name of the command.
And the Gnome guys promised to make a userfriendly GUI to help against that.
But they are still arguing about how to make it follow HIG, and how to adapt to the upcoming GTK4.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
Linus whines about everything, and everybody makes a big deal out of it. Who cares about his thoughts on the spam filtering, really?
A normal person would just adapt, maybe swear a little, and be done with it. Linus uses his email likely far outside of the norm which Gmail programs for. He should probably be running his own mail server.
Not everything Linus has to say matters. He's an average dude who did something really cool with Linux, but that doesn't make him a deity.