Slashdot Mirror
The New York Times Is Expanding Comments With the Help of Google's AI (recode.net)
An anonymous reader shares a Recode report: The New York Times says it is going to expand the availability of online comments from 10 percent of articles to 80 percent by the end of the year, without adding more moderators to its staff. How are they going to do this? With a machine-learning algorithm, of course. The Times today is rolling out a new structure of comment moderation using software from Google called Perspective, developed by the company's incubator, Jigsaw. The Moderator tool will automatically approve some comments and help moderators wade through others more quickly.
There is a link in the article to here where you can input comments that the system will judge to be 'toxic' or not. There is no sarcasm, irony and bullshit detection that I can tell, only a score that is generated by the combination of keywords used.
For example, "The cake is a lie" receives a 50% toxicity score, "The cake is bullshit" receives %90, and "There is no reason to believe the cake exists." is scored 3%. This system merely weeds out the laziest of trolls.
Washington Post allows comments on every article, that appear to be real time with excellent S/N ratio.
It's really easy to fool the system to let clearly offensive comments through. It's fooled by simply misspelling words that are deemed offensive, which essentially puts it on the level of Slashdot's l4meness filter (more on this later). Consider the following text, "I don't like [n-words]" that I can't even put in a Slashdot comment without triggering the l4meness filter. With the actual n-word, the Perspective API indicates that it's 87% likely to be perceived as toxic. However, replacing the i in the n-word with ii or a 1 lowers that score all the way to 13%. The same simple tricks that Slashdot trolls use to evade the l4meness filter also work to fool the Perspective API. If the m0deration is automated and people aren't reviewing the comments, it can easily be fooled in its current state.
Here's an experiment that I've been trying. Find any Slashdot article and paste in comments that have been modded up versus comments that have been modded down. Aside from the most blatant of personal attacks, it does a lousy job of identifying which comments are at -1 and have been deemed toxic by the human moderators here. The system can also be defeated by posting something offensive (like the n-word) followed by intentionally benign comments to lower the score. It reminds me of back in the day when trolls would add random text to the end of their posts to defeat Slashdot's l4meness filter.
I'm not convinced that it's any better than the very simple approach of just having a list of banned words. In fairness, it's early in development, but right now the system is a very complex way of implementing an easily defeated barrier against flame wars.
Automated systems are also prone to false positives. My comment is an example. Apparently saying the word "lameness" more than twice triggers the lameness filter. This comment is an example of the problem. I haven't been able to trigger a lot of false positives with the Perspective API, so perhaps that's an advantage.
Ironically, my captcha is "accepted."
I don't find the NYT comment section to be moderated for political views. Often at least some of the NYT Picks comments are what you'd call politically opposed to the Times' expected liberal bias. I have had some of my comments tagged NYT Picks that were explicitly critical of an article's journalism (lack of source diversity, bias, etc).
I do think that when measured in absolute terms, liberal comments dominate but that's mostly a reflection of their readership, but I think even the self-described liberals will often take the paper to task for dubious journalism judgement.
What I think is most pernicious about the Times is its frequent selectivity about what articles have comments enabled. I think the Times does this on purpose with some topics, and not merely because they attract trolls, but because they don't want people to question their narrative in that story specifically.
And I say this as a longtime subscriber, too. I think the NY Times is about as good as news journalism gets on a daily basis in terms of factual accuracy and intelligent reporting, but I do think they carry a very strong editorial bias. I wish there was a conservative version of the NY Times that wasn't filled with a ton of business reporting. I read FT, Economist and Bloomberg from time to time but find them thin on news that isn't business reporting.
You shit-eating moron! Big butts on hoes are da bomb!
And that would score 94% "toxicity." Or, the author could reflect a bit and write:
My good sir! Even a caprophagous rapscallion could determine the ultimate pulchritude of femininity, which lies most gloriously in lovely and great callipygian virtues.
And you'd score a mere 12% "toxicity," despite expressing a nearly identical sentiment.
I'm normally not a fan of trolls, but if a system like this could lead to, shall we say, more "creative" insults and "elevated" ways of expressing such matters, that might be very entertaining.
(Alas, I know this system isn't sophisticated enough to generate such an outcome, since it's easier to "break" than just using words with more syllables.)
Paid trolls, shills and bots are a real problem that pollute comments. Read comment sections like at the Washington Post, which is one of the few that still allow comments, and you will see that one or two poster with an agenda that follows up every insightful or informative comment with a short, one liner insult. You don't need a perfect insult detector in order to filter out most of the garbage out there, and if commenting sections such as the Washington Post is any indication, then it is absolutely needed.
That said, I prefer Slashdot's moderation system. I appreciate that a well deserved "fuck you" will not be automatically censored or considered 'toxic', but could be awarded the highest visibility as long as the message is appropriate.
It's meant as a shield, not a sword to go after people you hate.
I don't hate my abusers. I feel sorry for them. They could have done something useful with their life.