Slashdot's Meta Moderation

So its ready for testing on The Meta Moderation Page. Essentially eligible users are presented with 10 comments the moderation done to them. They are then asked to rate the moderation as Fair, or Unfair. Click the link below to read more comments on this addition to our moderation system. There have been a few minor tweaks to the system since I last mentioned it.

The most confusing issue in M2 is context. We don't have enough of it. The comment is seen without its parent comment and its replies. Hell, we don't even see the story it was written in reply to. We don't know what the score was when the moderator moderated. All of these things are issues that I'll fiddle with resolving as I have time to tweak them around. When in doubt, feel free to leave the button alone and simply not moderate. I've found that usually its pretty easy to flag "Fair/Unfair" on half the comments, but sometimes I need more context: Then the parent link is helpful to see where the comment was coming from. When in doubt, leave it alone. I've found that only a handful of moderation is ever really unfair (maybe 2 or 3 out of my 10).

Some details on the system:

First is some limits. Certain circumstances will cause your M2 to be disregarded. This is to prevent abuses. I'm not going to release the specifics as that would defeat the purpose, but essentially, if you vote to heavily in either direction (like say if you vote that all 10 moderations were "Unfair") your M2 may be discarded.

Second is the addition of a karma bonus for meta moderating. There is a sliding scale of karma that might be "earned" for meta moderating. It won't be enough to ever earn you "the bonus point", but its something. Since you aren't eligible for M2 if you have negative karma, this can't be used to dig you out of the hole. Most of the stuff in place is designed to make abuse difficult.

Third is the finalized restrictions on M2 eligibility:

• Registered/Logged in User
• Non Negative Karma
• Hasn't M2 moderated yet today
• Account is in the older 75% of all Slashdot accounts (filters out the newbies)

These are basically the same restrictions as the normal moderator access, except that M2 is once a day, and moderation tries to pick from people who read an "Average" amount of Slashdot a day (weeding out occasional readers and obsessive overreaders)

Finally is an interesting idea that has been proposed a few times that I just wanted to throw out for discussion. As it stands, you only see the moderator controls when you have moderator points to use. What if the moderator controls were always visible, but when you submitted the form, they were only counted if you had moderator points. Oh, and you wouldn't know if you had points.

On one hand, people would burn out on moderating fairly rapidly (the randomness in the existing system means people tend to use their points) but on the other hand, only the diligent would actually be relevant. The only question is would the people obsessive enough to do it be honest moderators, or guys trying to push around their own agendas.

Side notes I've been adding a few bits to the FAQ trying to address a few of the, well, FAQs. So read the the guidelines FAQ for answers to exciting questions like "Why don't you give my points back if I post into a discussion I moderated" and "I think I should have longer than 3 days before my points expire". I've also got a few new things in the "Ideas" space that I'm currently mulling over.

Update: 09/07 04:19 by CT :Answering the old questions:

1. Simply creating dummy accounts won't work.... the accounts need to be older to be eligible anyway.
2. Duplicates are OK. You're M2ing an individual act of moderation- it just so happens that you got 2 moderations upon the same comment. (this is very likely in cases where comments are moderated in rapid succession which often happens on new/extremely bad comments.
3. I'll probably link it from the left hand menu. Else from the top of the comments. Maybe right on the homepage "You haven't MetaModerated Today". I'll worry about that later.

Comments

Where are the Unbiased Moderators?

[Anonymous Coward]

Moderators are for moderating content not authors' identities, right? As recently as Slashdot's article in May 1999 on the new terms of use at Deja.com, AC posts with excellent content were being moderated up to score 5 (my own AC post in that article was scored 5). Since late Spring, however, perhaps due to the unpleasant anti-AC bias widely expressed lately on Slashdot, there have been no top scored posts by ACs. As an example, this comment in this AskSlashdot article on Linux MIDI gives useful information, is highly relevant to the original poster's question and might deserve a correspondingly high score but so far it has score 0.

Signed, Permanent AC

Moderations...

[strredwolf]

A few comments...

Moderation notification: The current system is good, since putting up buttons by default for moderation when the user isn't being a moderator is going to put more load on the server, possibly crashing it again (AIEEE!!!!). Not a good idea.

Logged in but Anonymous This is worth it, but can be abused. But then, it's psudo-anomymous, so it has to be handled by users with care. Flamers? Kick 'em off.

Posting restrictions: A good writer would take about five minutes to write up a good post, with nessisary proof backing up the claim. A "Me Too!" post is under one minute. So does a cheapshot flame. I propose a one message per IP/user per 4 minutes time, with adjustments baised on Karma on the time limit (less time to wait for more Karma, more time for less). Do you really care about a subject enough to write a virtual essay, which would be moderated high, or some short statement which really doesn't add more to the topic and stays scored at one or zero? I'd say the most well written ones are ones in which some time is gotten into it, and it shows with the proof used to back it up -- and it takes time to gather that proof. The system is very tuned to how people write!!!

I wonder what my Karma is looking at now... Is there a Slashbox for Karma?

---
Spammed? Click here for free slack on how to fight it!

Meta-Moderation unnecessary

[Sinner]

I don't think adding extra layers of moderation is a sensible way to deal with bad moderation. I think it should be obvious from the way the complexity level is going through the roof that this is a bad idea.

Instead, I think the system should judge moderators by how often their moderations get overridden by other moderators. Specifically, a moderator should lose karma when a comment they have moderated in one direction is subsequently moderated in the opposite direction. Not a lot each time, because you don't want to allow vendettas or suppress differing opinions, but enough for the system to detect genuine abuses.

For this to work properly, there would need to be more moderation points in circulation. I don't have a problem with it if some comments get up to +10 or +15, in fact, it would help differentiate them.

Somehow I don't think CmdrTaco is gonna drop the M2 system he spent all weekend coding, but I thought it was worthwhile getting my idea out there anyway. :-)

too-visible moderation?

[Ryandav]

I really hate to say this, as I hate sounding negative, but I don't see why it's better to have the moderator controls always visible. I, for example, browse most of the time at +2 so I can just read the first 30 good comments or so. If I'm bored later or really interested in a topic i go to -1. If I get moderator access, I need to browse at -1 and see everything. While I enjoy moderating from time to time, I don't want to have to always browse at -1 _just in case I might be mod_... It seems to defeat the purpose: to allow those with limited time or attention span to see a story and read just a few good comments on it. You also begin to make it so that only a few people will participate in the moderating.

Other than the "bonus point" system for certain people, I really think the system here has been working out great. I hope "creeping featurism" doesn't take over it all.

I'll say it if no one else will..

[witten]

Am I the only one who sees all of this moderation stuff as incredibly silly and ultimately futile? Rob's getting himself into a never-ending arms race with the trolls. He adds moderation functionality to slashdot. The trolls fight back. He adds restrictions on who can moderate what, and when. The trolls don't give up. He adds karma points and moderation of moderation. My educated guess, based on slashdot's track record, is that the trolls keep it up.

This War on Trolls is a lot like the War on Drugs. At a certain point, the effort you must expend to eradicate that last 1% of perceived evil is so great, and so harmful to "society," that's it's really not worth it.

Having said that, I can't say that I blame Rob. As slashdot's popularity grew, the small community-based discussions turned into a frothing troll-ridden free-for-all. And he saw moderation as his only choice in trying to combat the trolls. However, in the long run, moderation is a losing proposition. Try to squelch the trolls, and they'll be that much more determined to screw with the system. Sure, the casual trolls might give up, but the small percentage of sick people who get off by insulting dead people they don't know.. will be back with a vengeance. And no reasonable amount of moderation will silence them. That is unless you're willing to silence many good, reasonable posts in the process. And that seems to be exactly what you're doing.

Sure, with all these new moderation features, most of the trolls get moderated down, but the price is that many non-troll polls are mismoderated, both up and down. And if the same general group of people are doing moderation and meta moderation and meta meta moderation, whether or not they're trolls, you're going to get the same exact sort of abuses and mismoderations and poor judgement across the board.

So what I'd like people to think about is this: At what point is the cure worse than the disease? When you keep piling up the rules and regulations and meta moderation and percentages and sliding scales and point ratings, what are you really accomplishing, in the long run? (If I want scads of arbitrary rules and magic points, I'll go play Nethack.)

I know this isn't really constructive criticism. However, I'm not intending it as flat-out flamage of moderation, either. I'm merely wondering, is this all really worth it? I don't have a better solution, but this ever-increasing moderation simply doesn't feel right. In fact, it feels futile. And harmful. But hey, I'm just one voice in the crowd.

And now due to the beauty of moderation, this post, which expresses a dissenting opinion, will be moderated right through the floor. And no amount of meta meta meta moderation will change that. Sigh.

Re:I'll say it if no one else will..

[LL]

witten makes some very good points
As slashdot's popularity grew, the small community-based discussions turned into a frothing troll-ridden free-for-all. And he saw moderation as his only choice in trying to combat the trolls. However, in the long run, moderation is a losing proposition. Try to squelch the trolls, and they'll be that much more determined to screw with the system.

Given its positive connotations (more==good), I suspect that the karma system will only exacerbate the troll problem as more people post to gain enough feedback (funny or otherwise) to achieve extra height on their /. soapbox. While advertisers would be estactic at the extra traffic, it makes it harder for the time-limited reader to pick out the above average comments. As economists have noted, this leads to cogestion effects (everyone talks a little louder ultimately making the whole room too noisy) while imposing a negative externality cost on other people (increasing time taken to moderate, then moderate the moderators, then ....).

Solving problems like this are difficult (think trying to balance a real community). As noted in an earlier post, you have to look very carefully at the social dynamics and add appropriate dampers or incentive structures. One rather simple solution is to given posters a chance to self-moderate and then reward those who moderators concur (ie negative karma if someone thought their post was funny but everyone else thought it was a troll). Then hopefully a reputation system like E-Bay would be encouraged through peer pressure to conform and improve the quality of posts which is (so far) definitely above the average newsgroup.

The other thing that may be of some use is to encourage very high karma posters to write wrap-up reviews so that late attendees can get the whole picture without wading through all 200+ messages. Ultimately I think being more creative in the the choice of categories and a descriptive filtering language (as other have suggested) will control the complexity but given the wide variety of suggestions Rob has so far, it might be worthwhile setting up some careful double-blind experiments, say selecting random threads to compare 2 moderation techniques then looking at the distribution of scores to make quantitative measurements of effectiveness. You know, that thingy we call the scientific method :-). As we say here, if you can't measure it, you can't manage it.

LL

A new revenue stream.

[Booker]

Is all this leading up to a new revenue stream for Slashdot? "Get your Official Cmmdr. Taco Meta-Kharma Decoder ring! Only $9.95!"

On more serious notes,

1) I think this is all looking really good, and
2) I don't want to see much more of it.

You can legislate forever, but you're never going to codify every possible violation. Try not to let the jerks force you into too many coding contortions, as you'll probably just wind up creating more loopholes, and make it more complicated for everyone. (Think IRS)

Microsoft Sucks!

[um... Lucas]

Now, if that comment was posted because of an article about how Bill Gate's personal wealth had surpassed a trillion dollars, that comment might be moderated as being funny.

If it were posted because Microsoft bought Redhat, then one might call it insightful

If that comment was posted in context of a comparison between IIS and Apache, it could be considered flmebait.

If it were posted in a discussion about Sony's robotic dog, then it'd be a troll...

Basically without having any context to the comment (at least give us a link to the story and the it's parent) this meta-moderation seems rather pointless. We're (or rather Rob is) spinning in circles trying to figure out what to do about trolls... Just like anything else that's bad in life, it'd be hard to be sure to be rid of them without hurting ourselves as well.

How about this:
Moderators get 10 points, rather than 5.

They can choose which discussions they wish to moderate: so long as they're a moderator, they cannot change their comment threshold above -2 (or however low it goes). Also have it not show other people's moderations to the moderators (everything, regardless, would appear as being posted by the Anonymous Coward with a score of 1.

They can click a button at anytime that removes their moderations, shows original posters, other peoples moderations, and lets them comment. Kind of how it works now.

Basically, I think moderators were doing a good job thus far with the tools that have been supplied. What we need is more moderators, and more "un-biased" moderators, with unbiased meaning that they HAVE to view every comment, and not know what other people have said about it.

Agreed - where's the context?

[Yosemite Sue]

I just tried Meta Moderation, and though you can reasonably guess that certain posts are interesting/informative (or trolls/flamebait) by looking at them, without a link to the actual story it is difficult to truly know the context of the post. I don't think I'll feel comfortable M-Moderating under this system.

Lucas's suggestion of increasing available moderator points makes sense, especially given the recent scourge of trolls that have been wasting moderator points. Much less complicated than Meta Moderation, too, IMO!

YS

non-moderator moderation

[crow]

Moderation is a public service, much like jury duty. Let's not drag everyone into it when we don't have to.

On the other hand, there could be a system whereby you would specify in your preferences to always have moderation buttons on. Then you could moderate even if you have no points, but the moderation wouldn't really do anything for normal users. For real moderators, though, they would see something like:

Non moderators recommend:
-5 off topic
-2 flamebait
+4 funny

Then, using that example, a real moderator might use a point to lower the score of the post. This action would clear out all the recommendations, or at least those that said to move the score in the same direction.

This might also eliminate the need for M2, as it would allow anyone to attract the moderators to articles that have been miss-moderated.

Re:An interesting idea

[LL]

David Gould wrote
That brings up an interesting question: I'll admit that I'm pretty interested in my "karma", and want to gain as much as possible, insofar as it's a measure of "acceptance" within the community, or some such warm fuzzy feeling. But is it good for people to become obsessed with this?

Depends on how much emotional attachment you have on external peer pressure. I have heard of reports (but seen no formal studies) that females apply a greater value on the social relationships/status with others whereas guys put greater emphasis on the accomplishments they have achieved. Karma (if you ignore the hip connotations) is just a metric that provides feedback on what are the qualities that the /. audience is looking for. Personally I think it is a good idea as IMHO it encourages more critical reflecting thinking which is in a serious shortage in this day and age of impulse thinking/buying. Others may seek "karma" for the perceived kudos assocaited with it but unless /. becomes a global phenomena, has zero value outside the community. Think of it as a localised social currency :-).

Seriously, it's amazing how fast the stories go by these days. I guess it's an inevitable result of Slashdot becoming so big, but it makes it a lot harder to participate in the discussions. If all the stories have over 300 comments by the time you get to them, all you can really do is read. If you take the time to write a really thoughtful comment, chances are hardly anyone will still be reading the story by the time you post it. This
is often my problem. I'm not sure how your idea would help with this problem, but I think it's an important one.

Again, I think you have to look carefully at the effect that Rob is trying to achieve (at least fgrom my observations). The goal is to gain a representative and comprehensive sample of the opinion-space of /. readers, encouraging greater marginal participation and attempting to float to the top good ideas/commentary. Think of the general problem as a genetic search across a global semi-professional audience, attempting to find the centroid (median consensus view) plus boundaries (interesting perspectives/angles) under a self-optimising strategy (moderation). (Aside ... if the politicians ever get ahold of this, they'll kill to get their ratings up ... maybe Rob should release it only for non-profit use). Thus you want to encourage comments to fill missing holes (if someone has already said something, just agree), discourage repetition and irrelevant gunk, damp down inaccurate information, and dynamically redefine the objective goal to achieve greater illumination and agreement/action.

The size of the submissions and the short time span of posts may be a problem if /. expands as you point out. Some possible solutions include splitting levels -1 to 1 into separate geographical and/or time zones, but merging higher level comments (a la multigrid iteration). It would be an interesting research project to find how such communications patterns can scale to thousands or even millions of comments/people and the resulting time/energy efficiency for disseminating problems and creating solutions.

LL

Re:Microsoft Sucks! and other Trolls

[trongey]

Ok, I've already got negative Karma from one of the 2 or 3 times I posted in the past so I might as well chime in here:
>...what to do about trolls... Just like anything >else that's bad in life, it'd be hard to be sure >to be rid of them without hurting ourselves as >well.

What you're implying here is that "trolls" are intrinsically bad, and that moderators really know what a troll is. I think that we are guaranteed to hurt ourselves if we try too hard to eliminate the trolls.

If someone says something that the moderators don't like then they become a troll regardless of whether there is any validity to the posting (I think this might be true in my case, but I could be wrong).

I work with kids in a program where we accept that any comment - however stupid or boorish - can be the catalyst that starts a creative train of thought. I just hope we don't miss something really good because of the troll barriers.

'I used to have a really cool sig that went here, but I can't seem it keep it working.'

You're doing this all wrong.

[FascDot Killed My Pr]

Hardcoded separation of levels is only going to lead to confusion and ugly code. Instead, why not do this:

For moderators only, show a history of moderation action for each comment. This could be a 2D chart with history being vertical and different moderation "levels" being horizontal. So if I became a moderator I'd see something like this:

Score: 2
| Insightful +1 |
| Troll -1 | Stupid Moderator -1 |
| Interesting +1 |
(I hope the sense of this comes through the formatting) Each "decision" can be moderated individually. A moderator is allocated a certain number of points, say 10. For a cost of 2 points each then can moderate a "base comment". For a cost of 1 point each they can moderate anything above that level.

On a totally different topic: Why are we spending time on esoteric oddities like meta-moderation when simple features like the below need working on?

• Spell checking
• Link checking
• Anti-/. Effect Mirroring
• Basic uptime issues
• Non-retroactive signature changes

---
Put Hemos through English 101!
"An armed society is a polite society" -- Robert Heinlein

Re:I must have missed something, but...

[expunged]

Karma is the sum total of all moderation that's done to you.

So if you post a lot of stuff that's moderated up, you get good karma. After +20, you start to get a bonus point every time you post (posting at +2 instead of +1). If you post a lot of stuff that's moderated down, you get negative karma. After -10, you start to get a negative "bonus" point (posting at 0 instead of +1).

So if you start posting with brilliance your karma can go up, and if you start flaming, your karma will go down.

-nicole

Karma qualities?

[homunq]

I don't mind reading the stuff that has been explicitly moderated up as "funny". However, there are a few folks who posted a lot of hair-trigger joke responses. That's great; sometimes they score a direct hit. But when they accumulate enough karma to get a default score of 2, all those jokes start to become annoying.

So, what's the solution? More feeping creatures. The question is, which ones feep the least?

-People's karma comes with the adjective that is the "mode" of all their moderation adjectives. Their default-moderation-level posts also acquire this adjective. In your preferences, you can explicitly give a -1 to posts with certain adjectives ("I don't *want* to be amused, dammit!"). Feeps pretty hard, because it requires additions to the already-huge preference screen.

-You can't attain Enlightenment (enough positive karma for a default upgrade) without a mix of adjectives. Someone with 20 "funny" posts wouldn't make it, but someone with 17 "funnies", 2 "interesting", and an "informative" would. Also highly feepish from an implementation POV, but at least pretty transparent to the user.

-Only a certain number of posts a day get your default moderation bonus. If you're posting more than once or twice a day, chances are you're not putting excessive thought/new information into each post. This idea is by far the easiest to implement and the cleanest. It would probably solve the problem, even though it doesn't address it directly.

Moderator controls always visible...

[weave]

Finally is an interesting idea that has been proposed a few times that I just wanted to throw out for discussion. As it stands, you only see the moderator controls when you have moderator points to use. What if the moderator controls were always visible, but when you submitted the form, they were only counted if you had moderator points. Oh, and you wouldn't know if you had points.

PLEASE don't do this. Whenever I get moderator points, I spend a helluva lot of time reading stuff at -1 and make a real effort to be fair and thorough. If this suggestion was implemented and I just appeared to have access, but didn't know for sure, I just wouldn't bother at all.

I hate to sound snotty, but I am a technical services manager, have a lot of experience in NT and Linux, have a lot to offer, but I do NOT have time to sit around and waste.

If you occasionally want me to moderate, fine. I will, I will take it very seriously, and I will be honored by the privilege.

But please don't jerk me around. It'll get old fast...

Thanks.