Slashdot Mirror
Moderation Ideas
First Some Comments on M2 and Karma As a whole it seems to be working pretty well. Some people are really irritated, other people like it, and some people simply are fretting over their karma way to much. Let me just remind everyone that Karma isn't that important. You should expect to see some karma come and go since your activity is subject to both M1 and M2 moderation. As an aside, M2 moderation will not push you out of the -10...10 range (in other words, being a bad moderator will never cause you to get the Comment Penalty). The only thing karma affects is eligibility for moderation participation.
I really wish I didn't feel obligated to display peoples' karma on their user info page. It tends to be a distraction. It's not supposed to be an integer determining anyone's value as a human being, it merely functions as a sort of 'Risk Factor' for moderation related activities. High Karma means you are a low risk, negative karma means you shouldn't be given tons of moderator points and turned loose on the world ;)
"The Problem" As I see it, the major problem on Slashdot is bad people gaining moderator points. The M2 system is one way to counteract that: An unfair moderator is (theoretically) weeded out. They are still free to post and participate as always, but they can no longer moderate if their karma drops below 0.
"The Assumption" I'm an optimist. I assume that there are more good people than bad people in the discussion groups. I assume that the good people will participate, and help weed out the bad ones. I also assume that eventually there will be world peace and harmony amongst all races and creeds and religions and genders. I'm young and an idealist here, gimme a break ;) As a whole, I think this is true: I've Meta Moderated many times, and I rarely find that more than 20% of moderation is bad. Many moderations are questionable, but rarely do I read one and think "What a moron". But they are there, and I want to try to make it happen less.
"The Solution" On an abstract level, the solution is to restrict the amount of power that any single user has within the system. Certain restrictions are in place already: moderators occasionally get 5 points, they and they lose them after 3 days, can't post and moderate the same discussion etc. Each of limits is designed to reduce the risk that any given person can screw with the system.
Each of these limits suck for a good moderator. This is given: the naughty people force rules to exist for the good people. If those 84 year old blue haired ladies who drive on the yellow line at 20MPH with their left turn signal on weren't around, maybe we wouldn't have speed limits either (Note: I also tend to drive 5-10 MPH under the speed limit and regularly forget to turn off my blinker, so I am part of the problem ;)
"The Idea" The idea is inspired by M2 moderations removing of the ability to choose what you M2 moderate. (This is flawed of course because you can reload and get 10 new comments but I'll fix that eventually). It also is a pain because you have no context for many of the comments. By not choosing which comments you can moderate, the risk of doing something inappropriate is greatly reduced: eg, when Mr. Bad Moderator gets 5 points he waits until GNOME article appears and then moderates all the pro-KDE comments down.
The solution is to reduce the probability that Mr. Bad Moderator can moderate a comment that pushes his agenda. This is what the point limits and the time limits are designed to accomplish today, and what this idea is designed to accomplish should we decide to implement them.
So here are the ideas:
- Put 5x (Numbers are Variables here: Don't complain about them!) as many moderators into the system.
- When you have moderator access, only display moderate controls on say 20% of the comments.
- Increase the percentage of moderatable comments based on karma.
It makes sense: negative karma users would never moderate, and more karma would give you more choice in your moderation. It would cripple good moderators because they might see a bad comment but be helpless to moderate it, However since there are 5x more moderators, the odds are someone else will get it. Side benefits include more people given "a shot" at moderating and more people will be allowed to contribute. If the M2 stuff works, the bad moderators will lose karma and no longer be allowed to moderate, and the better moderators will be given a larger percentage of all comments to moderate.
Another Sort of unrelated idea:
- Integrate M2 into the comments display tree: this solves the context problem that the existing metamod page has. Simply randomly select the occasional moderated comment for M2 moderation. Plus its much more "Real Time" and becomes less of a secondary chore. Plus it allows more people to participate since M2 moderation has much less restrictions on it. The downside is that this might be a pretty serious performance hit. I haven't really thought through it yet, but its an interesting thought. Comments on it?
It seems like ratings are getting very bi-polar around here. Posts that are a little bit better-than-average are spiking out at +5, while ones that are just slightly sub-par are getting -1's. Why? Because moderators are interested in showing how much they agree with the other moderators, in hopes of increasing their karma.
I gave this some thought yesterday while I was mowing my lawn, and I came up with:
That's just my dollar/50. BTW, I've been a moderator a few times recently, so I'm not taking potshots at a different group of people.
Dewey, what part of this looks like authorities should be involved?
Yeah. I hate to be elitist about this, but it seems pretty reasonable. I might also suggest that humor points not be counted for karma purposes- while I enjoy reading funny posts, the authors of said posts (which are much more likely than a good, serious post to get a 5) really shouldn't get so much karma credit for it.
Keep up the good work, Rob- someday, people will study your code to see how to make internet conversations that work for large numbers of people. I do believe that this is the first time anyone has seriously tried to make anything like this work- and the quasi-experimentation that you have done will be a source for the next group that tries to apply this technique to more serious fields of endeavour.
IAAL,BIANLY
Rob,
Please accept my humble criticism of slashdot's comment system. I have been thinking about this system a lot lately and designing a replacement for it. I have no delusions of replacing slashdot: slashdot is a community that would be hard to duplicate. The only problem I have with slashdot is the comment mechanism, and I will present those problems and some possible solutions.
Problem: Moderation race conditions. Joe and Jane are both moderators reading a slashdot article. They both see a comment that they think is interesting, but not incredible. They both choose to spend one moderator point to raise the article to "Score: 2 (interesting)". They continue to read the comments and click the "moderate" button a few minutes later. Now the article is rated 3, but neither of them thought it was worthy of 3. Moderation has failed for both of them. One of them could choose to moderate it back down to 2, but then the article would be rated "Score: 2 (overrated)", which does not reflect their true opinion of the article.
Solution: Instead of incremental ratings (+1, -1), allow the moderator to score the article on a range, say from 0 to 5. Then if two moderators assign a score of 2, the article's score is 2, not 3.
Problem: Only a few people have a few moderation points at once. This leads to several worst-case failures. One of them is that no moderators will read a given article, and the comments in that article will be clustered around 0 and 1, making the "Show highest scores first" setting much less useful.
Solution: Allow all readers to be moderators all of the time. There will never be a deficiency in the number of moderators with this method. The number of people scoring articles will be directly proportional to the number of people reading that article. When 100% of the participants can cast votes, there is no outlying possibility if lack of moderation.
Problem: The community ranking does not allow for individual preferences. The individual may have a strong opinion for or against a certain author, but the score given by the moderators is unlikely to reflect his preference.
Solution: Use a network of preferences to tailor the rankings that a user sees and to delegate the ranking of articles.
I believe that the ideal system would allow the user to cast a vote on every comment posted. The comment display system would use the user's voting history to score and sort the comments on every new article. Take an example:
Joe, John, Jack, Jerry, Jane, and Jennifer are all users. In the past, Joe has given articles by Jack an average rating of 4.2 (out of 5), and has given John an average ranking of 3.8. Joe has consistently rated Jerry at 0. Joe, John, and Jerry all have overall ratings (the average rating given by all the system's users) of 2.
In a new article, Jane, who has never posted a comment before, posts an unusually insightful comment. John, Jack, and Jerry all post their usual stuff. All three of them rate Jane's article at 5, but the overall community ranking for Jane is only 3. When Joe reads the comments for the article, the article display system will give John and Jacks votes more weight than the community vote, because Joe has rated those two authors highly. The system will completely ignore Jerry's votes because Joe doesn't think much of him.
The result is that Jane's article (along with John's and Jack's) are listed at the top of the page, instead of lost in the middle of the pack. Other users with completely different preferences will get completely different displays. When there are no personal votes for an author or article, the community ranking can be used to score the article.
Whew, I hope that example was clear. It really isn't very complex or even original. It is just like what firefly was doing 5 years ago, but with a little less complexity and a lot more scalability. If slashdot were to implement a reputation management system like this, I think it would be a much better place.
Regards,
jwb
It stikes me, that moderators get 10 M2 points per day but merely about 5 regular mod. points per month(!) to do their job in the first place. Using 98% of the pontential ressources for control and merely 2% for the actual work might be acceptable when handling nuclear weapons or biotoxins, but is IMHO total overkill when it comes to managing voluntary moderator work for an online forum, esp. since the exactly same information as provided by the M2 scheme could easily been extracted by better analysing comment moderations. Here's how:
Hypothesis 1: The majority of moderators are fair and take thir job seriously.
Hypothesis 2: If one moderator rates a certain comment up, and another moderator rates the same comment down, it is reasonable to assume, that they would equally inclined to consider the other's decision as unfair.
Conclusion 1: The ammount of mutually nullifying moderator decisions on a certain post provides a measure for the total 'unfairness' p of a comments moderation, where p = 2*min(#_of_ups,#_of_downs). If hypothesis 2 is true, then this p should be proportional to the karma penalty generated by the current moderation scheme.
Conclusion 2: Since we assume that there are more good moderators than bad ones, it is reasonable to assume that in a contoversial comment moderation the majority has a higher propability to be "right" than the minority (provided that there are more than 2 moderation decisins involved)
Proposed Solution: In the case of a controversial comment (i.e. a comment with up *and* down votes), generate a (very small) karma penalty proportional to p and distribute #ups/(#ups+#downs) of the penalty among the down-voters and #downs/(#ups+#downs) among the up-voters.
Example: A comment gets 4 votes up and 2 votes down. Each up-voter gets 1/12 (33% total) and each down voter gets 1/3 (67% total) of the total penalty which ammounts to 2*min(4,2)=4 penalty points (which should IMHO be somewhere around 1/5 to 1/10 of a current karma point)
In order for this system to work, it would be necessary to increase the total ammount of moderation points floating around and it might be reasonable to give some karma reward for each moderation to compensate for the total penalty generated by this system (so that in the end only the guys who regularily make minority decisions take the punishment).
Of course, the used formulas are only a proposal and fine tuning could certainly increase the overall efficiency of the system, once empirical data is available.
IMHO, the 'Overrated' and 'Underrated' moderation items should have the effect of MetaModeration. These are, essencially, means by which moderators can correct one another's work.
-- What you do today will cost you a day of your life.
First off, MM is not really necessary.
/. being a self-correcting mechanism already, they just add another layer.
:) I, for one, would rather see Rob bring his considerable talents to bear on extending the article discussion forums. On many occasions, a great off-topic discussion has been sparked by an article. This discussion either flops for being off-topic, or dies when the article expires from the main page. It would be nice to have some quantifiable worth of such a discussion, that would enable it to live on until it dies a natural death.
A good comment is good. It is unlikely that a single, rabid individual, will sink a good comment into oblivion, without an equally determined benefactor raising it up again. By that same token, a conspiracy of evildoers to elevate each other's trash, whenever one of them has moderator access, is slim to none.
Occam's Razor applies, and a system of lesser complexity is preferable to one of greater complexity. Slashdot should not hamstring itself with rules, and rules about rules, else it will become a burocracy in which nothing ever happens.
Personally, I've never used MM, but I would rather see it go away - simply because Moderation points strike me as a rare commodity, so their negative effect seems to be outweighed by their utility to the holder. Also, with
MM is an interesting exercise for Rob, it nothing else, and as such it is as useful as anything that keeps a mind from being idle.
As for a redoux of the moderation philosophy, how about fractional moderation? Increase the number of moderators, certainly, maybe by a factor of four. But reduce the worth of each assignable point by that same amount. This way, several moderators would have to agree to elevate a post to the next higher integer score. The ability to abuse moderator privilege goes away, as does the confidence that a single vote of one individual will make a lot of difference.
As moderators, we would then add our voice to a comment, not knowing if we're the first, or the decisive ones to raise the message status. Follow?
This way, good posts will rise, bad posts will sink, and iffy posts will tent to teeter-totter on the edge of where they started.
Ah, all this opinionated rambling has tired me out. But, before my hands completely cramp... How's about making all posts require a Preview??
-- What you do today will cost you a day of your life.
I think that it is a bad idea to restrict moderation excessively. When I have moderator access, I change my prefs to browse at low scores, and look for new comments (i.e. ones that probably haven't haven't been moderated yet). That way comments that deserve to be moderated up get there. Comments that are at 3 or 4 don't need that much help.
However, once I'm done moderating, I switch the preferences back to a more filtered view, and go back and re-read the articles looking for the content, as usual.
I think it is *essential* that moderators be allowed to pick which comments they moderate during the 3 days they have their points. Giving moderator access more or less frequently based on karma is a perfectly good idea, and could achieve some of the same effect.
Finally, I would like to just throw a reality check out there. Slashdot should be good, but there is *no* way to weed out all the a-holes, flamers, trolls, etc. Slashdot already does a very good job of tagging their comments so that people like me who don't generally want to see them don't have to. I think that, at this point, the moderation system is sufficiently effective, and time would be better spent squashing some of those SQL problems, and implementing some of the great ideas for additional forms of content that could be added.
-Cheetah
I *really* like the idea of integrated M2 moderation in the comments section.
I can't tell you how many times I've thought the moderators-in the context of a comments thread, were either clueless or not paying attention or just plain dumb. However, with the current M2 setup, I don't understand the context and don't really want to take the time to go look. I am already taking the time to do the extra moderation.
When I'm not a moderator, it would still be nice to feel I am having some sort of role in the moderation process-even if its just keeping a check on the moderators.
Problems I foresee-Someone moderates a comment, and immediately the moderation is moderated (jeez, i should choose a new word) to be Unfair. So does the moderator go back and change it? Is that unfair pressure? And if you M2 a comment in a thread, are you unable to participate in that discussion?
I like the idea of Karma. Though I think it would be cooler to see it on a scale when presented on the user info page. I pay quite a bit of attention to this site, but I still don't remember the scale offhand. A graphical bar would be nice. And instead of just numbers, relate the the numbers to real world behavior. ie. karma of -1 is called 'troll' or something like that. Be creative!
-Lisa
As a moderator, you should have the ability to moderate where it is needed, and not have your moderating ability limited to randomly selected comments.
If a moderator sees a particularly outstanding comment, they should be able to moderate it, and not be hindered by a random 20% factor. If you are going to increase the number of moderators, do so at the expense of moderation points, not at the expense of the ability to moderate.
And I do have to agree with Rob: if you don't like the system he's worked so hard to build, you can get around it without asking him to take it away.
The M2 thing seems to be going nicely, but I would like to propose an additional idea. When you moderate, you're limited to having very few subjects that detail how you feel about the message. Sometimes they don't do justice to your feelings about the post. That's why I think that it would be nice to have a one line comment detailing why you think the moderation you just made was fair. This is something that I have actually wanted in the past, and it might even help with the M2 moderation.
Writing a comment would be optional, as many moderations are simply self explanatory. But, it could be useful to say "I moderated this down because it was repeated above in comment #xyz," or "I moderated this down because we aren't discussing foo here, only bar." Of course, positive feedback would be good, too. Good posters could be rewarded with feedback like "Good post," or "Nice job researching this subject."
This information could then be made available on the M2 moderation page, permitting people to get an idea about why a moderation took place. A comment that might look perfectly normal could have been grossly off topic. Further, it would give the original poster an idea about why he was moderated up or down. Everybody wins.
Of course, the moderation comments would have to have limits. Perhaps the complete history of the moderation comments would only be available to the original poster, while everyone else only saw the most recent. This would prevent silliness like whole discussion threads on why a post was moderated in such and such of a way.
Anyway, it's just a thought.
"There is no surer way to ruin a good discussion than to contaminate it with the facts."
Although I do appreciate all the time you've spent coding in all this crazy moderation stuff, rob, I am somewhat concerned about some things.
A large portion of comments deserve no more or no less than the 1 (or 0 for AC's) that they get.. when you only display moderation controls on only part of the coments, almost all the comments with controls won't really be deserving of a moderation up or a moderation down. I think when this happens, some moderators, esp. the new ones will be just "itchy" to spend their points on something before they go away, and will moderate all the posts they can, regardless of whether they are deserving. (I may be wrong on this though)
What I feel might be a better solution than only having the moderation controls on a few of the comments.. Have a lot of moderators with only 1 or 2 points so your exampled Gnome bigot won't be able to moderate down all the KDE comments he sees , yet the "good" moderators will still be able to put the deserved -1 on those "Let's petrify nitrozac and hump her leg" posts.
Currently, everybody sees posts that are ranked on the same scale: what the moderators think. I don't agree that this is (always) the right thing to do.
/. based on their preferences.
/. (and the linux kernel dev list, and comp.lang.perl.misc) there are some people whose posts I consistently value. I would like a way to always read their posts. Conversely, I would like a /. "killfile" so that I don't have to read posts by people who have irritated me in the past. Again, I'm not talking about reducing their karma so they can't post - I want a filter so I don't see their posts.
Rather, each reader should have the ability to see a customized view of
For example, if I see a post moderated up to "5" and I think's it's a bonehead post, I would like to de-value the opinions of the moderators who liked that post. This is different from meta-moderation: I'm not saying that these moderators are bad and should be prevented from moderating, I'm saying that I don't want to see the results of their moderations.
Similarly, when I read
Eventually, each user will build up a profile of posters and moderators whose opions they trust and distrust. One can picture people trading these profiles, merging profiles with other people's, etc.
As the amount of information on the net grows, developing a systematic way to figure out which of it you trust and which of it you don't is going to become a very important problem.
Didn't Brin have some sort of reputation management system in Earth? Reading up on this might be a good place to start.
Anyway, the point I want to make is the people have different tastes and want to read different subsets of all posts. There's no reason to force the same view of moderation on everyone.
So obviously it's important for Slashdot to have a good moderation system, otherwise we might end up with a SNR equal to that of newsgroups.
The good thing about Slashdot is that we can add and take away as many features as we want, because there's no protocol involved here, only code. So we can try one thing, see how it works, and if it doesn't work out, try something else. I think though, that we have to try not to make the moderation system *too* fancy. If we make moderation too difficult and subject to review after review after review, then I think moderators will begin to ask themselves "what is the point?". I would hate to see something like this, for example, become reality.
On another topic, I think that meta-moderation in the comments list itself is a bad idea. Let's say I'm person A, or a friend of person A. Person A gets their post moderated down. I might choose to meta-moderate that post. Even if we ban the poster themselves from meta-moderating where their own posts are involved, we can't stop them from telling all of their friend to go there and do it for them.