I only see one potential problem with this: One person could self moderate their messages very low in order to gain global ranking when he is going to be moderated back up. I think that if there is self moderation (and I like the ides), the adjustment to the global ranking should be:
Let f be the final rank of the message, Let d be the default rank of the writer Lef m be the self moderation (with m = d)
if m d, add f-d to global ranking.
That way, if a person over-estimates his message, he loses point, if he is being modest, but the message doesnt exceed his average, the score doesn't change, and if he is better than his default ranking, he gains points, regardless of his self-moderation. I think it's important that gaining points do not depend on the self moderation.
Moderation is great, and I'm thrilled Slashdot is taking a nice step in that direction. But designing a moderation system is also very tricky: it has to be user friendly, but safe and reliable. This balance will greatly depend on the integrity and quality of the moderators.
In other words, moderators have to be moderated. For now, this work (of moderating the moderators, let's call that 'metamoderation') is handled by Rob, and so it is up to him to tune the balance mentioned above to help him in his task.
I will try here to consider the challenges of metamoderation, and propose some more technical tools to assist Rob in that task.
The goal of moderation is to assign a rank to all comments, and rank them well. The available ressources for that work are the moderators. Of course, all moderators are not perfect.
Ok, what do we mean by "a perfect moderator"? Well, we have two possibilities: it can either be Rob (this is the case if you read slashdot passively, and have a total trust in Rob's jugement to guide you through your slashdot experience), or it can be you, the reader (obviously, you know better!). The second solution is obviously better, but is more difficult to implement. Now, we can define the true rank of a page to be the rank decided objectively by the perfect moderator.
If the perfect moderator were able to rank all by himself, it would be just great. Oviously, that is not possible. Thus we need to use non-perfect moderators, and thus, have some kind of control over the error introduced. The way it is done right now is totally static: In the first phase, Rob used his human jugement to hand-pick 20 moderators. In the second phase, he used the assumption that a person who posts good comments is a "good" moderator. One could design a more complicated system, but let me delay this discussion until later.
Ok, so now, we have a bunch of good moderators, so we need a voting system. The moderation guidelines talks about the requirements of this system for the rank results, but doesn't talk about the requirements for the voting system. This causes some problems. Here is a scenario using the current voting system:
Some article talks about night people vs. day people. If it is posted during the night, night people will push down messages bad for them until they becomes -1, and will be upped later to +4 by day people when they wake up. If the message had been posted during the day, the contrary would have happened.
Altough this scenario is very improbable, it reveals a few flaws in the current voting system. So here are a few requirements for a voting system that may solve some of the flaws
Stability & convergence: as time progresses, and moderators vote, the error should go down. When all moderators have voted, the resulting rank should reflect all votes. This is obviously not the case with the current system, as two rival factions of moderators could keep the rank oscilliating indefinitely, and the final rank is decided by the most stuborn faction.
It should be possible to evaluate the quality (or the error) of a ranking. This is not currently possible.
The voting system should be order-independent. By that, we mean that the resulting rank should not depend on the order in which the votes have been cast. It is just a bit more tricky to see that the current system does not respect that. If yoou consider that a person's vote is just "up" or "down", then, yes, it is order-independent, but in the present case, the true vote of a person is what he believes to be the rank of the comment, and his up or down depends on that and the current score, so that in reality, the final score will be decided by the last voter. This means that in the system we want, the vote should not depend on the current rank.
The weight of a vote should not depend on anything else than the quality of the moderator (which is decided by the mmmetamoderator). In particular, it should not depend on the amount of time the moderator is spending on slashdot. Since in the current system, the number of currently available points is maxed at 10, a moderator who is voting all the time has more weight (more points to give) than someone who is not. (if I understood the system correctly).
Don't limit the amount of work the moderator can do. Right now, a moderator can only cast a vote on one tenth of the messages.
Enough with the criticisms already. Here is a suggestion for the voting mechanism:
every moderator gives a score between -3 and 5 (or something) to any message they read. All other messages have the status "not read" for that moderator.
Ok, I admit the memory necessary to keep track of all that info can be a problem (but you could for example close the votes after 24hours, and delete that info), but there are clear advantages:
all of the above requirements are met
you can display not only the average rank, but also the variance which gives an information on the error. The metamoderator can then look at the messages with high variance to detect problems.
You can keep a count on how many moderators actually read a message, which can be useful for telling moderators what they still have to do.
compute the distance between some moderator, and the perfect moderator, and exclude the ones whose distance is too large.
do the same, taking some user as the perfect moderator to provide the user with a moderation tailored to his taste
etc.
Ok, I touched many things here, and I have many more ideas, but I can't keep on writing forever....
P.S. are you going to release the code for the new and improved slash?
Moderation is great, and I'm thrilled Slashdot is taking a nice step in that direction. But designing a moderation system is also very tricky: it has to be user friendly, but safe and reliable. This balance will greatly depend on the integrity and quality of the moderators.
In other words, moderators have to be moderated. For now, this work (of moderating the moderators, let's call that 'metamoderation') is handled by Rob, and so it is up to him to tune the balance mentioned above to help him in his task.
I will try here to consider the challenges of metamoderation, and propose some more technical tools to assist Rob in that task.
The goal of moderation is to assign a rank to all comments, and rank them well. The available ressources for that work are the moderators. Of course, all moderators are not perfect.
Ok, what do we mean by "a perfect moderator"? Well, we have two possibilities: it can either be Rob (this is the case if you read slashdot passively, and have a total trust in Rob's jugement to guide you through your slashdot experience), or it can be you, the reader (obviously, you know better!). The second solution is obviously better, but is more difficult to implement. Now, we can define the true rank of a page to be the rank decided objectively by the perfect moderator.
If the perfect moderator were able to rank all by himself, it would be just great. Oviously, that is not possible. Thus we need to use non-perfect moderators, and thus, have some kind of control over the error introduced. The way it is done right now is totally static: In the first phase, Rob used his human jugement to hand-pick 20 moderators. In the second phase, he used the assumption that a person who posts good comments is a "good" moderator. One could design a more complicated system, but let me delay this discussion until later.
Ok, so now, we have a bunch of good moderators, so we need a voting system. The moderation guidelines talks about the requirements of this system for the rank results, but doesn't talk about the requirements for the voting system. This causes some problems. Here is a scenario using the current voting system:
Some article talks about night people vs. day people. If it is posted during the night, night people will push down messages bad for them until they becomes -1, and will be upped later to +4 by day people when they wake up. If the message had been posted during the day, the contrary would have happened.
Altough this scenario is very improbable, it reveals a few flaws in the current voting system. So here are a few requirements for a voting system that may solve some of the flaws
Stability & convergence: as time progresses, and moderators vote, the error should go down. When all moderators have voted, the resulting rank should reflect all votes. This is obviously not the case with the current system, as two rival factions of moderators could keep the rank oscilliating indefinitely, and the final rank is decided by the most stuborn faction.
It should be possible to evaluate the quality (or the error) of a ranking. This is not currently possible.
The voting system should be order-independent. By that, we mean that the resulting rank should not depend on the order in which the votes have been cast. It is just a bit more tricky to see that the current system does not respect that. If yoou consider that a person's vote is just "up" or "down", then, yes, it is order-independent, but in the present case, the true vote of a person is what he believes to be the rank of the comment, and his up or down depends on that and the current score, so that in reality, the final score will be decided by the last voter. This means that in the system we want, the vote should not depend on the current rank.
The weight of a vote should not depend on anything else than the quality of the moderator (which is decided by the mmmetamoderator). In particular, it should not depend on the amount of time the moderator is spending on slashdot. Since in the current system, the number of currently available points is maxed at 10, a moderator who is voting all the time has more weight (more points to give) than someone who is not. (if I understood the system correctly).
Don't limit the amount of work the moderator can do. Right now, a moderator can only cast a vote on one tenth of the messages.
Enough with the criticisms already. Here is a suggestion for the voting mechanism:
every moderator gives a score between -3 and 5 (or something) to any message they read. All other messages have the status "not read" for that moderator.
Ok, I admit the memory necessary to keep track of all that info can be a problem (but you could for example close the votes after 24hours, and delete that info), but there are clear advantages:
all of the above requirements are met
you can display not only the average rank, but also the variance which gives an information on the error. The metamoderator can then look at the messages with high variance to detect problems.
You can keep a count on how many moderators actually read a message, which can be useful for telling moderators what they still have to do.
compute the distance between some moderator, and the perfect moderator, and exclude the ones whose distance is too large.
do the same, taking some user as the perfect moderator to provide the user with a moderation tailored to his taste
etc.
Ok, I touched many things here, and I have many more ideas, but I can't keep on writing forever....
Thanks a lot Rob, this is really great. Looking at this, and at the many more boxes I would like to be able to add, I was just thinking:
Why not allowing to put one or more boxes whose content is fetched at some URL. That way, if anyone creates a new great slashdot side box, he would just have to publicize the URL, and anyone who wants can just add it to their configurations.
Another application is if someone (actually, me) wants to put his own bookmarks in a sidebox, rather than Rob's, he could just put his own bookmarks in a file on the web, and configure his slashdot config to display it.
Slashdot is already the greatest site out here. This would make it completely unbeatable. All (well, I know, might not be that easy) you (Rob) have to do for this, is add that special box type that fetches its contents at some prescribed URL. Actually, all your boxescould already be written like that.
btw, are you going to release the code for the new and improved version of slash?
If Virtual PC (mac) ever releases an update with Pentium III emulation, it wouldn't be difficult to be able to set the Processor ID to whatever you like... If the ID is used by any soft for authentification, you can just fake being anyone with just a click, if you know their ID!
Slashdot Mirror
Sorry about that, I meant:
.le. f .le. d, don't change global rating .lt. m, remove m-f from global ranking, .lt. d, add f-d to global ranking.
Let f be the final rank of the message,
Let d be the default rank of the writer
Lef m be the self moderation
(with m = d)
if m
if f
if f
.lt. means less than
.le. means less or equal
Sorry, I can't seem to be able to display those characters correctly!
I only see one potential problem with this:
One person could self moderate their messages very low in order to gain global ranking when he is going to be moderated back up. I think that if there is self moderation (and I like the ides), the adjustment to the global ranking should be:
Let f be the final rank of the message,
Let d be the default rank of the writer
Lef m be the self moderation
(with m = d)
if m d, add f-d to global ranking.
That way, if a person over-estimates his message, he loses point, if he is being modest, but the message doesnt exceed his average, the score doesn't change, and if he is better than his default ranking, he gains points, regardless of his self-moderation. I think it's important that gaining points do not depend on the self moderation.
Moderation is great, and I'm thrilled Slashdot is taking a nice step in that direction. But designing a moderation system is also very tricky: it has to be user friendly, but safe and reliable. This balance will greatly depend on the integrity and quality of the moderators.
In other words, moderators have to be moderated. For now, this work (of moderating the moderators, let's call that 'metamoderation') is handled by Rob, and so it is up to him to tune the balance mentioned above to help him in his task.
I will try here to consider the challenges of metamoderation, and propose some more technical tools to assist Rob in that task.
The goal of moderation is to assign a rank to all comments, and rank them well. The available ressources for that work are the moderators. Of course, all moderators are not perfect.
Ok, what do we mean by "a perfect moderator"? Well, we have two possibilities: it can either be Rob (this is the case if you read slashdot passively, and have a total trust in Rob's jugement to guide you through your slashdot experience), or it can be you, the reader (obviously, you know better!). The second solution is obviously better, but is more difficult to implement. Now, we can define the true rank of a page to be the rank decided objectively by the perfect moderator.
If the perfect moderator were able to rank all by himself, it would be just great. Oviously, that is not possible. Thus we need to use non-perfect moderators, and thus, have some kind of control over the error introduced. The way it is done right now is totally static: In the first phase, Rob used his human jugement to hand-pick 20 moderators. In the second phase, he used the assumption that a person who posts good comments is a "good" moderator. One could design a more complicated system, but let me delay this discussion until later.
Ok, so now, we have a bunch of good moderators, so we need a voting system. The moderation guidelines talks about the requirements of this system for the rank results, but doesn't talk about the requirements for the voting system. This causes some problems. Here is a scenario using the current voting system:
Some article talks about night people vs. day people. If it is posted during the night, night people will push down messages bad for them until they becomes -1, and will be upped later to +4 by day people when they wake up. If the message had been posted during the day, the contrary would have happened.
Altough this scenario is very improbable, it reveals a few flaws in the current voting system. So here are a few requirements for a voting system that may solve some of the flaws
Enough with the criticisms already. Here is a suggestion for the voting mechanism:
every moderator gives a score between -3 and 5 (or something) to any message they read. All other messages have the status "not read" for that moderator.
Ok, I admit the memory necessary to keep track of all that info can be a problem (but you could for example close the votes after 24hours, and delete that info), but there are clear advantages:
Ok, I touched many things here, and I have many more ideas, but I can't keep on writing forever....
P.S. are you going to release the code for the new and improved slash?
Moderation is great, and I'm thrilled Slashdot is taking a nice step in that direction. But designing a moderation system is also very tricky: it has to be user friendly, but safe and reliable. This balance will greatly depend on the integrity and quality of the moderators.
In other words, moderators have to be moderated. For now, this work (of moderating the moderators, let's call that 'metamoderation') is handled by Rob, and so it is up to him to tune the balance mentioned above to help him in his task.
I will try here to consider the challenges of metamoderation, and propose some more technical tools to assist Rob in that task.
The goal of moderation is to assign a rank to all comments, and rank them well. The available ressources for that work are the moderators. Of course, all moderators are not perfect.
Ok, what do we mean by "a perfect moderator"? Well, we have two possibilities: it can either be Rob (this is the case if you read slashdot passively, and have a total trust in Rob's jugement to guide you through your slashdot experience), or it can be you, the reader (obviously, you know better!). The second solution is obviously better, but is more difficult to implement. Now, we can define the true rank of a page to be the rank decided objectively by the perfect moderator.
If the perfect moderator were able to rank all by himself, it would be just great. Oviously, that is not possible. Thus we need to use non-perfect moderators, and thus, have some kind of control over the error introduced. The way it is done right now is totally static: In the first phase, Rob used his human jugement to hand-pick 20 moderators. In the second phase, he used the assumption that a person who posts good comments is a "good" moderator. One could design a more complicated system, but let me delay this discussion until later.
Ok, so now, we have a bunch of good moderators, so we need a voting system. The moderation guidelines talks about the requirements of this system for the rank results, but doesn't talk about the requirements for the voting system. This causes some problems. Here is a scenario using the current voting system:
Some article talks about night people vs. day people. If it is posted during the night, night people will push down messages bad for them until they becomes -1, and will be upped later to +4 by day people when they wake up. If the message had been posted during the day, the contrary would have happened.
Altough this scenario is very improbable, it reveals a few flaws in the current voting system. So here are a few requirements for a voting system that may solve some of the flaws
Enough with the criticisms already. Here is a suggestion for the voting mechanism:
every moderator gives a score between -3 and 5 (or something) to any message they read. All other messages have the status "not read" for that moderator.
Ok, I admit the memory necessary to keep track of all that info can be a problem (but you could for example close the votes after 24hours, and delete that info), but there are clear advantages:
Ok, I touched many things here, and I have many more ideas, but I can't keep on writing forever....
Slef.
Thanks a lot Rob, this is really great. Looking at this, and at the many more boxes I would like to be able to add, I was just thinking:
Why not allowing to put one or more boxes whose content is fetched at some URL. That way, if anyone creates a new great slashdot side box, he would just have to publicize the URL, and anyone who wants can just add it to their configurations.
Another application is if someone (actually, me) wants to put his own bookmarks in a sidebox, rather than Rob's, he could just put his own bookmarks in a file on the web, and configure his slashdot config to display it.
Slashdot is already the greatest site out here. This would make it completely unbeatable. All (well, I know, might not be that easy) you (Rob) have to do for this, is add that special box type that fetches its contents at some prescribed URL. Actually, all your boxescould already be written like that.
btw, are you going to release the code for the new and improved version of slash?
If Virtual PC (mac) ever releases an update with Pentium III emulation, it wouldn't be difficult to be able to set the Processor ID to whatever you like... If the ID is used by any soft for authentification, you can just fake being anyone with just a click, if you know their ID!