Distributed Trust Metrics?

rw2 asks: "So I run a little political website and have had problems for years with users basically trolling the place. This is a problem that sites like Slashdot deal with through the familiar moderation scheme. Unfortunately that doesn't scale well to smaller sites. There are a couple reasons for this: a smaller sample size makes it easy to mess with the system; and with only several hundreds of people visiting everyday, it's hard to get regular enough moderation. So the question goes back to one of trust metrics. Advogato has a neat hack to deal with this, but even they have barely enough users to make it Work. Surely I'm not the first to desire this. I can think of several stumbling blocks sociologically. But technologically this is a dead simple idea. Has someone looked into developing such a system?"

"I've done some googling for systems that might work in a distributed fashion but turned up nothing. I'd happily register a key with an authority (ideally a distributed one, think supernodes rather than centralized structure) and have it verify my identity. Then, at each website participating in the trust network, I can provide my identity upon registration. As people moderate me and my comments, this feedback is applied to my profile both locally and network wide. The idea is that I may be all wet when it comes to tractors, but relatively well read on politics and technology (i.e.: my overall trustworthiness would be a 7, with a 3 on misc.rural, a 8 on slashdot.org and a 10 on poliglut.org). Now readers of my commentary have a more reasonable way of judging my trustworthiness on both a local and a global scale."

Comments

What I'd Do:

[Tumbleweed]

1) require e-mail response verified accounts to post

2) enable the ability to 'bozo-bin' someone: their account can be made so that they can still post, and they can see their posts, but noone else can. Most bozos won't even know they've been binned, and thus will not try to create a new account to get around it. Think of it as a honey pot for trolls.

3) Check for bozos all coming from the same domain - likely the same bozo who has realized he's been binned, and has created a new email address from (probably) his own domain - so bin all accounts from that domain.

That should cut down on the vast majority of problems, I'd think. Also, with a 'small' site, as you say, moderation doesn't work well. Well, with a SMALL site, you don't _need_ moderators to handle the load, so that should work out well, right? :)

Epinions.com

[inerte]

And its Web of Trust might help you. Let the social net filter the bad stuff.

It works very well on small samples, IMHO. In fact, I believe a Web of Trust doesn't scale in the thousands, or hundreds of thousands, because of the dilution of the metric.

Also, since you run a political website, a Web of Trust can help to "cluster" similar points of view.

Here's an idea

[anotherone]

Here's an idea I had a few days ago that I think would work very well... a Bayesian moderation system. Add or subtract points based on some kind of bayesian filter. Keep traditional user moderation, but use it to train the filters.

use reputation propagation

[DrEasy]

- Let each user decide who some of his friends/foes are, just like in Slashdot. Rate them accordingly, say on a scale of 0-1.

- that will filter posts by the people you've rated.

- as for the users you haven't rated:
- if there is a "path of trust" between you and that user, i.e. if there is a friend/foe of yours who has rated a friend/foe who has rated (...*x) this user, calculate a rate. You can try to multiply the rates, or use the average, whatever works best for you.
- if there isn't, or if the user is new, give him an initial rating, again, just like in slashdot.

At least that's the main idea... There are some complications when there are multiple path and so on. I have published a paper on this, so if you're interested, leave me a message and I'll send you the paper. I'm also curious about how the system can work in a non simulated setting.

Problems with moderation/filtering

[iangoldby]

Several people have suggested moderation or filtering schemes, in which users can say, essentially, which posts they like and which they don't like. Depending on the approach, the system could even learn, either by user (e.g. the Bozo filter) or by content (e.g. Baysian filtering), etc. It then promotes the posts people like, and hides the ones they don't like. (Sounds familiar?)

There is a fundamental problem with this though, which is particularly acute for a site such as yours that exists for the sake of political debate.

If you do this, then users will tend to be presented with opinions that are most similar to their own, and have dissenting opinion hidden from them. That's going to stifle debate and make the site much less interesting.

The only real way to do it is to find a small team of dedicated moderators who are able to objectively rate content according to its intrinsic worth rather than according to the opinion expressed. That's hard.