Slashdot Mirror

← Back to Stories (view on slashdot.org)

Ask Slashdot: Best Practices For Using a Reputation Engine To Rate Information?

Posted by timothy on from the well-if-grantrobertson-says-it dept.

GrantRobertson writes: For my graduate project, I am considering developing a web engine designed around sharing and organizing actual information in a way that people would actually like to and easily be able to use it. Unlike a wiki, the information will be much more granular with lots more metadata and organization. Unlike a web forum, the information will be be organized rather than dispersed throughout thousands of random posts, with little room for dominant personalities to take over. While I like Stack Overflow, I am planning far more structure. While I enjoy the entertaining tangents on Slashdot, I don't want those to take over sites created using my engine. Naturally, there must be some way to prevent armies of bots or just legions of jerks from derailing web sites created using this engine. Given that, what would you say are some good rules to include in the reputation engine for such a site. What kinds of algorithms have you found to be most beneficial to the propagation and spread of actual knowledge. What would you like to see and what have you found to be dismal failures?

14 of 100 comments (clear)

  1. You mean to tell me by Anonymous Coward · · Score: 4, Insightful

    you are counting on Slashdot to do your graduate project for you? That is a horrible idea in so many ways...

    1. Re:You mean to tell me by ShanghaiBill · · Score: 4, Insightful

      you are counting on Slashdot to do your graduate project for you?

      No, the submitter is just asking for ideas. 99% of the work is the implementation, and the submitter is not asking for any help with that.

      My 2 cents: The two problems to avoid are: 1) Groupthink, where dissenting opinion are drowned out or ignored. 2) Onerous or arbitrary rules that drive away experts, so you are left with only clueless idiots commiserating with each other (example: answers.yahoo.com).

    2. Re:You mean to tell me by Lunix+Nutcase · · Score: 2

      No, the submitter is just asking for ideas. 99% of the work is the implementation, and the submitter is not asking for any help with that.

      So then why don't they work that out with their thesis advisor? That's the entire reason you have one.

      Secondly, no, 99% of the work in a thesis is not implementation unless you're getting a degree from a shit school and you have a rubber-stamp committee. The vast majority of your thesis is in the original research you did and writing down your findings from that research. The implementation is just a prototype to show off a working example of your ideas.

    3. Re:You mean to tell me by Mikkeles · · Score: 3, Insightful

      No. He is asking users for features and characteristics that said users would find advantageous for a web engine that accumulates and organizes web data.

      <snark>Something I wish more programmers would do.</snark>

      --
      Great minds think alike; fools seldom differ.
    4. Re: You mean to tell me by bill_mcgonigle · · Score: 2

      why would his thesis advisor have more knowledge of attempts at reputation systems than the /. readership? Most of what's tried in industry is buried next to the bodies behind the data center, not in research papers. Field research is usually more valuable than literature surveys.

      --
      My God, it's Full of Source!
      OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  2. Impartial referees by forgottenusername · · Score: 3, Insightful

    Pretty much any set of algos is going to be easily defeated by humans trolling and no system is going to be anything near perfect. My thoughts;

    1) Create a small set of simple, concise rules that are inviolate
    2) Have a system so people can mark submissions as good (no rules broken/useful) or bad(rules broken)
    3) Have your referees do nothing but determine if that submission is breaking one of your rules
    4) Based your user trust as a derivative as how the user voted compared to what the referee votes

    The theory is any controversial submission is going to get flagged & referees attention. Their job is limited in scope to just determining if the post breaks the site rules or not, nothing to do with quality / content / opinion. If users are trying to game the system their votes are going to conflict with the referees so their user trust is going to go down, whereas if people agree their trust is going to go up.

    Eventually you'll have a group of users that you can generally trust to do the right thing so you can weight their actions accordingly.

    Obviously there are some weaknesses;

    - Referees are pretty much god (that's why the scope of their power is extremely narrow and simple)
    - You can end up with hive mind (though you can combat that if enough trusted users conflict with other trusted users). I'd argue it's a way better protection than pure crowdsourcing ala reddit where the demographics crush submissions into hivemind

    Just tossing that out there off the top of my head. It's not something to replace automated reputation management, just something augment it and limit some of the abuse.

  3. Ummm... by Lunix+Nutcase · · Score: 3, Insightful

    Isn't the whole point of thesis work that you find some novel solution to a problem through your own research not enlisting others to do it for you?

    1. Re:Ummm... by Lunix+Nutcase · · Score: 2

      Then you went to a diploma mill not a real university.

    2. Re:Ummm... by ljw1004 · · Score: 3, Insightful

      Isn't the whole point of thesis work that you find some novel solution to a problem through your own research not enlisting others to do it for you?

      That's maybe 5% of thesis work. Another 20% is the grunt-work to investigate the phenomena and gather up examples and counter-examples. Another 75% is getting a good understanding of the field and the existing state of the art.

      I think the poster has picked a good place (here on slashdot) as part of building up that other 95%.

  4. Semantic web? by monkeyporn · · Score: 2

    "Unlike a wiki, the information will be much more granular with lots more metadata and organization."
    Pretty sure the ideals behind a semantic web were supposed to cover this part. Never really took off though because, I think, people are to lazy to sort data to that degree of detail and the algorithms necessary to process and categorize human text with that level of granularity seem to be very hard to make.

  5. Reputations... by idbill · · Score: 2

    Look into mTurk (Mechanical Turk). Amazon doesn't provide a reputation engine, but anyone who posts any significant number of jobs there has some kind of version of it. I worked for several years on a project that integrated with mTurk and had its own reputation engine. There are a lot of gotchas where people try to game the system. It isn't a simple answer and depending on the situation I don't believe there is a one solution for all situations. Bill

  6. sounds like OneModel; maybe you can start there by lcall · · Score: 3, Informative

    Sounds like what I'm trying to do here (AGPL): OneModel.

    It doesn't have all the features, but what you describe is partly there, or planned for the future, though for now it's in the form of a text-only UI and you have to install postgres. The UI is something like a mix of git's "commit --interactive" and gopher (remember that, anyone?), but it is very efficient if you just read the screen and are a touch typist. Probably currently most suitable for someone who now uses emacs org-mode, or collapsible outlines of any kind, but wants to handle richer kinds of information (eg, GTD...) and a more task-specific UI.

    It's what I use as my own personal organizer and knowledge manager, but ~"sharing" features for collaboration, including reputation and others, are on the wish/plan list. Feel free to use it as a starting point, or join the list for discussion. I was hoping to get the web site updated with a later binary and an enhancement, and much more information on my future plans, by roughly next week. It still lacks a convenient installer but the INSTALLING file in github is current.

    If interested you could always get on the announcements list for when I add features. My health isn't great at the moment but I hope to be able to sell binaries or installers in the future for part-time income or the like. Patches or discussion on the list are welcome. I have been thinking hard about this since about 2000 and am glad to finally have something others can use, though the potential audience will be larger once there are better installers and other needed features, UIs etc.

    --
    A Free, fast personal organizer for touch typists: onemodel
  7. More structure? by YrWrstNtmr · · Score: 2

    While I like Stack Overflow, I am planning far more structure.

    More? Good grief. SO is already bad enough. Anything 'more' will simply chase users away, if they ever go there in the first place.

  8. DICE edited my headline. by GrantRobertson · · Score: 2

    I would like to point out that DICE edited my headline, which was originally, "Reputation Engine - Best Practices for Information-Based Site?" The existing headline makes it appear as if I am trying to use the reputation engine to rate the actual information. Instead, I merely want the reputation engine to cut down on the number of jerks on the site and reduce the influence of trolls, bots, and crusading armies. Once that is accomplished, I trust the "good" contributors to provide good and relatively accurate content by working together and collaborating. I do not expect any reputation engine to get to some ethereal "Truth."