Slashdot Mirror
Web Geniuses Or Web Dimwits?
ScribeCity writes "The Washington Post has a provocative piece about online experiments at identifying experts. One wonders when someone will come up with a truly effective formula for measuring human intelligence — or take a stab at doing so — that exploits all the stuff people are publishing online." From the article: "This wisdom of the crowd could be outsmarted by what Michael Arrington, editor of the TechCrunch blog, recently dubbed the 'wisdom of the few.' Sites like PicksPal rely on input from the masses chiefly as a venue for auditioning prospective experts, on the theory that these virtuosos could provide even more accurate information and predictions than the crowd. 'If you figure out which ones did the best and get rid of the ones who have no idea, you'd do even better. Distill it down to the people who really know,' Arrington said."
Just get a chimp to throw darts at the wall...
In order to effectively determine the rate of experts vs. everyone else, you could simply scan through all previous Slashdot posts (while removing those prefaced by IANAL) and easily determine those that are experts.
;)
Make sure you are browsing at -1, *those* people are the real experts
One wonders when someone will come up with a truly effective formula for measuring human intelligence
It won't happen, not because it's not possible, but because some group or another will have a lower mean score, and the cries of racism, sexism, ageism, redbluestateism, culturalism, OSism, haircolorism, footsizeism, dicksizeism, or whateverism will drown out the truth.
You know... the way it is right now.
Stephen Colbert would be proud.
A sword cuts both ways, after all. I fear this tech.
Make a group based moderation system, where you moderate in groups.
Step1: Lets say Democrat/Republican. When a Rep mods something up, all other Reps see it modded up. If a Dem mods something down, no other Reps see it modded down.
Step2: Identify posters who say stuff that gets modded up past a certain point. Lets say you get a point for the top 10 posts of each day. Then the posters with the most points are dubbed experts in their field.
Its simple, and I'm suprised no one has done it before. It's like Digg in some ways, but vastly superior as groups don't bicker over what they declare as news, and it identifies experts.... maybe even political candidates.
God spoke to me.
I'm not an 'expert' in anything, yet I read far and wide enough to pick up lots of random & indepth tidbits that 'experts' have not heard about.
Ever heard the joke about the phd professor who studied more and more about less and less, until he knew everything about nothing? Yea, many people would consider that professor an expert.
[Fuck Beta]
o0t!
There is a certain logic to this. How many times have "experts" told us screwy nonsense, and had lousy track records, and yet the public at large retained them as experts? Sometimes, the untrained may be able to see things that the supposedly well-trained can't.
;)
Or to put it another way, it all becomes a set of probabilities. If person X has guessed the outcome of something (say, a football game) correctly 80% of the time, then you're safer betting on his predictions than you are betting on expert Y who is only correct 30% of the time. If you aggregate the probabilities and successes, you should be able to develop a model with a high probability of being correct. You'll never be able to gain 100% accuracy, but that's just the nature of the Universe.
Javascript + Nintendo DSi = DSiCade
... you sound like you could be an expert.
sigs, as if you care.
'If you figure out which ones did the best and get rid of the ones who have no idea, you'd do even better. Distill it down to the people who really know,' Arrington said.
I've always said that elections should qualify each voter's ballot to make sure the decision is made by the people who are best equipped to decide. The first page of a voting ballot should be a questionnaire that asks simple unbiased questions that require the voter to demonstrate knowledge of who or what they are voting on. "What does candidate X say their stance is on abortion?" "When did you first hear about initiative I-456?" "Please specify which political party each candidate below belongs to", etc. The score a voter gets on their questionnaire would then be used as a "weight" factor when counting their ballot, so that people who know the candidates and the issues better get more of a say, which is clearly how things ought to be.
Moderator hint: a comment is neither "Flamebait" nor "Troll" if it is true.
1) Read old threads at +5, new threads at +2
2) If a person has a lot of insightful/informative posts, check their posting history
3) If they are consistently +3/4/5 informative/insightful, add them as a friend
4) add points to friend's posts so they start out +2.
OK, seriously, I don't do that but if I did, I'd see posts of "wise ones" and ignore posts from those that don't make the cut.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Yes, but who decides which issues make the questionnaire?
The questionnaire's authors would in-effect be defining the criteria for election.
Maybe I vote for someone based upon whether or not they annoy the crap out of me.
That's my prerogative.
------ The best brain training is now totally free : )
Let's say you have a pool of 10,000 prognosticators. You ask each one to pick the winner of 10 football games. The odds of getting all 10 correct are 1 in 2^10=1024. So out of the pool of 10,000 people, by random chance alone you're likely to get about 10,000/1024 = 10 people who pick all 10 games correctly. Are these people "geniuses"? No, they just got lucky during this particular trial. The odds of them getting game #11 correct are just 50-50.
BTW, this can be used as the basis of a scam against the "geniuses" if you can convince them that they have special powers as a result of the trial.
Moral of the story: Be very careful with statistics.
The most rabid believers in American Exceptionalism are the exact same people whose policies are destroying it.
Any monkey able to produce a component of Shakespeare's collective works should qualify.
It seems to me that Google could easily be used to find genius...
www.jmagar.com
-
I would love to input the data spewed out by all of the talking heads who make their political predictions (and other useless nonsense predictions) and are so often wrong. Then we could post disclaimers at the bottom of the screen when they talk:
(...Mr. Speak-for-my-Party has only been correct 25% of the time and he was on both sides of the issue for 75% of his correct predictions).
"Democracy is based on the assumption that a million men are wiser than one man. How's that again? I missed something.
Autocracy is based on the assumption that one man is wiser than a million men. Let's play that over again, too. Who decides?"
And...
"Always listen to experts. They'll tell you what can't be done, and why. Then do it."
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
The common parlance term for the "expert witness" among lawyers is "the courtroom whore" -- Lots of fancy sounding credentials, gotta have the doctroal degree, and willing to say anything for a price. Totally worthless idiots in most cases.
Besides:
"If you ask enought experts, you can confirm any opinion or theory."
Not sure who said it, but it's valid IMHO.
www.effectiveelectrons.com "chips that work" Analog, RF, Mixed Signal
So predicting the outcome of a sporting event or a stock's performance makes one a genius or expert?
Lord help us and save us. The idiocy of the media and general public never ceases to amaze me.
Yes, but who decides which issues make the questionnaire?
The questions are limited to those that are objective and factual in nature (that's what unbiased means), so it really doesn't matter.
Maybe I vote for someone based upon whether or not they annoy the crap out of me. That's my prerogative.
It shouldn't be, because it's a ridiculous and unsafe basis for choosing a candidate.
A candidate should win because they are the most qualified for the job and best represent the public's views on the issues. They shouldn't win because they are the best looking or the hippest. Therefore your vote should only count if you are voting for a legitimate reason.
Moderator hint: a comment is neither "Flamebait" nor "Troll" if it is true.
The problem with using public opinion about "experts" is that "experts" must then spout public opinion to be recognized.
True experts often have opinions contrary to public opinion. Just look at Slashdot. Insight is sometimes modded as flamebait. Counter intuitive opinions or assertions get derided and the author insulted.
Sorry, but "the masses" are generally stupid and would rather burn experts at the stake than question their tiny little world.
In Massachusetts, in our governor's race, one of the candidates is an attorney who was a public defender and he defended a couple despicable people. What does the opponent do? Leak private information about his brother in-law's criminal record, and accuse him of being "weak on crime" because he defended obviously guilty people.
The educated in MA, almost unanimously, call the ads appauling because it isn't attacking merely the candidate, but the whole justice system in our country. The "stupid" say things like "How could he defend that person." Never once thinking about "innocent until proven guilty." (Which remains true through appeal.)
So, when people try to parse the nonsensical ravings of the masses for reasoned information, I recall the old computer addage: "Garbage in, garbage out."
My mother is a professor of education at a college in California and this is something that educators have been talking about for a long while. Google for "multiple intelligences" for a lot more information, but basically there's a theory that says that "intelligence" can be divided up into a number of categories and that people tend to excell in one or two of these areas, but few are outstanding in all of them.
The standard breakdown is something like:
Atheletes tend to excell at Spatial and Kinesthetic, while the stereotypical geek is strongest in Logical-mathematical and weakest in Interpersonal.
I'm not sure I completely agree that this is the end-all-be-all for understanding intelligence, but it does provide an interesting look into ways to classify people who might not be "book smart." For instance, a terrific ballerina might not have excellent Interpersonal or Linguistic intelligence, but she certainly has some special "intelligence" that allows her to excell in an area where I would certainly be an abject failure.I encourage anyone interested in this idea of multiple intelligences to poke around and do some research. Again, it may not be the final answer, but it provides an interesting framework for thinking about the topic.
Interested in a Flash-based MAME front end? Visit mame.danzbb.com
Bollocks. Let's say we have two questions that both are answered correctly by the same percentage of people. It's pretty much guaranteed that different groups of people get those questions wrong. By choosing the right questions you can weigh the elections, even if the chosen questions are "objective and factual"...
Explains everything observed by these sites. I predict that a lack of science and math education will continue to result in people fruitlessly attempting to use past performances in predicting chance-based events (either because of true randomness or sufficient complexity to thwart casual analysis) to denote exports that will at some point start getting it wrong.
Isn't this just that same old thing, where for each sporting event, you send a mailer to 50% of the people picking one team, and 50% picking the other, and whoever wins, that 50% of your original audience gets split between the two possible winners in the next mailing? Eventually you end up with a small audience, but they're CONVINCED you have a flawless sports betting "system" and pay you to learn it.
Here, by pretending you're figuring out who the "experts" are, you're not diluting your audience with each round of guessing; instead, you're diluting your potential pool of "experts" (or systems), and eventually everyone decides that person X is always right, when really odds were that at least one person in a large pool of guessers would guess right 100% of the time.
Past performance is no guarantee of future performance, people.
This sounds like the old scam. Pick 1000 people. On day 1, send 500 of them a prediction that stock A will go up and send the other half a prediction that the stock will go down.
On day 2, the stock either went up or down. Either way, you made a correct prediction to 500 people. Split the 500 and send two more predictions on an all new stock.
Keep repeating this. On the fifth day, you'll have 75 people who have seen you make 5 perfect predictions in a row. Now ask each of them for $10,000 to invest in your next prediction...
Just because one person happens to have hit the mean each time doesn't mean he's got "the knack". Statistically, there's sure to be someone whose guesses approach the mean. But that doesn't mean that their next prediction is any more likely to be accurate.
Stick with the aggregated mass knowledge.
-- Don't Tase me, bro!
One thing you have to remember: Perception > Reality. Speaking intellegently and writing intellegently is usually enough to convince someone that you actually know what you're talking about, if you're audience is ignorant or naive. That makes for a lot of percieved experts in the field of technology. Take the example of an internet born initiative to ban dihydrogen oxide in some county California http://www.dhmo.org/facts.html. Read this. If you haven't already heard of this, well, dihydrogen oxide is water. See how easy it is to convince a bunch of soccer moms they need to ban water? (Or that apple needs to abandon hardware... hehe)
Similes are like metaphors
It'd be easy to tilt the questions. Ask questions that are bullet-points for Focus on the Family (such as candidates' position on "family values") and you give that bloc more weight. Ask Sierra Club questions (position on environmental issues) and that bloc is favored. Ask trigger questions like abortion, gun control, immigration policy and you mainly get extremists on both sides who know these positions best.
It'd be nice to think you're selecting for interested, educated people without bias. I think it'd be difficult to maintain a bias-free question set.
I always thought that it should be sort of hard to register to vote. Take a written and driving test and have to renew every so often. But, there are problems with that approach too. My version of the test would be blatantly unfair, disqualifying any voter who thinks they talk with a diety that has opinions on political policy.
Man, you really need that seminar!
If they are posting stuff on the Internet and sound like an expert, they are most likely NOT an expert.
"As God is my witness, I thought turkeys could fly." A. Carlson
Let's say we have two questions that both are answered correctly by the same percentage of people. It's pretty much guaranteed that different groups of people get those questions wrong.
There would have to be a positive correlation between "the group of people who didn't know the answer to question #1" and some other classification (such as "the group of people who are homosexuals"). It would be extremely unlikely as well as extremely difficult to intentionally rig.
Besides, you can solve it completely by requiring that each choice on the ballot get the same number of questions about them in the questionnaire. So if a question asks, "Where does candidate X stand on this issue?", then you'd have to ask the same question for each of the other candidates.
Sure, people are more likely to know where their own candidate stands than the others, but that's fine -- the more questions you answer correctly, the more your vote counts. So if you only know about your own candidate, your vote counts more than someone who doesn't know anything about any candidates, but your vote counts less than someone who knows about their own candidate and the other candidates too. Everything works out as it rightly should.
I'm sure some people will read this and ask, "But doesn't this discriminate against people who are less educated, or who don't have the time to learn about the issues or candidates?" The answer is yes, of course it does. It's just acknowledging the reality that you can't make a good decision without being informed. It's no different than requiring someone to have eyesight to drive a car -- that policy is intentionally discriminatory against the blind, but who's complaining about that?
Finally, even if there are imperfections with the system I'm suggesting, it would still be light-years better than how the process currently works. Nothing is ever perfect, and you can poke holes in anything, but the system in use now has such obvious problems that it's easy to think up ways to radically improve it.
Moderator hint: a comment is neither "Flamebait" nor "Troll" if it is true.
Foiled by statistics again!
You're as bad as the guy who takes issue with the statement, "We won't stop until all children are above average!"
You're as bad as the guy who worked at MegaHuge Hedge Fund in the late 90's. His boss walked into the office one day all excited about a new way to measure risk, called "Downside Risk Quotient." He asked the guy how often the stocks in their portfolio were below their mean price, or what their "Downside Risk" was. The guy foolishly answered, "50% of the time."
Do what you can, with what you have, where you are.
And now, everybody feels pressured to post. Great job- post an article about finding internet experts on slashdot.
The original eligiblity requirements for voting in the US were intended be the Founding Fathers to do just that. Unfortunately they where racist,sexist,and classist. I agree that it is odd that you need a test to be allowed to drive but not a test to vote. Maybe the best would be an unbiased video of a debate between all the people on the ballot that would be required viewing before going to the voting booths.
We are all just people.
Didn't Slashdot solved this problem with their moderation scheme? Oh, wait, nevermind... that would mean "Karma Whores" would qualify as experts. Nevermind.
I'm not fat, just big boned...
This quote, headlined today on google, is instructive: It has always been the prerogative of children and half-wits to point out that the emperor has no clothes. But the half-wit remains a half-wit, and the emperor remains an emperor. - Neil Gaiman I find that 'Experts' are largely chosen based on qualities other than their expertise. Usually they have good personalities and make friends easily, especially with leadership. The leadership prefers to pick and talk with experts who generally agree with their views and look good. So, it is very difficult for me to look at a given expert and think 'Hrmmm...this guy must really know something about X and that is why he is on CNN'. More likely he was friends with a CNN producer.
They have tried this before http://en.wikipedia.org/wiki/Literacy_test and it was flawed.
"The theoretical basis for them was that illiterate persons were not sufficiently informed about the candidates and issues involved to be able to make a truly informed decision. In practice, however, the literacy requirement was often used to prevent those determined by the ruling class to be undesirable, such as the poor, racial and ethnic minorities, and other groups that it wished to see disenfranchised, from voting."
Never trust a system rife with potential for abuse not to be abused.
And how can you have one definitive correct answer to a question like "Where does candidate X stand on this issue?" - it is impossible. Have you seen how long current voting takes at some voting places? Hours long lines and that is with just having to prove who you are, and not how smart you are...
"But this one goes to 11!"
So basically you could have a massive system to guage the stock market, based on each quote player's unquote picks. You could call it the New York Stock Exchange, for example... Must go now. It's time for my random walk.
Say hello to my little sig.
....but I did stay at a Holiday Inn Express last night!
On the other hand, if they asked factual questions like:
;-)
1. What is the hexadecimal opcode for the IBM 360 "Jump" instruction?
I would be perfectly happy with the idea.
9/11 Eyewitnesses to Explosive WTC Demolition 1 of 2
Given a collection of experts, there are algorithms for judging the reliability of individual experts on the basis of a set of trial tests, and combining the expert opinions so your error rate is guaranteed to approach that of the best-performing expert (roughly 2x worse, worst-case, asymptotically). This is related to the statistical learning technique of boosting. See the discussion here for more technical information, and a link to a paper.
nice attempt to clean up democracy but that won't work. the candidate who gets rid of this system is the one to get re-elected by the idiots allowed back in. besides that, there is no reasonable way to pick questions. what do want to suggest? voting on the questions. I would suggest trashing democracy all together for the failed system it is and just going with a well led authoratarian gov.
It is much different than requiring eyesight on a driver's test, because it presumes a motive.
Remember that freedom is really about choice. My right to vote does not presume WHY I vote. I have the freedom to vote however I want. I could vote for someone because of their stance on an issue, or because they have great hair, or because they're a buddy of mine, or because I once dated their sister, or because I like their accent, or because they're in the same party I'm in, or because I think their last name sounds good in that lame rap song I wrote in high school.
The government cannot decide for me that I must vote based on their position on issues. To continue your analogy from before, this would be like the DMV requiring me to take a test on the locations of grocery stores in my area before I would be given a drivers' license, thus presuming that the reason I would want to drive is to go to the store. What if I only want a drivers' license so I can get into bars more easily, and never intend to drive? What if I only want to drive to my grandmother's house 8 hours away, and walk to the store? The state has no reason to decide WHY I would drive.
Freedom is about choice, and that means I can choose to vote for a stupid reason, even if it's a reason you disagree with.
For security, the MD5 hash of this message and sig is 09f911029d74e35bd84156c5635688c0.
I've had my doubts about the idea that if you take the average of all the guesses a crowd makes it comes out to be near correct. So last time I was at an event that had a "guess the number of jellybeans and win a prize" contest I asked the organizer for all the stubs at the end of the evening. I went home and punched all the guesses into a spreadsheet.
The mean was in the ballpark but not accurate enough to win the prize. When I took the median I got an answer that was as accurate as the best guess. So in theory if I had sat there with a laptop and watched what people wrote down I could have won the contest.
I'm still not sure this phenomenon has a practical purpose (other than electing officials I guess) but it was an interesting thing to do. I'd be curious to do the same analysis at a very large gathering with hundreds of entries.
Okay, so what are the odds someone can flip a coin ten times in a row, and have it turn up heads every time? 1 in 1024. So now say we have 10,000 people try this and capture each on video, and we find which actually succeeded in turning up heads 10 times in a row. Now, what are the odds that person will flip another heads on their 11th try? It's still only 1:1 of course. However there are those that will believe, after watching the video, that the odds are better than that - the whole "gut feeling" thing. That's all this website is doing - reporting on a the few lucky enough to predict the correct result n times in a row. Obviously when it comes to sporting events and the like, knowledge certainly helps. However with a large enough number of people submitting, pure chance will result in a few getting great results even with completely random submissions.
The people that set this site up were simply smart enough to combine the statistics and the psychology and make money off of it.
Dan East
Better known as 318230.
Probably somewhere before posting a story on Slashdot... (everyone who reads this is an internet addict)
Linguistic intelligence
Logical-mathematical intelligence
Spatial intelligence
Bodily-Kinesthetic intelligence
Musical intelligence
Interpersonal intelligence
Intrapersonal intelligence
Naturalist intelligence
The problem with that breakdown is that it has as much data to support it as most of the stock market analyses one gets in spam. This pdf text shows one mathematical tool for debunking stuff like this "many intelligences" theory.
Of course, you can invent as many definitions for "intelligence" as you want, but that doesn't mean those definitions are valid. I wear jeans more often than my cousin, does that mean I have more "jeans intelligence" than him? If the data doesn't give any useful information, then the theory is bullshit, no matter how many "experts" vouch by it.
problems can be divided into three categorys
1) easily knowable answers, like who is the 10th president, or angelina jolies boyfriend
2) hard to know answers, like how may diff types of beetles live in california - with enough money, you could answer this, but it would take alot of work
3) unknowable answers, like what interest rates will be this year, or what the stock market will do tomorrow (obviously unaswerable, cause if you could, you would make gates look like a pauper), or what will happen in Iraq.
The wisdom of the crowds is simply related to #1;
the problem is that "experts" get paid to pretend that they know how to give answers to #3, like the talking heads that provide cheap filler for TV news.
Except that past performance is no guarantee of future performance. IOW, just because an 'expert' was right often, even recently, doesn't mean that person is well informed on the issue you might bring to the table this afternoon. You still have to look at other info about that person besides a ratio of wins/losses.
... should be achieved sufficiently with lie detector tests... that fail.
Take off every 'ZIG' !!
And the best is when one wise man takes the input of a million and makes the best decision...
The DHMO site is a joke, you know...
Boycott Sony
Somebody tell Jimbo Wales!
Tubby or not tubby. Fat is the question
Bottom line: representative democracy is more a matter of trust in the representation than a matter of well-defined policy. If I don't trust a politician, I don't trust what he promises. In general I do not trust any politician to deliver on their promises, as they always seem to find a way to do something else. So what's left? Voting for the one you 'guess' does least damage and has the right general idea. Not the one with lots of specific ideas that I might actually agree with.
>and just going with a well led authoritarian gov.
Yes, but the important question then becomes: "how do you prevent ending up with an atrociously led authoritarian gov"?
Any system that needs qualifiers like "well led" or "unbiased" to work well, immediately raises the question how to ensure those qualifiers.
> Web Geniuses and Web Dimwits
Don't you mean Webiniuses and Webimwits?
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
I'm not sure what you mean by aggregating probabilities, but I think you can only combine weak classifiers and get better results if all the classifiers have greater than 50% accuracy.
Say I have 3 classifiers which are 51% accurate. If I take their predictions as votes and go with the majority then that will give me an accuracy of 51.5% (0.51 * 0.51 * 0.49 * 3 + 0.51 * 0.51 * 0.51)
A majority of 101 classifiers would get you 58% and 1001 about 74%. The more classifiers the closer to 100% you could get.
But if you have some classifiers with less than 50% accuracy then the best you can do is ignore them!
rt
If 10,000 people each flipped a coin ten times in a row, chances are good that nine or ten of them would have it land heads-up each time. Have identified the "expert" coin-tossers, then? I don't think so.
Thanks, John Madden, I don't think anyone realized that.
Similes are like metaphors
The real geniuses on the internet are posing as idiots. That way, when they say something worthwhile, they are more relatable to the common man and what they say is given more weight. Either that or, when they want to say something intelligible, they know to stay off the grid by only posting using their army of zombies or as an Anonymous Coward.
But what would I know?
Extremely difficult? ok... Let's say you want to rig elections and have a decent budget. you get 4000 people to answer a survey of 1000 questions. One of the questions is "Who are you going to vote?". The correctness of the answers to the 999 other questions will be checked for correlation with the vote-question. Quite probably you are going to find a few questions that correlate in the right direction.
I don't think that would be difficult, and I certainly don't think that would be unlikely -- every party out there would be doing it.
Think about it. Mutual fund managers give advice because they think it will make them money, not because they think it will make you money. They lie to dump stocks they wish they hadn't bought. The percentage would be 100%, but sometimes stocks they don't like do well anyway.
--
Bush lied. Thousands died. Impeach.