No, you don't ever conclude anything by repeating inconclusive studies, except maybe that you're doing something wrong. Read the linked paper by Ionnidis. He's over the top and misrepresents some things to make his point, but he's not wrong - doing many inconclusive studies actually INCREASES the chance that your conclusion is wrong. It DECREASES your knowledge. Especially when those inconclusive studies are misrepresented as conclusive.
Economics and psychology have varying standards, but the standards of discovery in particle physics are very high. Particle physicists don't do a lot of repetition of inconclusive studies, and they don't make any serious conclusions based on such things.
Specificity, in science, is the number of true negative results over the total number of negative results. More generally, it's how specific something is, i.e. how well it excludes false alternatives or how well something rules out an effect. You seem to be using the word to mean precision, or accuracy, or some combination of the two. Your statements actually make more sense switching "sensitivity" for "specificity." Sensitivity is the ability to detect something that is present.
Your claim: averaging two or more values cannot provide you with an improvement in precision, therefore you should never report average values with more significant digits than your individual measurements. This is fairly easy to prove incorrect mathematically. Basic statistics classes cover the topic, and generally cover the derivation of the square root law: the mean of N independent measurements has a precision that is sqrt(N) times greater than the precision of the individual measurements. Sometimes it's stated as the noise is decreased by a factor of sqrt(N).
You seem to have taken someone's (your high school chem teacher maybe?) insistence on using significant digits in calculations and extrapolated it WAY too far. Significant digits are a quick and dirty type of error propagation calculation, and point out the fact that you can't increase information by calculation. You can reveal existing information, or display it so it is easier to see, but you can't create it. Significant digits is a poor method though, because it does not take into account magnification of errors when you do things like multiply, which proper error propagation analysis does, and it assumes that the precision of the measurement is equal to the number of quoted significant digits, which it rarely, if ever, is. Almost any non-trivial scientific measurement is made multiple times and a proper standard error is calculated. And the standard error has a denominator that is... sqrt(N).
Averaging is NOT a simple calculation - each measurement you're averaging brings new information to the table, and so averaging them together increases precision. That's the point of averaging.
As for your example, it's kind of unfortunate as well. You CAN increase resolution by taking multiple images that are related in specific ways and performing calculations to combine them into a single, higher resolution image. It's called super resolution, and is possibly most commonly used in light microscopy. You can also increase the resolution and other characteristics of specially sampled images by adding information about their sparsity. This is called compressed sensing.
You started this thread by criticizing scientists for doing something correctly, when you apparently don't even know the basics of the field. Then you brushed off the gentle suggestion that you might be wrong as "a semantic argument." May I suggest that the actual issues are a little more involved than the application of a rule of thumb taught to sixteen year olds?
I'm sorry, from your first post I thought you were an interested layman who had a few common misconceptions. From your second post it seems you're more of the anti-intellectual and doesn't want to hear anything contrary bent.
To be clear, your first post was flat out erroneous. Not semantically challenged, or debatable. Wrong. If you're interested I can explain to you why it is wrong, or you can continue thinking your smarter than those idiot scientists who average two numbers together and end up with a more precise one.
And please stop using the word specificity. You have no idea what it means.
Okay, it may not be exactly a myth. We can't tell. I strongly suspect there is actually a negative publication bias.
What most people think are "negative" results are actually inconclusive. A non-significant p-value is NOT a negative result. That misunderstanding is very widespread, and leads to lots of high level mistakes. Half of the neuroscience papers published in top journals including Nature the last two years that could make a mistake based on that fallacy, did. And neuroscience didn't seem to be particularly worse than most other fields.
A non-significant p-value is just that - not significant. Inconclusive. Getting an actual negative result is considerably more work than getting a positive one. You need to figure out what the minimum effect size you're interested in is (you should do that for positive results too, but almost everyone just uses zero for that) and show that your confidence intervals do not include it. As Ionnadis points out, you really should consider the power of your study as well (also for positive results), and take a stab at estimating the priors too.
If you go and do all that, and also do a quality experiment, in my experience you actually have a pretty good chance of getting published, because a) it's clear to the reviewers you've done a really thorough job (any idiot can run some data through a t-test and get a p-value, negative results are harder) and to show a negative result your study is probably much higher powered than a positive result one, meaning an impressively big p-value.
The problem is not that there's a positive publication bias, it's that most scientists don't know how to show negative results so there are very few negative papers around.
I bet what you think are negative results are actually inconclusive results. LOTS of people, including many of the ones writing papers about positive publication bias, make that mistake. An insignificant p-value is NOT a negative result. It's an inconclusive one. In order to actually show a negative result you have to do more work, and delve into the (usually very simple) stats that few people know. Most studies don't have the power to show actual negative results, but in my experience if you actually do the other work you get published. It's almost a given because the reviewers never see that sort of thoroughness.
"When and if I average those numbers the final average can't have more then two significant digits."
Yes, the average can have more precision than the individual measurements. That's actually kind of the point of an average. It can't improve accuracy though, for the most common definitions of accuracy.
Your definitions of accuracy and precision are sort of right, but also sort of misleading. And the way you use specificity is incorrect. But as you correctly point out, a lot of working scientists are a bit fuzzy on all these concepts too.
"also takes into account whether they are consequential or not"
That makes the problem worse. We need to evaluate papers based on whether they are good science or not, and not publish the ones that are bad science. Currently the "negative results" which are actually inconclusive results, are not published because they are, well, inconclusive. Unfortunately a lot of the "positive" results are also inconclusive, but they ARE published. The solution is not to publish more "negative" results, it's to stop publishing the flawed "positive" ones.
We generally call observation, modelling, prediction and model testing "studies."
The problem is that a lot of the observation is flawed, the modelling based on that observation may be flawed, the predictions are unreliable and the model testing is insufficient.
The basic problem is not any kind of bias, it's that the majority of working scientists don't know how to do adequate stats. It's quite simple to fix. Most scientists could probably learn the majority of what they need to know in an afternoon seminar focusing on the actual problem.
He claimed to visit more factories than he did, and claimed he spoke with more workers than he did. So his sample size was inflated. He claimed to meet with workers who were poisoned. He probably didn't since the incident happened a long way from where he said it did. He completely fabricated a scene where he interviewed a worker. He lied to his producer about the name of his interpreter.
The author himself admits that his piece is fiction. He says "it uses a combination of fact, memoir, and dramatic license." You can make all the silly justifications about it "being a show he's doing in a theatre" but the fact remains that it is not a factual work, as it was originally presented.
It may paint an accurate picture of working conditions in China. It may not. You can't tell from that piece, because it's a work of fiction. Which is fine. The problem is, he originally presented it as a factual documentary. That's wrong.
Um, do you realize that the post you replied to, from an actual tax accountant, is saying that Forbes's conclusion is correct (much higher tax rate than 9.8%: 24.2%) but that their reasoning about why the NYT made such an error is somewhat incorrect?
It seems the NYT didn't do their proper due diligence before publishing an inflammatory anti-Apple article.
Maybe if you're writing a letter. Try writing something even twenty or thirty pages long with a couple of figures. It's nowhere near as fast as it SHOULD be, but it's a LOT faster than it used to be.
Yeah, my iPad handles figures in text better than Word on an i7, but both do a MUCH better job than the 386 DX40 we bought because you needed a fast computer to do "desktop publishing."
Established scientists use Word to write grant applications and edit papers their students have written.
When I was a grad student my supervisor was a computer scientist who used IDL in his PhD. He hadn't written code, of any kind, for ten years. He wanted to learn some Python (because he wanted to write a hockey pool calculator) but couldn't find the time.
Yes, but we're talking about the SCIENTIFIC community. Engineers use MatLab because that's what they learn in school, and that's mostly because that's what their professors are familiar with.
Some of the people doing heavy duty number crunching still write Fortran code, but they're a small minority. Yes, most scientists use libraries that were written in Fortran all the time, but very few write code in it. It's not a very good language for modern general use.
High level, powerful languages like Python and Ruby are very useful for science. Why do you think MatLab has been so popular? MatLab is falling from favour now because it hasn't kept up with the times (the object extensions aren't great for example), is closed, expensive, and more difficult to extend. But Python (or Ruby) with standard numerical libraries doing the heavy lifting is an excellent solution for most scientists.
Even your way, I'd happily surround the locked block with a couple of comments to indicate it, in return for getting rid of curly braces and yahoos who DON'T use indentation to make things clearer.
Excel isn't a language. MatLab might beat Python, but it's been losing ground. R? I love R, but it's not a general purpose language and very few scientists know how to use it.
Personally I find R is much nicer wrapped up in Python.
Well, in 1979 (okay, maybe a wee bit earlier) at many universities, one of the major jobs of the IT guy was to receive couriered boxes of punch cards from other universities that didn't have computers, run those through the computer, then mail back the resulting syntax error.
There's a slight difference between IT in 1979 and IT in 2012.
Good software engineers know CS like good civil engineers know physics. A good civil engineer has to have an excellent knowledge of things like Newtonian mechanics, but doesn't really need to know much or anything about relativity, quantum mechanics, or most of the rest of physics. And he really doesn't need to know how to produce new knowledge of physics.
Slashdot Mirror
Taken as fractions of the entire relevant markets, this theft must rank among the biggest of all time.
When the Internet Fun Bucks are specifically made to be a libertarian free market ideal of untraceable cash, yes.
No, you don't ever conclude anything by repeating inconclusive studies, except maybe that you're doing something wrong. Read the linked paper by Ionnidis. He's over the top and misrepresents some things to make his point, but he's not wrong - doing many inconclusive studies actually INCREASES the chance that your conclusion is wrong. It DECREASES your knowledge. Especially when those inconclusive studies are misrepresented as conclusive.
Economics and psychology have varying standards, but the standards of discovery in particle physics are very high. Particle physicists don't do a lot of repetition of inconclusive studies, and they don't make any serious conclusions based on such things.
Specificity, in science, is the number of true negative results over the total number of negative results. More generally, it's how specific something is, i.e. how well it excludes false alternatives or how well something rules out an effect. You seem to be using the word to mean precision, or accuracy, or some combination of the two. Your statements actually make more sense switching "sensitivity" for "specificity." Sensitivity is the ability to detect something that is present.
Your claim: averaging two or more values cannot provide you with an improvement in precision, therefore you should never report average values with more significant digits than your individual measurements. This is fairly easy to prove incorrect mathematically. Basic statistics classes cover the topic, and generally cover the derivation of the square root law: the mean of N independent measurements has a precision that is sqrt(N) times greater than the precision of the individual measurements. Sometimes it's stated as the noise is decreased by a factor of sqrt(N).
You seem to have taken someone's (your high school chem teacher maybe?) insistence on using significant digits in calculations and extrapolated it WAY too far. Significant digits are a quick and dirty type of error propagation calculation, and point out the fact that you can't increase information by calculation. You can reveal existing information, or display it so it is easier to see, but you can't create it. Significant digits is a poor method though, because it does not take into account magnification of errors when you do things like multiply, which proper error propagation analysis does, and it assumes that the precision of the measurement is equal to the number of quoted significant digits, which it rarely, if ever, is. Almost any non-trivial scientific measurement is made multiple times and a proper standard error is calculated. And the standard error has a denominator that is... sqrt(N).
Averaging is NOT a simple calculation - each measurement you're averaging brings new information to the table, and so averaging them together increases precision. That's the point of averaging.
As for your example, it's kind of unfortunate as well. You CAN increase resolution by taking multiple images that are related in specific ways and performing calculations to combine them into a single, higher resolution image. It's called super resolution, and is possibly most commonly used in light microscopy. You can also increase the resolution and other characteristics of specially sampled images by adding information about their sparsity. This is called compressed sensing.
You started this thread by criticizing scientists for doing something correctly, when you apparently don't even know the basics of the field. Then you brushed off the gentle suggestion that you might be wrong as "a semantic argument." May I suggest that the actual issues are a little more involved than the application of a rule of thumb taught to sixteen year olds?
I'm sorry, from your first post I thought you were an interested layman who had a few common misconceptions. From your second post it seems you're more of the anti-intellectual and doesn't want to hear anything contrary bent.
To be clear, your first post was flat out erroneous. Not semantically challenged, or debatable. Wrong. If you're interested I can explain to you why it is wrong, or you can continue thinking your smarter than those idiot scientists who average two numbers together and end up with a more precise one.
And please stop using the word specificity. You have no idea what it means.
Okay, it may not be exactly a myth. We can't tell. I strongly suspect there is actually a negative publication bias.
What most people think are "negative" results are actually inconclusive. A non-significant p-value is NOT a negative result. That misunderstanding is very widespread, and leads to lots of high level mistakes. Half of the neuroscience papers published in top journals including Nature the last two years that could make a mistake based on that fallacy, did. And neuroscience didn't seem to be particularly worse than most other fields.
A non-significant p-value is just that - not significant. Inconclusive. Getting an actual negative result is considerably more work than getting a positive one. You need to figure out what the minimum effect size you're interested in is (you should do that for positive results too, but almost everyone just uses zero for that) and show that your confidence intervals do not include it. As Ionnadis points out, you really should consider the power of your study as well (also for positive results), and take a stab at estimating the priors too.
If you go and do all that, and also do a quality experiment, in my experience you actually have a pretty good chance of getting published, because a) it's clear to the reviewers you've done a really thorough job (any idiot can run some data through a t-test and get a p-value, negative results are harder) and to show a negative result your study is probably much higher powered than a positive result one, meaning an impressively big p-value.
The problem is not that there's a positive publication bias, it's that most scientists don't know how to show negative results so there are very few negative papers around.
It's just you. Corporate R&D produces products. Sometimes incremental engineering improvements. Not science.
Corporations USED to do some decent science, even more or less basic science. But not anymore.
I bet what you think are negative results are actually inconclusive results. LOTS of people, including many of the ones writing papers about positive publication bias, make that mistake. An insignificant p-value is NOT a negative result. It's an inconclusive one. In order to actually show a negative result you have to do more work, and delve into the (usually very simple) stats that few people know. Most studies don't have the power to show actual negative results, but in my experience if you actually do the other work you get published. It's almost a given because the reviewers never see that sort of thoroughness.
"When and if I average those numbers the final average can't have more then two significant digits."
Yes, the average can have more precision than the individual measurements. That's actually kind of the point of an average. It can't improve accuracy though, for the most common definitions of accuracy.
Your definitions of accuracy and precision are sort of right, but also sort of misleading. And the way you use specificity is incorrect. But as you correctly point out, a lot of working scientists are a bit fuzzy on all these concepts too.
"also takes into account whether they are consequential or not"
That makes the problem worse. We need to evaluate papers based on whether they are good science or not, and not publish the ones that are bad science. Currently the "negative results" which are actually inconclusive results, are not published because they are, well, inconclusive. Unfortunately a lot of the "positive" results are also inconclusive, but they ARE published. The solution is not to publish more "negative" results, it's to stop publishing the flawed "positive" ones.
We generally call observation, modelling, prediction and model testing "studies."
The problem is that a lot of the observation is flawed, the modelling based on that observation may be flawed, the predictions are unreliable and the model testing is insufficient.
The basic problem is not any kind of bias, it's that the majority of working scientists don't know how to do adequate stats. It's quite simple to fix. Most scientists could probably learn the majority of what they need to know in an afternoon seminar focusing on the actual problem.
He claimed to visit more factories than he did, and claimed he spoke with more workers than he did. So his sample size was inflated. He claimed to meet with workers who were poisoned. He probably didn't since the incident happened a long way from where he said it did. He completely fabricated a scene where he interviewed a worker. He lied to his producer about the name of his interpreter.
The author himself admits that his piece is fiction. He says "it uses a combination of fact, memoir, and dramatic license." You can make all the silly justifications about it "being a show he's doing in a theatre" but the fact remains that it is not a factual work, as it was originally presented.
It may paint an accurate picture of working conditions in China. It may not. You can't tell from that piece, because it's a work of fiction. Which is fine. The problem is, he originally presented it as a factual documentary. That's wrong.
Um, do you realize that the post you replied to, from an actual tax accountant, is saying that Forbes's conclusion is correct (much higher tax rate than 9.8%: 24.2%) but that their reasoning about why the NYT made such an error is somewhat incorrect?
It seems the NYT didn't do their proper due diligence before publishing an inflammatory anti-Apple article.
Also, the "one thing" the Foxconn documentary "got wrong" was actually several utter fabrications. If a "documentary maker" lies and fabricates evidence, he SHOULD have his reputation dragged through the mud, and his documentaries ARE worthless. Daisey himself has said that the production "is theatre." http://www.washingtonpost.com/lifestyle/style/this-american-life-cites-fabrications-in-documentary-on-apple-suppliers/2012/03/16/gIQAsJ6sGS_story.html
Maybe if you're writing a letter. Try writing something even twenty or thirty pages long with a couple of figures. It's nowhere near as fast as it SHOULD be, but it's a LOT faster than it used to be.
Yeah, my iPad handles figures in text better than Word on an i7, but both do a MUCH better job than the 386 DX40 we bought because you needed a fast computer to do "desktop publishing."
You might want to take a bit of a break from Fortran. You seem to be a bit personally attached. Kind of a Slashdot stereotype and all that.
Established scientists use Word to write grant applications and edit papers their students have written.
When I was a grad student my supervisor was a computer scientist who used IDL in his PhD. He hadn't written code, of any kind, for ten years. He wanted to learn some Python (because he wanted to write a hockey pool calculator) but couldn't find the time.
Yes, but we're talking about the SCIENTIFIC community. Engineers use MatLab because that's what they learn in school, and that's mostly because that's what their professors are familiar with.
Some of the people doing heavy duty number crunching still write Fortran code, but they're a small minority. Yes, most scientists use libraries that were written in Fortran all the time, but very few write code in it. It's not a very good language for modern general use.
High level, powerful languages like Python and Ruby are very useful for science. Why do you think MatLab has been so popular? MatLab is falling from favour now because it hasn't kept up with the times (the object extensions aren't great for example), is closed, expensive, and more difficult to extend. But Python (or Ruby) with standard numerical libraries doing the heavy lifting is an excellent solution for most scientists.
Better, and in Python:
if acquireLock():
code()
moreCode()
yetMoreCode()
releaseLock()
or
try:
acquireLock()
code()
moreCode()
yetMoreCode()
releaseLock()
except:
panic()
Even your way, I'd happily surround the locked block with a couple of comments to indicate it, in return for getting rid of curly braces and yahoos who DON'T use indentation to make things clearer.
That's fine in Python.
Excel isn't a language. MatLab might beat Python, but it's been losing ground. R? I love R, but it's not a general purpose language and very few scientists know how to use it.
Personally I find R is much nicer wrapped up in Python.
It's called "independent" and it's the only party that should be allowed in a representative democracy anyway.
Yes, but they're a little fuzzy on the SE != CS part.
Well, in 1979 (okay, maybe a wee bit earlier) at many universities, one of the major jobs of the IT guy was to receive couriered boxes of punch cards from other universities that didn't have computers, run those through the computer, then mail back the resulting syntax error.
There's a slight difference between IT in 1979 and IT in 2012.
Good software engineers know CS like good civil engineers know physics. A good civil engineer has to have an excellent knowledge of things like Newtonian mechanics, but doesn't really need to know much or anything about relativity, quantum mechanics, or most of the rest of physics. And he really doesn't need to know how to produce new knowledge of physics.
Okay. Are you volunteering as the first target?