The Neuroscience of Screwing Up

resistant writes "As the evocative title from Wired magazine implies, Kevin Dunbar of the University of Toronto has taken an in-depth and fascinating look at scientific error, the scientists who cope with it, and sometimes transcend it to find new lines of inquiry. From the article: 'Dunbar came away from his in vivo studies with an unsettling insight: Science is a deeply frustrating pursuit. Although the researchers were mostly using established techniques, more than 50 percent of their data was unexpected. (In some labs, the figure exceeded 75 percent.) "The scientists had these elaborate theories about what was supposed to happen," Dunbar says. "But the results kept contradicting their theories. It wasn't uncommon for someone to spend a month on a project and then just discard all their data because the data didn't make sense."'"

Comments

Why most scientists and engineers screw up

[Anonymous Coward]

It's because they are not very good. We could use 75% fewer, and we'd get more done. They picked the wrong major in college.

Re:Why most scientists and engineers screw up

[Anonymous Coward]

Disclaimer: I am an academic research associate

Here's a quote from the article:

"But the results kept contradicting their theories. It wasn't uncommon for someone to spend a month on a project and then just discard all their data...The details always changed, but the story remained the same: The scientists were looking for X, but they found Y."

The dirty little secret is that the Y is not always unexpected, just too politically incorrect and dangerous to be released to the public. For example, my team at Rutgers just completed a comprehensive experiment measuring a variety of factors including intelligence and genetic makeup(read: race). What we discovered would have caused a political shitstorm orders of magnitude worse than that of Don Imus when he referred to our Women's basket ball team as "Nappy-headed Ho's", so we declared it unsucessful and quietly buried it.

We tried, we really did. We developed formulae which would account for environmental/nutrure factors. We were very forgiving with the fairness of our methods, and yet the numbers still added up in a way that was unflattering to our hypotheses.

Oh, well. Maybe they'll finally figure it out when a monkey-ass coon tries to blow up a plane and ends up lighting his nuts on fire. Wait, what? HA HA! Oh, man! What a Gorilla!

Re:Why most scientists and engineers screw up

[A beautiful mind]

I think the parent post is a brilliant example of what happens when someone perfects trolling to a science.

Re:Why most scientists and engineers screw up

[ResidntGeek]

Any chance you have a preliminary write-up, or even raw data I could read? Sounds exquisitely interesting. ResidntGeek@gmail.com if you can.

Re:Why most scientists and engineers screw up

[Jurily]

The dirty little secret is that the Y is not always unexpected, just too politically incorrect and dangerous to be released to the public.

So, when reality is racist, you change it?

Re:Why most scientists and engineers screw up

[spiffmastercow]

That's what wikileaks is for, man. If you're telling the truth, then you have an obligation to see that the data is released so that it can be evaluated, tested, and verified (or debunked).

Re:Why most scientists and engineers screw up

[lq_x_pl]

No, when reality is politically incorrect, you bury it. Just imagine if the ACLU sued life, the universe, and everything...

Re:Why most scientists and engineers screw up

[rikkitikki]

They'd get 42 dollars?

Re:Why most scientists and engineers screw up

[Hurricane78]

If he is, then he got nothing on you, who is also an expert in using reverse psychology for trolling. ^^

Re:Why most scientists and engineers screw up

[Anonymous Coward]

Or when the results you get aren't acceptable to the people responsible for continued funding.

Years ago, I worked for months trying to reproduce the Polywater research,
http://en.wikipedia.org/wiki/Polywater
and eventually reported that I was unable to do so.
The department considered my work a failure (as in, I must have been incompetent) and did not publish my findings. When, years later, the publications reporting successful discovery/creation of Polywater were shown to be fraudulent, and my results were correct, I did not even receive an apology.

Throwing out results is unethical as well as irresponsible. Many discoveries have come from re-evaluating what appears to be "bad" data. It might not be possible to use it now, but it should be at least stored.
For instance, it has been reported that the "bit of "scruff" on her chart-recorder papers that tracked across the sky with the stars"[1] looked like bad data to Jocelyn Bell Burnell's supervisors. Today we call the phenomenon a pulsar.
[1] Wikipedia

Re:Why most scientists and engineers screw up

[Alien Being]

Really? Or is it that you are SO politically correct that you cannot see truth.

I happen to have mod points and my on-the-fly ranking went from insightful to interesting to troll and back to interesting.

I've lived long enough to understand that each of the 6 billion people on this earth is different than every other. Some are remarkably good and some are remarkably bad. Most of us are just average in our own interesting ways.

But still, I do believe that genetic differences affect what we are and that genetic differences can be attributed to where our genes came from.

Those who choose to never risk offending anyone are perhaps the most intellectually dishonest among us.

Should I post this or should I mod YOU the troll?

Re:Why most scientists and engineers screw up

[TapeCutter]

"Those who choose to never risk offending anyone are perhaps the most intellectually dishonest among us."

All fine and good, except the OP does not contain anything more intellectual than a bunch of bald assertions wrapped in the emotions of a xenophobe. In other words, you should have modded the GP informative, the OP is a well formed troll.

Re:Why most scientists and engineers screw up

[joocemann]

The dirty little secret is that the Y is not always unexpected, just too politically incorrect and dangerous to be released to the public.

So, when reality is racist, you change it?

Don't attempt to reason or argue with Anonymous Cowards. There is no way to corroborate anything he/she said.

Just a little advice.

Re:Why most scientists and engineers screw up

[bzipitidoo]

Are groups of people from very different locations, such as whites and blacks, different? Of course they are! Not so different as to be separate species (yet, and with global communication, maybe never), but evolved in different directions to adapt to different conditions. Just being different means there are activities for which one group will be better suited than the other group. There has to be, otherwise they aren't different, are they? Get over both the racism and the political correctness, and admit this basic fact. Skin color is a fairly superficial difference. Africa has by far the most diversity-- the Eastern African perhaps has less in common with the Western African or the Pygmy than with whites.

Much more harmful is the tendency to oversimplify fitness to a single, grossly over broad measure of a difficult to define concept that is not universally relevant, such as IQ, and declare one group superior to another based on only that. When a declaration of superiority is made, you may be sure it is for purposes of propaganda. Geniuses (defines as people with IQ > 140, or perhaps > 160) make their share of fatal mistakes, have flaws that can render their supposed advantages much less valuable, take risks and sometimes lose, sometimes let success go to their heads (most recently, Mike Leach), just like everyone else. Bobby Fischer was a genius, but that monomania which made him able to be World Chess Champion hurt him in so many other ways. The Soviet Union really bought into the idea of chess (and other contests such as the Olympics) as a good measure of a society's fitness, and devoted so much effort to it that except for Fischer, they pretty much mopped the boards with other nations' best players. And what was it all for? Propaganda that ultimately proved empty when Communism collapsed. I expect the Space Shuttle astronauts all do very well on IQ tests, but is that a smart gamble, risking their lives on that thing, for the fame and money they get? For other sorts of fitness, there are many fantastic athletes, rock stars, leaders, and the like who ended tragically. A small change in conditions can at a stroke reverse the fitness of most any trait.

Re:Why most scientists and engineers screw up

[NeutronCowboy]

But still, I do believe that genetic differences affect what we are and that genetic differences can be attributed to where our genes came from.

The theory that race has nothing to do with intelligence has nothing to do with political correctness, and all with science: specifically, the scientific discovery that the taxonomy of human races is not definitive, not specific and has no basis in genetics. Which in turn means that the ggp's assertion that race was a statistically significant factor in their research means that their research was utter crap to begin with.

So let me ask you this then: what makes you think that race is the same as genetics, or that you can even reliably a race? I mean, outside of some outdated and non-scientific notions of physiognomy and phrenology?

Re:Why most scientists and engineers screw up

[Anonymous Coward]

Are you really asking an AC troll for sources?

You're wasting your time.

Re:Why most scientists and engineers screw up

[Ethanol-fueled]

The dirty little secret is that the Y is not always unexpected, just too politically incorrect and dangerous to be released to the public.

Eh, so the scumbag may have been racist. Dosen't make the first 5/6 of his post any less true. In science, the experiment comes first. The troll is probably just disappointed that science didn't prove his racist views wrong, as well as advance his career and get him more funding.

HehehehehHAHAHAHAHAHAHAAH!

Re:Why most scientists and engineers screw up

[indiechild]

A troll from Anonymous Coward suddenly equals reality?

How the fuck did that get modded interesting?

Re:Why most scientists and engineers screw up

[Toonol]

The idea that race is a fiction is a bad, well, fiction, and a clear example of the distortion of thought due to political correctness.

There are a number of human traits (and the genes which cause them) that statistically cluster into groups that correspond to what we consider race. You can test a person's DNA and determine their racial heritage, to a fairly accurate degree. Obviously race is real, if you can nearly automate measuring it. The fact that statistical clusters don't have firm boundaries doesn't mean those clusters don't exist.

Is race relevant? Not for most purposes, but it is for some. I understand that Asians are more likely to have difficulty digesting milk, for example; blacks have a higher tendency to have sickle-cell anemia. Declaring that any test that shows a tendency for races to vary based on genetics is CERTAIN to be flawed because you don't believe race exists is ludicrous.

Re:Why most scientists and engineers screw up

[joocemann]

The thing that makes it less true is that it is from an anonymous coward and the science was not published. Nothing that was said can be 'proven' or sought out based on what is there.

Re:Why most scientists and engineers screw up

[kdemetter]

So let me ask you this then: what makes you think that race is the same as genetics, or that you can even reliably a race? I mean, outside of some outdated and non-scientific notions of physiognomy and phrenology?

It's simple evolution based on enviromental conditions :
Everyone simply evolved so they can best survive in there current environment.

Like the difference between a polar bear and a black bear . It's basically the same bear , that has evolved to suit it's environment better.

Re:Why most scientists and engineers screw up

[NeutronCowboy]

Here's the problem. If you can't order every single human into one race or another, your model is flawed. If you're forced to resort to mixes of races, well, then you don't have any distinct race left.

Race concepts fall apart once actual taxonomic principles are applied to them. Your examples actually illustrate the problem quite nicely: not nearly all asians have problems with milk - specifically the Japanese the do. Indians (from the Indian subcontinent in Asia) do not. Blacks do not have a higher tendency for sickle-cell anemia, a certain group of people in Africa do. Blacks in the US do not have that trait.

How much does it suck to be so wrong? Your cognitive dissonance must be at a record high.

Re:Why most scientists and engineers screw up

[FiloEleven]

Indeed!

No scheme of inequality can be defended as corresponding to natural fact.... Superior and inferior can be determined only with respect to a single quality for a single purpose. Nor can a man's qualities be added together and averaged to give a final score or merit. In short, men are incommensurable and must be deemed equal.
- Jacques Barzun

Re:Why most scientists and engineers screw up

[Alien Being]

"So let me ask you this then: what makes you think that race is the same as genetics,..."

I don't think they are the same, but there is correlation between certain genes and certain groups of people. Whether to classify those groups as races or as extended families is more of a political question than a scientific one.

Just to clarify where I'm coming from, I don't believe that all people of a given skin color should be grouped together as a single race.

Re:Why most scientists and engineers screw up

[bsane]

Which is why people who claim scientists only care about the truth are wrong. You only cared about the truth and were fired. Plenty of other people would have been looking out for their job first, and made sure their results confirmed what the department expected.

That said- the great thing about science is that eventually the truth will be discovered despite the pressure for money/jobs. It may not happen in a lifetime, but as long as science continues, it will happen.

Re:Why most scientists and engineers screw up

[VoidEngineer]

Here's the problem. If you can't order every single human into one race or another, your model is flawed. If you're forced to resort to mixes of races, well, then you don't have any distinct race left.

Race concepts fall apart once actual taxonomic principles are applied to them.

Sort of. In a traditional hierarchical phylogenetic taxonomy, yes, race concepts fall apart. But if they don't necessarily fall apart with a cladistic genetic taxonomy.

Defining race is a classic problem of, well, classification. Put another way, it's like organizing books. Where do you place 'War and Peace'? In the fiction section? In the history section? In the classics section? In the russian literature section? It could legitimately be placed in any of those sections. The problem is that the book has a single physical instance. The book only exists in one place at one time. So, it can only be placed in one category at a time. And this is the problem with any phylogenetic based hierarchical taxonomy. It's not unique to race; it also applies to species, books, weblinks, and any other number of objects. It's why, before search engines, we had all these portal sites, like Yahoo!, who were focused on creating giant taxonomies of weblinks. And it was always a pain, because we had this intuition that a weblink should only exist in a single category at a time. This was a hold-over from library systems, where any particular book can only be placed on a single shelf at a time.

But then we discovered tagging. With tagging, a new type of taxonomy is possible, where a single entity can be placed in multiple categories at a time. And it turns out that tagging is equivalent to a genetic taxonomy. Each tag is equivalent to a gene (or meme, to be more precise). And we now give webpages lists of keywords, which function like a genome of sorts.

So, you're correct that race concepts fall apart at a hierarchical, phylogenetic based taxonomy. But with a genetic based taxonomy, race is 'tagged' by combination of genes... melanin count, lactose sensitivity, sickle-cell anemia, etc.

And what's more, this tagging and clustering, is a precursor to speciation. Consider the following simplified hypothetical example: a) mutant gene (A) interacts with the gene for lactose sensitivity such that, together, they cause a change in sperm mobility due to a lack of calcium, and b) another mutant gene (B) interacts with the gene for sickle-cell anemia such that, together, they cause a change in permeability to an egg due to lack of iron. If these two things were to hypothetically occur, it would make for a situation where sperm and egg couldn't unite, and a lactose intolerant father and sickle-cell anemic mother couldn't have children. Now then, one more consideration: say that these two mutant genes were actually very advantageous. Mutant gene A protects against flu and pnemonia, and mutant gene B codes for sexy pheremones. If these mutant genes are advantageous, then they'll spread throughout the population. But as the mutant genes spread through the population, the carriers of those genes, who also carry the genese for lactose intolerance and/or sickle cell anemia, would lose the ability to breed together. And this would be defined as a speciation event. Not only would those people be of different races, they would be unable to breed together, and would be different species.

Anyhow, it's worse than people fear. Not only does race actually exist, it's a precursor to speciation. Race just doesn't fit neatly into hierarchical phylogenetic taxonomies. Genetic taxonomies allow for overlapping, fuzzy boundaries. And that's exactly what Race is. Race doesn't fit into neat little hierarchical tree structures; rather, it's a fuzzy network of genes.

Re:Why most scientists and engineers screw up

[Anonymous Coward]

If you can't order every single human into one race or another, your model is flawed? That doesn't mean the model is wrong either. If you're forced to resort to mixes of races, well, then you don't have any distinct race left? So what? Call them populations and everything is okay? different alleles of a lot of genes cluster; and they do so relating to ancestry and population history. Thats neither surprising nor shocking.

Re:Why most scientists and engineers screw up

[Aram Fingal]

There are a number of human traits (and the genes which cause them) that statistically cluster into groups that correspond to what we consider race. You can test a person's DNA and determine their racial heritage, to a fairly accurate degree. Obviously race is real, if you can nearly automate measuring it. The fact that statistical clusters don't have firm boundaries doesn't mean those clusters don't exist.

While this is true, it still doesn't validate existing concepts of race. You can pick out a preexisting notion of race and, indeed, find genetic markers which will correlate with that concept. However, if you do it the other way around, throw out all such preconceived notions, look at the data and derive groupings of humans, you get totally different results. You don't get what people typically think of as the major races. ALFRED has some of this information although it takes a lot of work to go through all the data and the maintainers of the site try to stay out of discussions of race.

Re:Why most scientists and engineers screw up

[dasunt]

We tried, we really did. We developed formulae which would account for environmental/nutrure factors. We were very forgiving with the fairness of our methods, and yet the numbers still added up in a way that was unflattering to our hypotheses.

So how do you figure out formulas for environmental/nurture factors?

"Well, group X is more likely to smoke during pregnancy than group Y, leading to lower amounts of blood oxygen, which results in an average difference of 3 IQ points."

"Well, group Y is less likely to be stressed during pregnancy than group X, leading to an healthier environment for the fetus, which is good for a difference of 2 IQ points."

"Studies have shown that teachers are 25% less likely to call on students of X race, which should be good for a cumulative effect of 12 IQ points?"

I'm kind of curious how this works. And are the effects cumulative? If someone smokes and is stressed during pregnancy, do these two factors slightly cancel each other out? Or do they reinforce each other to cause more damage?

Re:Why most scientists and engineers screw up

[Just Some Guy]

Blacks do not have a higher tendency for sickle-cell anemia, a certain group of people in Africa do. Blacks in the US do not have that trait.

ORLY? The US Government says:

In the Unites States, it affects around 72,000 people, most of whose ancestors come from Africa. The disease occurs in about 1 in every 500 African-American births and 1 in every 1000 to 1400 Hispanic-American births. About 2 million Americans, or 1 in 12 African Americans, carry the sickle cell trait.

...making your closing amusingly ironic:

How much does it suck to be so wrong? Your cognitive dissonance must be at a record high.

Re:Why most scientists and engineers screw up

[omris]

But polar bears and black bears are separate species. "Races" are not. Races are more like breeds of cats, although significantly less distinct. So maybe more like gray squirrels.

Gray squirrels live all over North America, but don't look the same. Gray squirrels where I live are small and in fact, gray. Gray squirrels where my parents live are reddish brown and sort of chubby, and gray squirrels in Canada are huge and very dark, almost black. All the same species. They just look different. So maybe Canadian squirrels have fat storing genes that are, on average, 1% more efficient than the eastern American squirrels to deal with the cold. So now imagine that gray squirrels have developed air travel and boats and cars and the internet, and started meeting squirrels who looked different than they did. Now you can easily imagine some large, black squirrels who developed a fetish for tiny, gray, New England squirrels. They have babies and move here to RI, and now the eastern US has a lot of black squirrels, because black fur is a strong gene compared to light gray. But they do not share much more genetic heritage with big, black, Canadian squirrels than with midwestern squirrels.

Are the black squirrels in my back yard still the same race as the black squirrels in Canada? It depends a lot on what the word race means. If you define is "do they have black fur?" then, yes. If you define it as "do they have the same percentages of the telltale gene clusters as the black squirrels in Canada?" then, no. Maybe some of these squirrels have that more efficient fat storing gene, but only a few, since not does it not hinder their survival if they lack it, it's been watered down with the normal fat genes of the New England squirrels.

I think the issue really is that the way we define our race as individuals has very little bearing on anything other than appearance, which is a poor indicator of all but a small handful of genes, none of which relate to anything OTHER than things like hair , eye color or shape, skin color... yes, you are more likely to have a gene for sickle cell anemia if you have dark skin, but the frequency of the gene does not replicate the frequency found in most native African populations.There probably was a time in human history where there were populations which could more easily be genetically defined, but it would have to have been a time before there was any considerable interaction between cultures from different geographical areas. Which was clearly long before we knew we were made of cells, or that there was such a thing as DNA.

I plan on listening very closely when a study comes out of trying to link something like IQ to race in populations that have not had a lot of external genetic influence, but good luck designing that study. It would not be possible in the US, and good luck designing a reliable IQ testing method that will work on , say, aboriginals, isolated pockets of native Africans, and a small community deep in the Appalachian mountains. And good luck finding enough subjects to make statistical significance.

Re:Why most scientists and engineers screw up

[NeutronCowboy]

Heh. Looks like I was indeed wrong on sickle-cell anemia in the US. However, the original statement is still wrong: blacks do not share a uniform likelihood of coming down with the disease. On top of that, it isn't even restricted to black people. As a result, race is not a predictor for, or even correlated with, the disease.

Re:Why most scientists and engineers screw up

[NeutronCowboy]

Absolutely. Genetic taxonomies work great, because you have a simple definition that can be applied to everybody and comes up with a result that works.

However, when people talk about race, they never talk about a genetic taxonomy: they always talk about an appearance taxonomy. Flat nose, dark skin, curly hair: black. Blond hair, blue eyes, white skin: white. Etc. But that only classifies a fraction of the world's population, and means nothing outside of appearance.

Re:Why most scientists and engineers screw up

[VoidEngineer]

Ah, yeah. I'd agree with that. :)

I thought you were arguing against racial taxonomies in general. But it seems that we're in agreement that it's just the hierarchical phylogenetic taxonomies that don't work.

Re:Why most scientists and engineers screw up

[c6gunner]

Where did he say that he was fired?

Ben Stein? Is that you? :)

As for the bit about "plenty of other people ... confirming what the department expected" .... if you replace the word "plenty" with "a few" you'd be much closer to the truth. As you correctly point out, eventually the truth will be discovered (although it tends to take a much, much shorter amount of time than what you suggested). Scientists who intentionally fake data tend to be relegated to janitorial duties once their deception is discovered. So while there may be some short-term incentive to fake data, and while some scientists may be unethical enough to consider doing it, it rarely happens.

Of course, that number could probably be even further reduced if there wasn't a tendency in the scientific community to throw out negative results. That's one area that definitely needs improvement.

Re:Why most scientists and engineers screw up

[bsane]

Where did he say that he was fired?

Doh! I could have sworn it said that... Apparently I falsified my reading to support my point.

I also never said anyone had to fake their data- not outright at least. Just throw away what you don't like and maybe 2nd or 3rd try will be close enough to keep the grant payers happy. Its hard to shame someone if they can claim they made a best effort- even if they didn't.

I really hope its not prevalent, but especially in the 'softer' sciences it'd be hard to tell, wouldn't it?

Re:Why most scientists and engineers screw up

[Toonol]

Here's the problem. If you can't order every single human into one race or another, your model is flawed. If you're forced to resort to mixes of races, well, then you don't have any distinct race left.

I think you've just proven that colors don't exist.

The lack of distinct boundaries between states doesn't mean those states don't exist in reality. In the case of statistical clustering, they can even be rigorously mathematically defined.

ObClimate reference

[Anonymous Coward]

"The science is settled!" :P

Re:ObClimate reference

[Anonymous Coward]

Hide the decline!

Sometimes screwing up leads to success ...

[xmas2003]

The WIRED piece threads what is written in the summary around the story of how Arno Penzias and Robert Wilson at Bell Labs discovered Cosmic Radiation after being puzzled for a year about background noise on their radio telescopes ... even scraping pigeon poop off their gear as a possible source until they realized the signal was real - Homer Simpson would have said D'OH! ;-)

Re:Sometimes screwing up leads to success ...

[shoor]

When I first heard about background radiation, I thought to myself, didn't George Gamow predict that in one of his Mr Tompkins books which I read either in high school or junior high school? (and these books were written for juveniles). In fact Gamow did predict it. I read years later in Timothy Ferris's book "The Red Limit, The Search For the Edge of the Universe" that Gamow was astonished that the discoverers of the background radiation did not credit his insight. The book also mentions various scientists who were aware of the theory predicting background radiation but who didn't make the connection. Some of them apparently admitted feeling stupid about it afterwards. I guess I get worked up about this because Gamow actually gave a talk at the college where I was a student perhaps a year before he died. I had looked forward to seeing him speak but he was obviously in very bad shape. Thinking about it now, I don't know if it was in the Mr Tompkins books specifically, but it was in a book of Gamow's written for laymen. I remember he had something he called ylem. So, if I, a layman could make the connection as soon as I heard about the detection of the background radiation, how come the experts couldn't? This is something that used to amaze me, but I've come across enough stories of experts screwing up over the years (just saw a documentary about Bernard Madoff for instance), that nowadays I'm less amazed. Still, what can you do? Nothing is certain; we're always playing the odds with any information we're given.

Re:Sometimes screwing up leads to success ...

[s2theg]

"Sometimes screwing up leads to success." the moral of the story here is: discover, don't invent.

That failure leads to success is hard coded into the Scientific method.

Part of success is being wrong, facing it, and adjusting your assumptions to fit the facts.

Re:Sometimes screwing up leads to success ...

[Anonymous Coward]

Wow, how apropos. This article explains the East Anglia University CRU folks and other global warming alarmists "doctoring" their data to prove their pet CO2 theory. The facts now appear to strongly suggest that cosmic radiation penetration of earth's atmosphere - regulated by the solar wind - is what controls cloud development and cover, and thus the global climate of earth. About 1 Billion years worth of ocean floor sediment data strongly backs up this new theory. If this is new to you, please do some research or watch this video.
http://www.youtube.com/watch?v=dKoUwttE0BA

http://en.wikipedia.org/wiki/Henrik_Svensmark
Is he the next Darwin?

Ridiculous

[MrMista_B]

"It wasn't uncommon for someone to spend a month on a project and then just discard all their data because the data didn't make sense."

That doesn't mean the data is wrong, it means the /hypothesis/ was wrong, if not the theory, and needs to be modified.

If they're really throwing out date just because it 'doesn't make sense', they're doing religion, not science.

Re:Ridiculous

[wizardforce]

If your equipment is malfunctioning, you may end up with data that is fairly random where there should be some pattern or your measurements on your controls don't remotely match the values they should be. As an example, a standardized solution tests for a markedly different concentration than it should; a good sign that something is wrong. Things go wrong occasionally. That is why it is imperative that experiments be repeatable and have good experimental design.

Re:Ridiculous

[MyLongNickName]

And this is what bothers me. If you are willing to run an experiment enough times, you will eventually get data to support your assertions. Get a statistical 90% certainty, and it could be that you ran the scenario 100 times, and throw out the 99 times that did not give you this certainty. The scientific process is bullet proof. The folks who "do science" not necessarily so.

Re:Ridiculous

[Shadow of Eternity]

Not always, sometimes your data doesn't make sense because you made a mistake somewhere that wound up turning your results into garbage.

Re:Ridiculous

[labnet]

Often the data is crap, because the measurements are so hard to make.
For example, you would think measuring temperature is easy. Not so.
Lets say you wish to determine the cooling capacity of an airconditioner.
How do you measure the temperature and air velocity gradients across both the return and supply air streams. Do I use 1 sensor, 10 sensors, 100 sensors. Do you create turbulence or laminar flow? How accurate is the humidity measurement?
The point is, the data is often crap, because measurements are hard to make, time is limited, can't afford the right equipment, not enough labour, could not fully simulate the enviroment etc etc.

Re:Ridiculous

[Anonymous Coward]

I'd like you to take your scientific method fail and go test it by jumping off a building. Don't worry, if you repeat it enough times you'll get data to prove that you can fly.

Re:Ridiculous

[Anonymous Coward]

Yeah, I remember when I was in college I used to throw a lot of dates out because they didn't make any sense.
Usually the dates threw me out, but that is another thing.

Re:Ridiculous

[stms]

Scientist are bullet proof, I have a 90% certainty of that.

Re:Ridiculous

[TapeCutter]

"If you are willing to run an experiment enough times, you will eventually get data to support your assertions."

Yes, I belive Edison tried over 5000 different hand made bulb/filiment combinations before he found one that supported his assertion.

Thowing out data is not about proving pet theories, it's about admitting you cocked up the experiment. eg: Prof Sumner Miller never edited out failed demonstrations from his TV show, nor did he claim the failed demo proved accepted theories of physics were wrong, rather he would simply exclaim - "Experiments never fail, it is I who have failed to set the right conditions for nature to cooperate" and then try again.

Re:Ridiculous

[wizardforce]

If there's nothing wrong with your equipment or procedures and your experiment fails to find X then it's indicative that your hypothesis is wrong and any group that repeats your experiment is going to know about it immediately.

Re:Ridiculous

[nedlohs]

It can also be that their methodology is wrong, their equipment is wrong, they are simply incompetent.

I suspect that the main thing that differentiates scientists who make a major lasting contribution to science and those that just push knowledge slowly forward is whether they treat such problems as "Oh... that is interesting" or "Crap, must have screwed it up better start it over".

Of course I'm sure those in the first category spend a lot of their time chasing ghosts because most of the time they did in fact screw something up...

Re:Ridiculous

[rbannon]

Not religion, but federally funded dogma. More than 20 years ago I became aware of how dogma gets grounded in fundamental research: you need to write grants that fit the dogma. One hapless soul actually stood up during a big AIDS conference and suggested that the researchers were mere lemmings. He, of course, was shouted down, but he was only trying to tell the lemmings to keep an open mind. Fast forward 20+ years and the lemmings are still in control.

Our educational system is totally broken when the educated just want things to fit. Even in mathematics, we're promoting a crop of "just tell me what to do!"

Re:Ridiculous

[KliX]

If the search space was that simple, we'd know everything by now.

Re:Ridiculous

[Anonymous Coward]

That's ridiculous. Experiments are designed to test hypotheses. However not all experiments succeed, where success means "provides significant information confirming or contradicting the hypothesis". Designing and executing meaningful experiments is hard, and it's something that smart people routinely fail to pull off.

Re:Ridiculous

[Culture20]

The scientific process is bullet proof. The folks who "do science" not necessarily so.

What exactly are you advocating?

Re:Ridiculous

[BlueParrot]

"It wasn't uncommon for someone to spend a month on a project and then just discard all their data because the data didn't make sense."

That doesn't mean the data is wrong, it means the /hypothesis/ was wrong, if not the theory, and needs to be modified.

If they're really throwing out date just because it 'doesn't make sense', they're doing religion, not science.

a) You've clearly never done any real research or you would be well aware of the hundreds of millions of ways you can screw up an experiment and get nonsense data ( bad machinery, you wired up a detector wrong, the cell lines you were feeding vitamin K happened to get contaminated by bacteria halfway through etc... )

b) There is almost never a clear difference between data and theory. The only raw data you have is a bunch of numbers on a piece of paper, in order to determine if they correspond to your theory or not you need to interpret the numbers somehow, and it may just as well be the interpretation that is wrong as is the theory you were trying to test using the interpreted data.

c) Because you are often restricted by cost and time it's often not feasible to do a full analysis of why your experiment did not work. Hence if you did not get any useful results ( uncertainty was too large, it seems obvious you must have messed up somewhere etc.. ) then frequently the only sane option is to conclude your experiment was a failure.

d) If scientists followed your advice we would never have got the electronic equipment you used to make your post.

Basically your ideas about what science is or should be are extremely naive and to anybody who has done even a high school chemistry experiment it should be clear you have no idea what you're talking about.

Re:Ridiculous

[RobertF]

Anyone who has spent time working in a lab knows that not all data is equal. You can get useless results if something isn't quite as clean as necessary, or perhaps you were in a bit of a rush and didn't connect everything perfectly. Any interesting experiment usually has numerous points at which humans can mess things up. Errant data is usually a sign that you have improperly set up the experiment, so you'll spend most of your times reviewing and fixing procedures until you get what you expected.

Re:Ridiculous

[Anonymous Coward]

It makes no difference unless you can get EVERYONE ELSE to agree to throw out each of their 99 times as well.

Until then, you're just a Crackpot.

Re:Ridiculous

[Anonymous Coward]

The bulletproof, 'scientific' statement is something like "if you hook up the apparatus exactly so, then you will get these results," when you really want a meaningful statement like "cosmic ray muons travel at 0.99c." These are very different claims.

The whole point throwing out data is because "hooking up the apparatus exactly so" does not seem to correspond to measuring the speed of cosmic ray muons.

Re:Ridiculous

[Thing 1]

The scientific process is bullet proof. The folks who "do science" not necessarily so.

What exactly are you advocating?

That, in battle against an opponent armed with a firearm of some sort, one should adorn one's self with the scientific process, and not the folks who are following it, for greatest effect?

Re:Ridiculous

[Arancaytar]

It is possible to end up with crap data because the premise of your experiment is wrong. You can ignore a variable that should have been controlled or kept equal, or you can measure the wrong variable.

You can also end up with data that neither confirms nor denies your hypothesis, because it allows no statistically significant conclusion.

Re:Ridiculous

[Anonymous Coward]

Lesson #1 for any scientist:

Nature does whatever the hell it wants to. The best that a scientist can ever do is try to model it. If the model doesn't fit, it is wrong. Nature is always right.

Lesson #2 for any scientist:

Don't discard data. If there is a systematic error, then verify that it skewed the data and try to get data without that error. If there is no significant systematic error, then the data is correct.

Lesson #3 for any scientist:

Be humble. Pride, past successes, and the need for it to be right do not affect whether a theory is correct.

Re:Ridiculous

[Anonymous Coward]

I'm a scientist. Throwing out data is not good practice. Instead, some view 'weird' data as a great opportunity for further theorizing, which is probably how it should be considered.

Nonetheless, under some conditions I could understand throwing out data. When there's too much of it, you need to focus analysis on that of which you can make sense. If nothing makes sense, then there's nothing to contradict or confirm.

Re:Ridiculous

[steelfood]

Sounds like they're in need of engineers to design those experiments.

Re:Ridiculous

[Shadow of Eternity]

And then eventually they'll have a child and that child, born of science and engineering, will be our new god.

Re:Ridiculous

[DriedClexler]

Very true. Sometimes the data really are wrong. (The Minimum Discrimination Information criterion is a way of rigorously answering the question of whether your data or your theory is in error.)

But simply throwing out the data is the 100% wrong thing to do. That destroys the information that would have eventually told you if you were really doing something wrong in your experiment, or if you've discovered something new.

It also creates an information cascade-type situation where everyone culls any non-conforming data they see, and then all available data is conforming, which makes later scientists more skeptical of future non-conforming data, and so on. This cascade can make it so that even a randomly-chosen theory could be supported by the literature, even in the face of being completely unrelated to reality.

Supposedly, that's what happened with the Millikan oil drop experiment and everyone biasing themselves in the direction of his initial, wrong value for electron charge.

Re:Ridiculous

[honkycat]

Your post is spot on. I'd mod up, but I wanted to clarify (I think you'd agree) that there's a difference between a successful experiment that is inconsistent with a theory and a failed experiment. The purpose of an experiment is not to prove a hypothesis, it's to TEST a hypothesis (or to gather data toward that end). Success means you make a useful statement that aids in the test. Failure means the data were not useful. It has nothing to do with the correctness of the theory or hypothesis.

In the specific quote mentioned, the data "not making sense" doesn't mean that they disagreed with what the experimenter was expecting, it means that they came back in a way that "couldn't happen." That is, that something had gone wrong making the experiment a failure. For example, in some tests I was doing a couple years ago with a prototype radio receiver, I needed to measure its noise level. As a signal, I would sweep a resistive load up and down in temperature---the load outputs noise with intensity that depends on its physical temperature. In this case, as a check, I would start with the load at a low temperature, then heat it past the point of interest, and then cool it back to the starting temperature. I would measure twice, once on the way up and once on the way down. What I found was that the results disagreed between the two measurements. That "does not make sense" in the sense of the article---the testing method was flawed.

In a sense, it was a successful test of a hypothesis. The hypothesis was that the receiver behaves in a particular way (which is what you'd consider the REAL hypothesis under test) AND that the test setup was a valid way to measure that. I disproved the joint hypothesis. In this case, it was the latter part that was invalid---the test was invalid---and I could say nothing about the receiver. This was simply a failed experiment. There is no religion going on by my not claiming that receivers don't behave as we think they do when I just discarded my results.

Every now and then, the reason for a failure might be interesting. This is rare, but when it happens can be responsible for amazing discoveries. In my case, it was a problem of thermal equilibrium. My devices were operating in a vacuum at very low temperatures (about 20 Kelvin) and it can be difficult to affix a heater or a thermometer to just the part of a device that you want to heat or measure....

The OP's statements mirror the general misunderstanding of the scientific method that is rampant in the non-scientific community. We need to help people understand this.

Re:Ridiculous

[sarkeizen]

Ok I'm really not clear on what you're trying to illustrate here.

Sure you can get a minority result from an experiment but the less likely the outcome, the more costly it is to get those results and the less likely anyone else who repeats that experiment will see the same result set.

In other words if you tried to rig a test with a 95% confidence level. You would have to run the same test twenty times just to guarantee a result outside your confidence interval *but* that doesn't guarantee that the result shifts in the direction you wish the result to go.

Not only that but any other group who repeats that experiment is more likely than not to end up with a result that differs from yours. Worse yet, if they happen to have a significantly larger N (or some other feature that would tend to improve accuracy) then it's your experiment not theirs that will tend to be looked upon skeptically.

So what exactly is bothering you here? That people rig results by spending twenty times the amount of grant money? At those prices and at that risk you might as well just "shape" your data or some other kind low-tech scam.

An actual thing to be concerned of is *NOT* the idea that there is some slight chance of fixing results but rather that there is no obligation to publish. So that someone who funds a study who doesn't like the outcome can make sure that it doesn't see the light of day. This is a real problem in fields like medicine.

Re:Ridiculous

[ArcherB]

The scientific process is bullet proof. The folks who "do science" not necessarily so.

What exactly are you advocating?

Maybe he is saying that as long as scientists stand behind science, they'll be fine when the bullets start flying. When they ignore science and try to do their own thing, they get a cap in their asses.

Re:Ridiculous

[The_mad_linguist]

Edison didn't believe in Ohm's law, so it isn't surprising it took him that long to get his labmonkeys to make something useful.

Re:Ridiculous

[ibsteve2u]

Sorta kinda. Per Harold Evans' book The Spark of Genius as quoted in U.S. News & World Report

The real secret, Edison found, arguing it out with Charles Batchelor, was to raise the voltage to push a small amount of current through a thin wire to a high-resistance filament. It was an application of the law propounded in 1827 by the German physicist George Ohm, but it was still imperfectly understood. Edison himself said later, "At the time I experimented I did not understand Ohm's law. Moreover, I do not want to understand Ohm's law. It would stop me experimenting." This is Edison in his folksy genius mode. Understanding the relationship linking voltage, current, and resistance was crucial to the development of the incandescent lamp, and he understood it intuitively even if he did not express it in a mathematical formula.

Re:Ridiculous

[dcollins]

This is well known and called the "file drawer effect" (or publication bias).

http://en.wikipedia.org/wiki/Publication_bias

"In September 2004, editors of several prominent medical journals (including the New England Journal of Medicine, The Lancet, Annals of Internal Medicine, and JAMA) announced that they would no longer publish results of drug research sponsored by pharmaceutical companies unless that research was registered in a public database from the start.[11] In this way, negative results should no longer be able to disappear."

Re:Ridiculous

[oldhack]

Depending on the area of research, it's difficult to tell which bit is off - which source, which equipment and their calibration, ambient parameters, etc. Acoustics is a well-known example.

This is one among many reasons to record everything in your notebook and to not discard anything.

It's true, though. Scientific research is one frustrating line of work, but the reward, when you hit on it, even minor ones, make it all worthwhile for those with talent and aptitude.

If I had such a talent, I wouldn't be spending my life writing crummy software code to throw off QA freaks.

Re:Ridiculous

[Anonymous Coward]

If they're really throwing out date just because it 'doesn't make sense', they're doing religion, not science.

Religion never makes any sense.

Re:Ridiculous

[Rich0]

Yup - good controls are critical to good science.

They're especially important when the work is more routine, because that's when you're more tempted to just believe results because they look like what you expect. Or, perhaps you're talking about quality control testing and have a financial interest in having the results pass.

The only way to know that your results are actually right is to have a controlled analytical process. There are lots of ways to do that, and there is a whole field of pursuit around minimizing the cost of doing all those controls. However, the important thing is that analytical data must be controlled.

Re:Ridiculous

[KnownIssues]

The article says nothing about scientists keeping bad data simply to support a hypothesis. This is what separates it from religion.

Re:Ridiculous

[Anonymous Coward]

But how good are most of us at formulating new hypotheses? I am inclined to interpret that claim that the the data didn't make sense as an admission that I can't make sense of the data, which is something else. However, useful interpretation of such data does requires a hypothesis of some kind.

If you can't think of a new hypothesis, or if you can think of one but not of a valid scientific method to test it, then it may be best to move on, because data that cannot be understood are of little use to to the scientists that generated them. (Perhaps they might be to someone else, but they are almost impossible to publish.) I concede that people who can never think of a new hypothesis are not good scientists, but sometimes even the best are just baffled.

Besides the threshold of finding a new hypothesis, there is also the problem of having it accepted by your colleagues. I once lost a battle to get a new hypothesis inserted in an article, not even because its merit was questioned, but purely on the ground that its explanation involved mathematical formulas, and I was assured that Journal X would never print mathematical formulas. I was told that in the branch of science concerned, that just wasn't done. And no, I am not that old: This was in 2008.

There also is the discouraging factor that most hypothesis turn out to be wrong, even if great care is taken in constructing them. Most laboratory scientists have some confidence in their ability to generate valid data, but writing down a hypothesis is a larger risk...

A recent paper by Munos (Nature Reviews Drug Discovery of December) points out the universal failure of the various methods that the pharmaceutical industry has tried to predict success or enhance the probability of success. Despite being motivated by expense bills that run in the hundreds of millions of dollars, the number of successes is primarily linked to the number of attempts, with very few indications that human attempts to outsmart nature are of any avail.

Re:Ridiculous

[Anonymous Coward]

That, in battle against an opponent armed with a firearm of some sort, one should adorn one's self with the scientific process, and not the folks who are following it, for greatest effect?

He's dead, Jim.

Jim: "Pity, he was a good research assistant. Made good coffee and muffins, too."
 

Re:Ridiculous

[Simetrical]

"In September 2004, editors of several prominent medical journals (including the New England Journal of Medicine, The Lancet, Annals of Internal Medicine, and JAMA) announced that they would no longer publish results of drug research sponsored by pharmaceutical companies unless that research was registered in a public database from the start.[11] In this way, negative results should no longer be able to disappear."

But I guess research by anyone other than pharmaceutical companies is allowed to disappear? I guess researchers who aren't tied to corporations never have pet theories that they would like to see come true because they've invested the last ten years of their life in it, for instance? Or for social or moral reasons?

All studies should be published, that anyone conducts. Maybe just put up on the web with a note to the effect of "We obviously messed this one up pretty bad, huh?", but put somewhere. That's the only way to ensure that metastudies aren't hopelessly corrupted by publication bias. Which can be due to researchers throwing out data because they think they messed it up, research funders throwing out data because it doesn't help their bottom line, reviewers rejecting studies because they don't like the conclusion or the researcher isn't prestigious enough, lots of things.

Re:Ridiculous

[Simetrical]

Our educational system is totally broken when the educated just want things to fit. Even in mathematics, we're promoting a crop of "just tell me what to do!"

As a grad student in pure mathematics, I'm curious: do you mean in low-level math education, or mathematical research? Basic math education is often just about giving you the tools you need to do your job, so there's nothing wrong with just telling people what to do. Higher-level courses (meaning the kinds only pure-math majors typically take) do require you to actually understand the material and be able to prove things from first principles, probably far more so than any other field.

You never discard the data

[techno-vampire]

If the data don't make sense according to your theory, you don't discard the data, you discard the theory and work out a new one that fits the facts as you've observed them. TFA says that Dunbar was watching postdocs doing research, and if so, they should have known better. Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

Re:You never discard the data

[Tablizer]

Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

Are you trying to tell me that Underoo-wearing mice don't jump higher?

       

Re:You never discard the data

[A beautiful mind]

Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

It's just a result of how science is performed. Science doesn't have low hanging fruits anymore, consequently any problem that someone investigates takes dedication, because it's intellectually hard or takes lots of effort or both. Most people aren't going to be motivated enough to put that much effort into it without already having an axe to grind, a point to prove, a pet theory to push into the limelight.

Also, in a lot of cases you don't know there is something interesting in the area you're looking at. I think what separates bad scientists from good scientists is how you realize when something doesn't match up to your preconceived notions and how you recover from conflicting data.

Re:You never discard the data

[topham]

The problem is you generally do not get money to simply study X.

You get money to show that X affects Y in manner Z. If X doesn't affect Y in manner Z they pull your funding, give you a failing grade, or otherwise find ways to punish the results.

They do this over and over again and then wonder why researches fake data, toss good data out and re-do the study looking for results they want instead of what is.

Want to do a drug study that says Drug A is safe to fight cancer? Got results that indicate an increase risk of heart attack? Have the study declared flawed, re-do the study with a slightly different mix of subjects and repeat. With luck your new study shows the heart attack risk is below the error threshold of your study and you can ignore it. Release your drug, make your millions and, after you leave have the real-world implicates show up on the 5 o'clock news.

Re:You never discard the data

[caramelcarrot]

As other people have pointed out - sometimes the data is just crap due to the difficulty of making measurements. Sometimes you've measured something other than what you actually need to compare to theory, sometimes there's too much noise. The skill of a great experimentalist is being able to take good enough data that you can't justify ignoring it if it comes out different to what you expected.

Re:You never discard the data

[bcrowell]

If the data don't make sense according to your theory, you don't discard the data, you discard the theory and work out a new one that fits the facts as you've observed them. TFA says that Dunbar was watching postdocs doing research, and if so, they should have known better. Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

This is a beautiful explanation of how science is supposed to work. In reality, science doesn't really work this way. It doesn't work this way in my experience as a scientist, and it doesn't work this way if you read the history of science.

For some good historical examples, see Microbe Hunters, by de Kruif (one of the best science books of all time, although you have to look past the racism in some places -- de Kruif was born in 1890). A good example from physics is the Millikan oil-drop experiment, where he threw out all the data that didn't fit what he was trying to prove -- but then claimed in his paper that he'd never thrown out any data. Galileo described lots of experiments as if he'd done them, even though he didn't actually do them, or they wouldn't have actually come out the way he described.

Michelson and Morley set out to prove the existence of the aether, published their results believing they must be wrong. Nobody else believed them, either. Various people then spent the next 30 years trying to fix the experiment by doing things like taking the apparatus up to the top of a mountain, or doing the experiment in a tent, so that the aether wouldn't be pulled along with the earth or the walls of a building. By the time Einstein published special relativity in 1905, most physicists had either never heard of the MM experiment, or considered it inconclusive.

When your results come out goofy, 99.9% of the time it's because you screwed up. You don't publish it, you go back and fix it. If every scientist published every result he didn't believe himself, the results would be disastrous. If you try over and over again to fix it, and you still fail, only then do you have to make a complicated judgment about whether to publish it or not.

The way science really works is not that scientists are disinterested. Scientists generally have extremely strong opinions that they set out to prove are true using experiments. The motivation is often that scientist A dislikes scientist B and wants to prove him wrong, or something similarly irrational, personal, or emotional. The reason this doesn't cause the downfall of science as an enterprise is that there are checks and balances built in. If A and B are enemies (and if you think the word "enemies" is too strong, you haven't spent much time around academics), and A publishes something, B may decide just to see if he can screw that sonofabitch A over by reproducing his work and finding something wrong with it. It's just like the adversarial system of justice. Society doesn't fall apart just because there are lawyers willing to represent nasty criminals. Einstein was famously asked what he would do if a certain experiment didn't come out consistent with relativity; his reply was that then the experiment would be wrong. Einstein fought against Bohr's quantum mechanics for decades. Bohr fought against Einstein's photons for decades. They were bitter rivals (and also good friends). It didn't matter that they were intensely prejudiced, and wrong 50% of the time; in the end, things sorted themselves out.

Re:You never discard the data

[dwguenther]

The people who are not motivated enough to put in the effort are not scientists - they are pundits. Researchers who are truly interested in their work - and that would be most of them - put in decades of observation and analysis looking for some truth, because simply grinding an axe would never be personally satisfying. It is lazy and disrespectful of you and other armchair commentators to simply dismiss all that work with a three-line opinion.

Re:You never discard the data

[Rising Ape]

If the data don't make sense according to your theory, you don't discard the data, you discard the theory

Not really, assuming the theory is something well-established and tested. Popper oversimplified things - experimental data is rarely so unambigous that you can outright discard a reliable theory. It's much more likely that you messed up than you proved it wrong, or maybe the theory needs a fairly minor modification rather than complete rejection.

That's no reason to discard data though - not until you understand *why* the discrepancy arises, or at least have established that the data is unreliable in some other way. If, despite diligent effort, you can't find anything wrong with your analysis, then you publish and see if anyone else can explain it. The evidence may well then accumulate to the point at which the original theory is untenable, or alternatively it may demonstrate a different explanation for the original result that doesn't invalidate the theory at all.

Re:You never discard the data

[nedlohs]

Most of the time that the data doesn't make sense it's because the scientist fucked up and didn't calibrate the sensor, or mislabeled a sample, or made a tpyo.

But yes, discarding it is the wrong thing to do. Repeating it again (and if you now get what you were expected, doing it a third time) is usually the thing to do.

Of course if it is expensive, then just write down the results the theory predicts ith a fudge factor for error. Better to retard humanity's knowledge of the world than to maybe look silly.

Re:You never discard the data

[dwguenther]

This comment displays a vast ignorance of how research actually gets done. Most research is funded to gain knowledge, regardless of the result (otherwise it would simply be called 'knowing', not research). Note for instance that the drug companies continue spending millions dollars on basic research year after year, even when they don't get an immediate result. So don't broad-brush the dedicated work of hundreds of thousands of scientists with your own questionable view of ethics.

Re:You never discard the data

[dwguenther]

You may be the first actual scientist I've seen post in one of these discussions. That's a very well written explanation of the realities of research.
Mod this guy up!

Re:You never discard the data

[syousef]

If the data don't make sense according to your theory, you don't discard the data, you discard the theory and work out a new one that fits the facts as you've observed them.

If it's a well established theory, you want to eliminate sources of the error before trying to overthrow it. For every Einstein that moves us to the next level from a well established theory there are 3 million cranks that just can't set up a well controlled experiment to save themselves. If you've conducted the experiment sufficiently badly the chances of working out what happened are nil. What you do is repeat the experiment correcting the errors and see what you get.

Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

It's a human frailty. Einstein wasted the later half of his career because he believed "God does not play dice", rather than accept Quantum theory. What a pity superstition had to come into it. A decade after his death Bell's Theorem has proven him wrong.

Re:You never discard the data

[Anonymous Coward]

If the data don't make sense according to your theory, you don't discard the data, you discard the theory and work out a new one that fits the facts as you've observed them. TFA says that Dunbar was watching postdocs doing research, and if so, they should have known better. Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

Not a climate "scientist", are you?

Re:You never discard the data

[rkfig]

Wow. Well spoken, good information, and level headed. If I hadn't noticed the user ID, I would say that you are new here. Thanks for the input.

Re:You never discard the data

[techno-vampire]

Not a climate "scientist", are you?

No, I'm not, and I don't play one on TV, either. I am, however, glad you asked, because soi disant "climate scientists, especially those of the CRU are exactly the people I was thinking of.

Re:You never discard the data

[Anonymous Coward]

Recovering from conflicting data often (though not always) requires discarding bad data, because things go wrong---samples get contaminated, measurement apparatuses get miscalibrated, the planets don't line up just right (yes, I'm kidding---except maybe if it's some kind of astrophysics experiment...), etc. In those situations the conflicting data are simply invalid, being taken from a (necessarily) flawed apparatus/environment, and it's perfectly reasonable to discard them (or more likely, label them "bad", keep them somewhere out of the way, but not publish them), make any fixes you can, and try again.

The question is how to recognise when your conflicting data are actually real. This is what takes so much time and effort to eventually determine, and while it does take longer, compared to confirmation of existing theories, science inevitably gets there eventually.

Re:You never discard the data

[Metasquares]

It is lazy and disrespectful of you and other armchair commentators to simply dismiss all that work with a three-line opinion.

Doing precisely this is one of the more distasteful parts of most scientists' jobs.

Re:You never discard the data

[Anonymous Coward]

Nonsense. I've known quite a few researchers who've worked to find Least Publishable Units acceptable to funding agencies and who've discarded data that wouldn't help them get the next grant. Not that I have huge amounts of respect for them, but hang around in most universities for a while and you'll find more of them than you'd like.

Re:You never discard the data

[fotoguzzi]

A good example from physics is the Millikan oil-drop experiment, where he threw out all the data that didn't fit what he was trying to prove -- but then claimed in his paper that he'd never thrown out any data.

Another take on Millikan: http://www.americanscientist.org/my_amsci/restricted.aspx?act=pdf&id=2706085559588

Re:You never discard the data

[bcrowell]

There's a paywall. Maybe you could summarize for us?

Re:You never discard the data

[fotoguzzi]

An accessible link on defending Millikan: http://eands.caltech.edu/articles/Millikan%20Feature.pdf

Re:You never discard the data

[Mutatis Mutandis]

Yes, but that pre-supposes the ability to invent a new theory, because scientists are very unwilling to discard a theory if they have no alternative. After all, having no theoretical framework at all is very uncomfortable.

And there I do think that Dunbar makes a perfectly valid point: Any group of specialists who are all of the same mind is very bad a thinking "out of the box" and inventing a new theory. To be able to do that, you need a healthy mixture of different backgrounds, and enough dissent to stimulate the debate. Unfortunately scientists often assemble in excessively homogeneous groups, sometimes on their own initiative ("old boys network") and sometimes through deliberate but foolish policy ("center of excellence").

This is part of the reason why publishing and attending scientific congresses is a vital part of scientific activity. I've often noticed that in industry, this is regarded as a kind of bonus, a perk that allows scientists to travel to nice places. But is absolutely essential, even if that means talking things through with direct competitors.

Re:You never discard the data

[nine-times]

Have you ever done any kind of experiment? I remember back in college doing a bunch of relatively simple stuff, like testing the rate of acceleration due to gravity. Half the class got bad data.

Sure, they were just students, but these were really simple experiments, designed with full knowledge of what the results should be, and designed specifically to illustrate scientific principles to students. People were careful, the equipment was adequate, and still they got bad data. Imagine how much harder it would be if you don't really know what your results should be, and you're making up the experiment without even being sure that your experiment will be capable of testing what you want it to test.

I bet people get crap data all the time. If you get data that's all over the place, no apparent pattern, then there isn't necessarily a theory to work out. It probably means that your experiment is bad or you performed it badly. Even if the experiment was good, it means there's probably not a strong connection between the variables your testing and your results, or else there are other variables you're not controlling for. It's kind of back to the drawing board at that point.

Now if the results weren't random, but were consistently showing a trend contrary to the established theory, then scientists shouldn't toss that data. I doubt that's what we're talking about here, though.

Re:You never discard the data

[Quirkz]

If the data don't make sense according to your theory, you don't discard the data, you discard the theory and work out a new one that fits the facts as you've observed them.

I'm more computer science than research science, but I've seen plenty of examples where throwing out data is the sensible thing to do. Computer analogy time:

Sometimes an application isn't working. So you set up experiments, adjusting settings, trying to find out what's wrong with the application. You get a good setting, for ten minutes, but then it goes bad. So you turn it back to the old setting that used to mostly work, but had one flaw. Now that flaw is gone, but you're getting a completely different error. The data itself is inconsistent and nonsensical. That's when you step back, reconsider what you think you know, and possibly throw out all the data as worthless.

Why? Well maybe the software isn't bad at all. It's actually a memory problem or a bad hard drive. Something else isn't the way you expected, so everything it seemed like you were learning was worthless.

Now if bad data leads you in the right direction, that's still valuable, even if you throw the data out in the process. Other times bad data eventually leads you to a human error (troubleshooting that network problem was unnecessary once you realize you didn't have the cable plugged in) and you've got to start over.

Re:You never discard the data

[bcrowell]

Thanks for the link. The crucial part seems to be on p. 8. I just don't find his argument very plausible. Goodstein has to resort to "What he means to say is ..." I've tried repeating the Millikan oil drop experiment using student-lab equipment, and it's an extremely difficult technique. I find it plausible that the technique can be used (1) to determine e based on a tendency for the charges to clump around certain values, but I don't find it plausible that it can be used (2) to disprove the existence of intermediate values of charge. It's #2 that Millikan really wanted to do, and he clearly made false statements in order to do so. Goodstein says that Millikan's selection of which points to publish didn't "bias" his result. I don't understand what he means by this. The graph on p. 7 shows that he threw out more points that disagreed with his hypothesis, and fewer points that agreed with it. This seems to clearly indicate that the selection biased the result.

obligatory (insert famous movie here) comment

[Anonymous Coward]

Question *Everything*.

Throw away data?

[Anonymous Coward]

Proving a theory incorrect is often just as valuable as proving a theory correct.

Re:Throw away data?

[MathiasRav]

Proving a theory incorrect is often just as valuable as proving a theory correct.

I'd rather say proving a theory incorrect is just as valuable as proving a *hypothesis* correct. If it's a hypothesis, it's no fun proving it wrong (it wasn't established anyway, it might go against your intuition but nobody cares), and if it's a theory, it's no fun proving it right (what are you talking about, of course it's right, we already knew that).

I would elaborate on this but that would just be filler.

Re:Throw away data?

[Thing 1]

(Your parentheses elaborated.) And -- completely agree. :)

Good!

[RyanFenton]

If problems occur as you postulate elaborate hypothesis, then stop piling up the elaborate hypothesis! But be sure and still make available your existing (complex) hypothesis, methodology and unexpected data - preventing others from going down the same path with the same methodology is still highly valuable!

Let's say you're looking at a production and consumption cycle involving neurotransmitters and neuroreceptors of some sort, and the various channels of input and output involved. Your starting presumption you base your hypothesis on is that there is a buildup which triggers an electrical signal to stop consumption and clear the channel. The only evidence you can realistically gather for now is protein density at a certain output channel - but others have worked to ensure this is a reliable approach specifically under these circumstances.

So, you do the specific experiment, trigger the signal, but you get a wildly different result - the stop in consumption occurs, but the protein density does not change at all in the output channel. What actually happened is still unknown, only you haven't verified any correlation with your hypothesis. You still have valuable data, but no mechanism to verify under the circumstances. Either your methodology failed, or you misunderstood what was happening - and the world of knowledge is made larger by either... even if your paymasters won't get happy about the result.

Science is often like throwing pebbles in complete darkness - it takes a lot of stones and close listening to make out a mental picture of the scene - especially when there's a lot of noise already around. Everyone would love it if we could just flip the lights on - but we have yet to invent a light that can see into the inner workings of the functioning brain very well. Gotta keep throwing those pebbles for now.

Ryan Fenton

Re:Good!

[dcmoebius]

Neurotransmitter consumption cycles? Come on, if you want the +5 Insightful, you really need to couch your hypotheticals in terms of cars.

Re:Good!

[Thing 1]

[...] preventing others from going down the same path with the same methodology is still highly valuable!

Exactly. Thomas Edison "discovered" over 5,000 ways how not to create a light bulb. Had he published each and every one of them, perhaps the light bulb would have been invented sooner -- perhaps by someone else, or perhaps by him, collaborating with someone else who had read his published accounts of "how not to create a light bulb."

Or you can edit your data....

[sl149q]

Is it just me or does this sound like an explanation for some of the Climategate science... But in that case they just massaged or ignored data that didn't agree with their conceptual framework of CO2 causing global warming.

Not that the skeptics are all that immune. They seem to cherry pick data almost as well (just not quite as successfully from the POV of selling their story to the media and political left ..)

Re:Or you can edit your data....

[dwguenther]

Yeah, it's just you. AP News found no evidence of massaged or ignored data (http://news.yahoo.com/s/ap/20091212/ap_on_sc/climate_e_mails). So climate science is a poor example of this thesis.

Re:Or you can edit your data....

[Rising Ape]

By "almost as well" I assume you mean "all the time". The "sceptic" arguments are nothing but a parade of cherry picking with little attempt at genuine investigation.

And there's no real evidence of the proper scientists massaging or ignoring anything. Just because a detailed, written account of everything doesn't exist in stolen, incomplete private documents doesn't mean it doesn't exist at all.

Re:Or you can edit your data....

[khallow]

Where's the AP story for the computer code?

Re:Or you can edit your data....

[DNS-and-BIND]

The Associated Press are scientists now? Oy vey, what is the world coming to when commenters on science.slashdot.org quote the media as an authoritative source?

Re:Or you can edit your data....

[Anonymous Coward]

no evidence of massaged or ignored data

Well, not really. Quoting from the article you linked, they used tree rings when it supported AGW and ignore them when it didn't:

The "trick" that Jones said he was borrowing from Mann was to add the real temperatures, not what the tree rings showed. And the decline he talked of hiding was not in real temperatures, but in the tree ring data which was misleading, Mann explained.

Sometimes the data didn't line up as perfectly as scientists wanted.

Re:Or you can edit your data....

[J Story]

And there's no real evidence of the proper scientists massaging or ignoring anything. Just because a detailed, written account of everything doesn't exist in stolen, incomplete private documents doesn't mean it doesn't exist at all.

The behaviour surrounding the data is certainly indicative of a lack of confidence in the findings. Refusing FOI requests and claiming that "the dog ate it" do not show a group filled with the belief that their research is unassailable.

Re:Or you can edit your data....

[Anonymous Coward]

You mean besides how NASA's big charts of the world's average annual temperature from the past 100 years look vastly different now from how they did five years ago? Even if it's just being fixed due to error correction, it obviously isn't stable enough yet to draw conclusions from.

Re:Or you can edit your data....

[NeutronCowboy]

As opposed to a random blogger? Really? Be careful throwing around the authority argument, it might come back to bite you.

Re:Or you can edit your data....

[Anonymous Coward]

no evidence of massaged or ignored data

Well, not really. Quoting from the article you linked, they used tree rings when it supported AGW and ignore them when it didn't:

The "trick" that Jones said he was borrowing from Mann was to add the real temperatures, not what the tree rings showed. And the decline he talked of hiding was not in real temperatures, but in the tree ring data which was misleading, Mann explained.

Sometimes the data didn't line up as perfectly as scientists wanted.

Well, technically the GP post is correct.

That's neither "massaged" or "ignored".

However, "buried" would be an accurate characterization. Although I don't think that GP poster realized that when he fell for the cleverly-worded handwaving.

Re:Or you can edit your data....

[phantomfive]

The "sceptic" arguments are nothing but a parade of cherry picking with little attempt at genuine investigation.

Only if you don't actually look around. Richard Lindzen is a climate researcher at MIT, and has investigated it well (he was one of the authors of the IPCC report). His argument is that there is no strong evidence linking anthropogenic CO2 and a global crisis.

And he is right. Check out the evidence for yourself. Look at it critically, and try to see if they can establish a link. They can't.

Re:Or you can edit your data....

[Anonymous Coward]

I think what parent was saying was that the only authoritative source of science is from another scientist. Or in this case, the open scientific community. The media only reports the results, but doesn't make them reliable if it's from one unverified proprietary source.

Re:Or you can edit your data....

[Anonymous Coward]

All research is assailable to a sufficiently uninformed assailant.

I think that my research is pretty good, and when there's a problem I try to be honest about it - but I work in a field that the general public doesn't care about. If I knew that there were billions of dollars riding on the outcome, and thousands of people waiting to tear my arguments to shreds (and this is the important part) regardless of how bulletproof my work was, because it was more important for them to score political points than to understand the research before criticising it - then I'm not sure what I'd do. Leave the field, probably.

Re:Or you can edit your data....

[Anonymous Coward]

"His argument is that there is no strong evidence linking anthropogenic CO2 and a global crisis."

But Lindzen is not a "skeptic". He believes in AGW. He believes that the increases in temperature will not be "significant" and that the predicted model temperatures are unlikely to happen (the models have a great deal of unertainty). He has a defensible position. What comforts him makes me rather uneasy. Of course, both he and I will be dead before it really affects either of us.

Re:Or you can edit your data....

[Anonymous Coward]

Those "sceptic" arguments just plain stink.

It's called a "hard pivot" by politicians

[Anonymous Coward]

When you come up with a stinker of a health care plan that causes your 60th vote in the Senate to wind up 31 points behind the opposition, you call your fuck-up a hard pivot.

Better than discarding data

[Anonymous Coward]

"It wasn't uncommon for someone to spend a month on a project and then just discard all their data because the data didn't make sense."

No need to discard perfectly good data when all you need to do is adjust it a little. Don't they know about Mike’s Nature trick?

The problem... (maybe?)

[pieisgood]

I can't help but think that Neuroscience needs to calm down, sit back, and take a deep breath. We are examining a system and we are trying to reverse engineer it. We can't start out by trying to create elaborate hypothesis for large systems, we need to go low level and examine the simpler systems. I really think they should hold on to the higher cognitive models for a later time because we can't even completely model C. Elegans and it has the least neurons of any, current, living organism. The way I see it, I total expect their hypothesis to be wrong, because they don't thoroughly understand the low end of the system.

Re:The problem... (maybe?)

[neurophil12]

We've been at the lower levels for a while now, and we've uncovered quite a bit. We do need a lot more data at the molecular, neuronal, and local network level, but to develop that data without good theories for what those lower components are doing is a non-starter. We need to know what questions to ask and where to go from one result to the next experiment. Having theories for how individual components fit together into larger systems provides a framework for how to make those determinations. We would have a pretty hard time figuring out how local networks work without thinking about how those local networks fit into how the larger network or brain region works. In other words, having a few good working theories for the larger system makes it easier to determine what to test and look for in the simpler underlying systems.

Re:The problem... (maybe?)

[phantomfive]

Why not look at data wherever we can find it? If they find certain patterns that the human brain tends to go through, why not observe them and record them and understand them as well as possible?

It isn't always necessary to know the underlying, simpler systems before we get useful information. I can calculate the resultant change in velocity of two objects after a collision, even though I don't understand the full quantum-mechanical underpinnings of the collision. I know what a computer will do when I call the printf() function, even though I'm not sure of all the math behind transistor construction. Take data wherever you can find it.

Re:The problem... (maybe?)

[Anonymous Coward]

This is a fallacy of composition.

Kepler deduced important facts about planetary motion without understanding that gravity was a force or that planets were attracted to the sun.

Newton understood the interactions of forces and bodies, though he knew nothing of the atoms that made them up, or even the underlying structures of the universe which gave them force.

Darwin understood the overarching concept of the evolution of species, though he had no concept of what a gene was, what DNA was, or how cells copied themselves.

We don't need to understand the "low-end" of the system to understand to some extent its "high end" features. That's a bit too much reductionism.

Re:The problem... (maybe?)

[nine-times]

What you say sounds pretty smart to me. If we don't understand the simplest of nervous systems, then how could we understand the most complex? On the other hand, it does seem worthwhile to come up with elaborate theories of complex systems.

For one thing, we could analyze a simple system and come to completely incorrect ideas if we don't keep the more complex system in mind. Any theory of simple brains and nervous systems has to offer some room for explanation as to how they can get as complex as our own brains. To take it to the extreme, there's probably a limit to the knowledge that can be gained from analyzing a neuron on its own.

Or thinking of another subject, it's unlikely that Newton would have come up with universal gravitation by looking at the sun's apparent motion. He had to look at the whole universe-- the motion of all the stars and planets as well as looking at the motion of objects here on earth-- and then say, "These motions must all be caused by the same force." Sometimes it's worth looking at the whole complex system.

Besides that, the fact is that we probably care most about our own brains. In a lot of ways, we're studying these simpler brains in order to increase our understanding of our own brains. We want to be able to fix and control some of the problems in our own minds, and we want to replicate our minds. Insofar as that's the goal, it makes sense to constantly be looking at what we understand and what we can accomplish now instead of waiting for complete and perfect knowledge sometime in the future.

Re:The problem... (maybe?)

[Khelder]

So would you also advocate giving up on, say, weather prediction? Until we can come up with a model of the weather based on experimentally-tested theories such as how individual molecules of air move and interact?

It would be really helpful to me professionally if we had models of people that were more like our models of bridges, car engines, airplanes, integrated circuits, etc. But modelling humans explicitly as systems of millions of individual units (e.g., cells, neurons) is going to be a long time coming. Current models of human perception, congition, and so on, are far from perfect, but they're sitll a lot better than nothing.

posting to undo bad mod

[chris mazuc]

...

What a good article

[zogger]

It came late in the year but I would nominate this linked article as the best of slashdot, 2009.

Re:What a good article

[Anonymous Coward]

And I'm still not going to click through and read it.

Racism and science.

[tjstork]

So, when reality is racist, you change it?

I thought World War II empirically proved that the master race is not all its cracked up to be. American mutts and Soviet subhumans kicked the living shit out of the master aryan race. The whole concept of NAZI ideology was that they were the master race, they were not only deserving of victory, but destined, thus, by the most racist rules there are, they proved themselves inferior.

PS. Polish women are the hottest of all European women.

Re:Racism and science.

[Tynin]

PS. Polish women are the hottest of all European women.

Pics or it isn't true! That said, I'm sure the GP was trolling. But due to the PC movement (more specifically, our backlash to the PC movement) it is too easy to claim to be an authority of something like this and get people to believe their might be a hint of truth to it without him even providing any real details. If you condense the post down to its finer points, it quickly becomes obvious he is bashing people with brown skin, especially by the time you get to the end of the post.

Re:Racism and science.

[Jurily]

Seriously, what the fuck?

Re:Racism and science.

[Jurily]

it quickly becomes obvious he is bashing people with brown skin, especially by the time you get to the end of the post.

Without wanting to delve into the finer points of troll detection, that post does have a point: what happens when either the people working on the data have strong opinions about the outcome, or the people giving the funding do?

Re:Racism and science.

[Tynin]

No quick answers to that one other than to allow the research to kill itself overtime. Since no one but funders/researchers would be aware of any bias, no one could do much about it until they publish their findings. But once they publish, if it is bias then an independent review with repeated testing and evaluation against the original research should either show it to be legitimate or not. If the research was obviously faked, then I hope the "scientists" behind it and any other research they have done are closely reviewed as well.

Re:Racism and science.

[Tynin]

As soon as I hit submit, I realized I answered the wrong part of this question. I'm expecting them to publish, but what likely trolling AC was saying is they did the research but now armed with their findings they aren't publishing due to possible bias of the funders not wanting to hear that answer. I don't think we can combat this, as once again no one would know about it as it would be an inhouse secret. Unless a researcher on the team decided to go rogue and release it to wikileaks, I think no one would be the wiser.

Re:Racism and science.

[Anonymous Coward]

Just 'cause the Aryans weren't the so-called "master race" doesn't mean there isn't one. I think right now, it's a toss up between money and women.

Re:Racism and science.

[Anonymous Coward]

Just 'cause the Aryans weren't the so-called "master race" doesn't mean there isn't one. I think right now, it's a toss up between money...

Oh no! Jews might be the master race?!

...and women.

Oh no! Niggers might be the master race?!

Re:Racism and science.

[bidule]

PS. Polish women are the hottest of all European women.

I am sorry, but I worked with one such woman. She was built like a tractor and had shoulders of a linebacker. A very enjoyable person, but in no way hot.

Re:Racism and science.

[tjstork]

. She was built like a tractor and had shoulders of a linebacker. A very enjoyable person, but in no way hot.
--

Just think of her as a baby making machine. You got to sometimes let yourself be a total man-pig and look at women for the efficacy as breeding machines. I mean, there's no better sex then when you think you might get a woman pregnant. If it was not for social mores about father hood and my own religious sensibilities, I'd have a 100 kids at least simply because knocking up a chick is great.

Re:Racism and science.

[c6gunner]

She was built like a tractor and had shoulders of a linebacker.

Must have had Russian ancestry. You know what they say about the Russian girls .... strong like truk ... smart like traktor .... sexy like faktory!

Integrity?

[Anonymous Coward]

Throw out data or Reform Hypotheses should not be the first step when you collect data that doesn't make sense. This is the last step, not the first.
1)Check your instruments.
2)Check them again.
3)Check the calibration of said instruments.
4)Check system again.
5)Check the software, what assumptions are being made in the software? Are there misplaced decimals? Are your variable selections all appropriately classed/typed?
If you complete this checklist without coming across any failures, RECOLLECT DATA.
Compare new data with old data. What are your applicable/appropriate error measurements?
Perform steps 1 - 5 again.
Your sets of data should help you decide whether it is your hypotheses, or first data sets that need to be thrown out.

If you have been hired to prove something that is mostly false: gut check time. Are you a scientist or a really smart P.R. hack?

Anonymous Coward

[Anonymous Coward]

As a researcher myself, I certainly hope they don't throw out data too often. There is occasion to do so...sometimes, when trying to establish correlations (admittedly the weakest form of describing a phenomenon, etc), you learn that there is not one. There are times you obtain data that simply says, "These two phenomenon do not strongly affect each other" or "Something we do not know about or have not accounted for is happening all over this mess."

This data could be kept forever in the unlikely event it will prove useful, especially if there is something else going on...could be as simple as a RF/EM noise (which actually happened to a coworker of mine, though I helped to figure out the issue and make alterations to block/filter this noise out.) In previous years, data storage was sometimes at a premium, although lately this is not an issue as HDD climb to extraordinary capacities (until that capacity becomes the norm, then it is merely ordinary.) My point is that rejected or discarded data, at least in my experience, is due to situations such as these.

Things such as "massaging" or ignoring data are not only horribly bad scientific practice, they are a tremendous drag on humanity's progress...you usually learn through failure, but we are led away from the truth by practices such as those.

Bugs

[graft]

If the data doesn't fit your theory, the problem is most likely neither with the data (which is fine) nor with your theory (which may also be fine) but with the method you used to produce your data. You probably wired in an incorrect resistor, forgot to close a parenthesis in your Perl code, forgot to add the correct amount of EDTA to your reaction, etc. Then your results ended up looking like shit, and not surprisingly. Doing science is hard.

There's no need to postulate any grand conspiracies or take pot-shots at science in general. This paper is examining real people doing real shit. Most of the time we fuck up, and we're not smart enough to figure out where we made the error.

Re:Bugs

[sorak]

Exactly! When I was much younger than I am now, I was constantly finding "bugs" in things. Windows was buggy because the network connectivity was broken. VB was buggy because my commands were correct, but it was doing some crazy thing that made no sense.

In time, I realized that it was usually a mistake on my part. I'm not saying there are no bugs in Visual Studio, or Windows (I seem to remember the C++ STL library having one or two), but I have become a much better programmer, because the first thing I ask is "what did I do wrong". And that is exactly why most of these scientists are throwing their data away.

"The probability of me revolutionizing my field is low. The probability of me being a twit is much higher."

Seconded.

[Estanislao Mart�nez]

If the data don't make sense according to your theory, you don't discard the data, you discard the theory and work out a new one that fits the facts as you've observed them. TFA says that Dunbar was watching postdocs doing research, and if so, they should have known better. Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

This is a beautiful explanation of how science is supposed to work. In reality, science doesn't really work this way. It doesn't work this way in my experience as a scientist, and it doesn't work this way if you read the history of science.

Indeed. The sort of thing being discussed in TFA is one of the classic themes of late 20th century philosophy and history of science: the disconnect between traditional philosophy of science and the actual practice of science.

Kuhn's Structure of Scientific Revolutions is a good place to start. Just one tiny example of the book: Kuhn goes on about how during normal science, scientists perform experiments to confirm the results that they expect to get. When an experiment contradicts the theory, they don't automatically assume that the theory is wrong; on the other hand, they assume that the experiment was flawed.

Feyerabend and many other philosophers of science take a complementary stand to this by stressing the theory-ladenness of "facts." The claim that the "facts" contradict a hypothesis is never a theory-independent observation, but rather, the conclusion of a different theory that we may overthrow. Feyerabend's classic example is the Tower Argument that Aristotle used to refute the theory that the Earth moves. Wikipedia's article on Paul Feyerabend has a decent, if terse, explanation of this:

"The tower argument was one of the main objections against the theory of a moving earth. Aristotelians assumed that the fact that a stone which is dropped from a tower lands directly beneath it shows that the earth is stationary. They thought that, if the earth moved while the stone was falling, the stone would have been "left behind". Objects would fall diagonally instead of vertically. Since this does not happen, Aristotelians thought that it was evident that the earth did not move. If one uses ancient theories of impulse and relative motion, the Copernican theory indeed appears to be falsified by the fact that objects fall vertically on earth."

Feyerabend goes on to argue that many of our most successful contemporary scientific theories (e.g., heliocentrism and geodynamicism) became so because their Renaissance and Enlightenment proponents held on to them and continued to elaborate on them despite them being contradicted by "the facts," as judged by the application of theories that were better established at the time (e.g., Aristotelian mechanics). That is, new scientific theories often succeed because their proponents keep working on them and improving them despite being contradicting by the "facts"; then as the new theories become stronger and better accepted, people start juding the "facts" by the lens of the new instead of the old, and forget the problems that the new theories were judged to have and never resolved (e.g., things like Newtonian physics not having the same explanatory range as Aristotelian physics).

Re:Seconded.

[bcrowell]

Feyerabend and many other philosophers of science take a complementary stand to this by stressing the theory-ladenness of "facts."

Yep, they're totally right. For example, this 2003 paper claimed to have empirically verified the prediction of general relativity that gravitational forces propagate at the speed of light. The authors made some technical errors, which were rapidly pointed out by others in the field. The final answer is that actually nobody has the faintest clue how to test this specific, century-old prediction by Einstein. The reason is that nobody has figured out any alternative theory of gravity that (a) fits presently known experiments, and (b) predicts that gravitational forces propagate at some other velocity than the speed of light. There are other theories of gravity that satisfy a, and are inconsistent with general relativity, but they are all consistent with general relativity on b. Since there is no alternative theoretical framework, there is no way to design or analyze an experiment to test the question.

Exceedingly common

[pigwiggle]

in my experienced - I'm a physical chemist doing atomic resolution condensed phase computer modeling. It's so common that I am troubled when the first analysis gives the answer I expected. I likely spend more time looking for errors when the answer makes sense the first go through. Really.

Re:Exceedingly common

[NeutronCowboy]

in my experienced - I'm a physical chemist doing atomic resolution condensed phase computer modeling. It's so common that I am troubled when the first analysis gives the answer I expected. I likely spend more time looking for errors when the answer makes sense the first go through. Really.

This. Getting everything right the first time is like winning the lottery - you don't believe it, and you shouldn't. People doing experiments is a messy thing. Isolating variables is difficult, and much more difficult than just making something happen.

Re:Exceedingly common

[jamesh]

winning the lottery

Nobody ever wins the lottery anyway. I took a quick sample of the subset of the population that participated in lotteries and none of them had ever won, and by extrapolating that data I was able to prove that nobody had ever won a lottery ever.

Some other studies have been done that came to different conclusions but I believe that their data collection methodology was flawed as their results didn't agree with mine, so I think they can be safely excluded.

Experimental error

[dbIII]

It depends. If most of your data is noise it's fairly worthless anyway and you are better off trying to limit the sources of error and try again.
For example consider seismic data. You've got 50Hz or thereabouts induced in the cables near powerlines, you have wind blowing on the geophones, passing cars or trains, differences in soil above the rock and other sources of noise. A lot of seismic data processing seems to be about throwing away the noisy data and stacking up what is left to limit the effect of noise even furthur.
For other things there are different sources of error which may not be obvious. It's tempting to think it really is 27.23 Celcius becuase the digital thermometer say so, but the little semiconductor measuring probe may be out a full half a degree or more even if it does spit out numbers that fool people into thinking it is more accurate. Sure enough ten seconds later it could be telling you it is 26.8 Celcius when nothing has changed.

Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

If what is actually going on is that a train went past when the reading were taken or if the mains power had a minor spike then nobody really cares. It can take a while to set up a good experiment or set of measurements and some of the initial information collected may be rubbish. I've had bits of mid range steel tested where the results came back with large amounts of tungsten - and instead of compiling some theory about how it got there I've told the lab to kick the new kid off the machine, clean the electrodes and spark test it again.

Or more likely....

[Anonymous Coward]

More likely B ends up on the journal peer review panel because he is a respected pillar of his field, and causes pesky upstart A's paper to be rejected for publication. Forcing the field to wait 40 years for B and his ilk to shuffle off before followers of "crackpot" A can finally get their corroborating data published.

"Somebody Else's Problem"

[Anonymous Coward]

Once again, we see the prophetic genius of Douglas Adams. The investigations described in the article are working out the basic science required to enable the future engineering advance known as the "Somebody Else's Problem Field."

Because nothing works first time

[JohnFluxx]

It's pretty rare for everything to go right.

I work with holography. I shine a laser at a piece of film, then develop the film. And presto, I get no image. Do I throw out the theory that exposing film to light should produce an image? No, I assume that I screwed up and go back and start again. It's not uncommon for me to spend 3 months of cleaning, aligning, measuring and so on until I produce a proper image. I then throw away all the "bad" data. Maybe, theoretically, that data could be useful, but there's too many parameters to account for.

Re:Because nothing works first time

[Anonymous Coward]

It's not uncommon for me to spend 3 months of cleaning, aligning, measuring and so on until I produce a proper image. I then throw away all the "bad" data. Maybe, theoretically, that data could be useful, but there's too many parameters to account for.

But that's exactly what the author is talking about. You do know you could be first to document this as a real physical phenomenon? Way to dismiss this out-of-hand. You could name the physical culprit for the general effect of problems in producing an image in holography as "John's Imp" or something.

Re:Because nothing works first time

[JohnFluxx]

How would I remove all the other variables to the point where I can say, with certainty, that there really is something strange going on? It's just not possible. There's a hundred or so components that need to be aligned exactly, be totally clean, work perfectly, etc.

Re:Because nothing works first time

[pwfffff]

If you can't say with certainty that something strange is going on, then you've created a bad experiment. Seriously, how can you say, with certainty, that there really is ANYTHING going on in your experiment? If you can't track down the source of your bad data, then how do you know the source of your 'good' data? It's only 'good data' when it matches your hypothesis?

Re:Because nothing works first time

[Anonymous Coward]

"Seriously, how can you say, with certainty, that there really is ANYTHING going on in your experiment?"

Because after fixing all of the known and possible errors he got it to work. Most science is not random-experiments are set up to acheive a purpose. Why invent a magical and/or undefinable concept where none is needed. This is supposed to be science, not religion or alt/med.

In rare cases you do miss the next nobel prize. But you are still more likely to die in a horrible lab accident.

"If you can't track down the source of your bad data, then how do you know the source of your 'good' data? It's only 'good data' when it matches your hypothesis?"

Good data is data that can be used. If it can't be used, it's not good data. If an experiment doesn't produce data then by definition you didn't really get any good data. Other than knowing you likely screwed up.

Proper lab procedure is to record all data, even the lack thereof. Then not use the bad data. I would consider this data to be "discarded" even though it was still in a lab book somewhere. Of course, there is some data that HAS to be discarded due to storage limitations. As long as it is documented and can be recreated that shouldn't be an issue.

Re:Because nothing works first time

[Icarium]

That's a horrible analogy since you're no more doing science than I am when I start my car in the morning. You're not attempting to prove anything, you're relying on a technique based on a theory that has already been proven to produce a known result. In which case it's an entirely reasonable conclusion that it's your technique (or experiment) at fault.

Re:Because nothing works first time

[Anonymous Coward]

How would I remove all the other variables to the point where I can say, with certainty, that there really is something strange going on? It's just not possible.

I think you've missed a (obviously badly delivered) joke.

From my earlier post:

You could name the physical culprit for the general effect of problems in producing an image in holography as "John's Imp" or something.

The idea is that something will always go at least a little bit wrong, so you could imagine an imp (a playful mythological creature) taking some photons away from the image or scattering them, similar to Maxwell's Demon in thermodynamics manipulating slower/faster molecules. And because all experiments would have some error, this "effect" would be entirely reproducible -- hence, you would get to name it.

Re:Because nothing works first time

[JohnFluxx]

Huh? You can't prove a theory. I could say with quite a large amount of certainty, based on repeated observations, that the sun will rise tomorrow. But I still cannot prove it until tomorrow comes.

Two Relevant Quotes

[bill_mcgonigle]

"It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong."
  - Richard Feynman

"The most exciting phrase to hear in science, the one that heralds new discoveries, is not Eureka! but rather, "hmm.... that's funny...."
  - Isaac Asimov

Re:Two Relevant Quotes

[Anonymous Coward]

For the original speaker and probably his intended audience it's implicit, but on Slashdot I think it's appropriate to extend Feynman's quote to add "well-designed and independently-confirmed". I've performed a few experiments in my day that Conservation of Mass disagrees with.

Re:Two Relevant Quotes

[DriedClexler]

"It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong."

Correct, as long as you take that to mean "experiments in general". It is possible to make mistakes in experiments, and you shouldn't throw out General Relativity because some lab newbie got the setup wrong.

As I said in my other comment on this issue, you have to decide which is more likely: that you got the experiment wrong, or the hypothesis is wrong, and this depends on your confidence in both. But you should never hide the result, or else you can get an informational cascade that leads to conformism to bad theories.

Re:Two Relevant Quotes

[Anonymous Coward]

Don't forget the corollary to Asimov's statement. "More Discovery's are heralded by Oooops than Eureka."

K.I.S.S.

[Anonymous Coward]

Keep it simple, stupid. Only measure one thing at a time. It is amazing how many people screw that up.

Re:K.I.S.S.

[lq_x_pl]

I am shocked that this wasn't modded up. Seriously.

What do your examples have to do with race?

[Anonymous Coward]

Well, tell me then, just how do you define "race"? What are you testing for? How much melanin is in a person's skin? It's not like there's some "German" gene out there and another "Nigerian" gene and another "Japanese" gene that's common to all people who share a certain heritage. Culture isn't genetic, you know.

And then you changed the question: you said that we can test for certain traits of populations. That's true. But what has that got to do with race? Are you going to tell us that the amount of melanin in the skin correlates with IQ or something? (Even though "IQ" is poorly understood and is something that gets defined as "what IQ tests measure").

I'm pretty sure no one has ever claimed a race of sickle-cell anemiacs, after all. If you put them all in a room, would you really think that anyone would mistake them all for relatives?

Re:What do your examples have to do with race?

[Rakshasa Taisab]

I'm having difficulty figuring out if you're being serious or not... That just doesn't make any sense.

If a guy of mainly scandinavian ancestry gets a kid with a japanese ancestry, they're not going to pop out a kid who looks like a somalian guy. Political correctness be damned.

Your problem is you're looking for a 'specific gene' that defines ethnicity... It's not. We're dealing with a large set of genes here, where certain collections of patterns predominate in various populations. And these collectively lead to what we see as 'racial traits'.

Re:What do your examples have to do with race?

[NeutronCowboy]

Japanese is not a race. Scandinavian is not a race. Somalian is not a race.

You're confusing nationality with race. Kinda the issue Hitler had.

Re:What do your examples have to do with race?

[AniVisual]

He said ancestry. You're the one who's confusing things.

Collisions with Outsiders

[dcollins]

Don't think the summary quite found the central point of TFA.

"Dunbar found that most new scientific ideas emerged from lab meetings, those weekly sessions in which people publicly present their data. Interestingly, the most important element of the lab meeting wasn't the presentation -- it was the debate that followed. Dunbar observed that the skeptical (and sometimes heated) questions asked during a group session frequently triggered breakthroughs, as the scientists were forced to reconsider data they'd previously ignored. The new theory was a product of spontaneous conversation, not solitude; a single bracing query was enough to turn scientists into temporary outsiders, able to look anew at their own work."

"I saw this happen all the time," Dunbar says. "A scientist would be trying to describe their approach, and they'd be getting a little defensive, and then they'd get this quizzical look on their face. It was like they'd finally understood what was important."

So that's it: The keys are multiple viewpoints, skepticism, and intellectual competitiveness.

The Neuroscience of Scientific Illiteracy

[DynaSoar]

I am calling this neuroscience because it has nothing to do with how the nervous system operates. In this sense I am following the lead of WIRED and/or Dunbar, who can't tell a neuro from a social. From TFA: "Kevin Dunbar is a researcher who studies how scientists study things". OK, he studies things called scientists. scientists are people. The study of people and how they behave is psychology. Science is a social activity. Investigations of social activities are sociology when taken as a whole, or social psychology when considered in terms of the activities of individuals operating within a social group. Dunbar studied social psychology, not neuroscience. There's not a speck of neuroscience cereal in it anywhere. There's very little if any actual social psychology, and psychology, or any science at all. There's talking about science, there's talking to scientists about doing science, and there's watching them do science. There's watching and talking about getting good results and not getting good results, and what people do in the matter case. If Dunbar thinks he's doing neuroscience, I suspect he's not even very clear on science itself, much less the various branches. And it does say he's "a researcher in", not that he's a scientist. I do research in curry recipes from different countries and cultures. I'm a researcher, but not a cultural curriology scientist.

In fact I'll go s far as to say he's a researcher because he knows precious little and is trying to find out basic things, not as is the case with most scientists, someone who knows a fair amount and is trying to build on that with new knowledge. He is apparently not clear on the difference between 'screwing up' and not getting good and/or clean results. This may well be because he was unclear himself as to what it was he was looking at and talking about, and he thought he was just not getting good or clean results, when actually, guess what?

He doesn't let loose any secrets. Anyone can talk to scientists and as what happens if and when things don't turn out as expected. If you get an honest (ie. less concerned with appearances than truth) scientist, anyone would get the same answers. Or one could simply read work from real social psychologists and others who study science and scientists and learn the same things. I myself always recommend Collin's & Pinch's "The Golem" as an illuminating, instructive and entertaining starting point.

And a technical point on methodology: a study that does not find a difference between groups, treatments, whatever, 'fails to reject the null hypothesis' (the assertion that there is no observable difference). It does not prove there is no difference, it merely fails to find one. It fails, but only to find a difference, not to produce a result. It can't say there is no difference, it can only say that it couldn't find one. And, it fails to find a difference, no matter how nicely or hapazardly the data come out. The only studies that "fail" produce no data. Scientists may further fail to find an interpretation, but there's no limitation on trying to figure this out, and it applies to both 'results' (reject null hypothesis) and 'no results' (fail to reject null). Studies that produce data that 'makes no sense' produce data that fails to reject the null. The 'making no sense' is a post hoc evaluation of the data based on an incomplete understanding of the design, collection, analysis or interpretation. Such evaluations are done in science, but they are not part of the scientific process. Therefore when this occurs, it is not a "scientific" result and cannot be taken to reflect in the nature or quality of the work done. If you can't figure what it means, you can't figure out. You cannot say that since you cannot figure it out, then you figure out that it fails. If you think you can take something that 'doesn't make sense' and then say that it makes sense in that it represents a failure, then you've contradicted the assertion that it makes no sense. All you can say is that you don't understand it, and since you d

Re:The Neuroscience of Scientific Illiteracy

[Iyunkateus]

This is the greatest comment I have ever read.

Re:The Neuroscience of Scientific Illiteracy

[DingerX]

A few things about your post:

A. It's not as readable as Mr. Lehrer's article.
B. Mr. Lehrer is not the same as Dr. Dunbar, so what a journalist says about a scientist is not what scientist says.
C. I gather the discussion of using fMRI on test subjects and noting ACC and DLPFC firing at the same time is, by your analysis, in the realm of cognitive psychology, and not neuroscience.
D. So, according to your analysis, neuroscientists use the terms "Cognitive Dissonance" and "Confirmation Bias", while psychologists use ACC, DLPFC and the rest.
E. There is 'nothing new' here. But formalizing what we already know, and pointing to it can lead to new conclusions and uses.

You mean this isn't a wacky study

[NotSoHeavyD3]

About sex with the woman on top? (Yeah, I know bad joke.)

Obligatory

[Anonymous Coward]

I bet Einstein turned himself all sorts of colors before he invented the
lightbulb.

Anonymous Coward

[Anonymous Coward]

Data that is not published is still valid but wasted. Can we collect this data somewhere more public
even if the data does not fit a well known or lesser known theory - maybe a person viewing the
data can connect the dots. Asimov was correct 99% hard work and 1% inspiration leads to
"hmmm - thats funny"

Linus' law

[Anonymous Coward]

A little voice in my mind screams:
Given enough eyeballs, all bugs are shallow.

So essentially, Linus ported critical thinking to software development! :-)
Maybe we could backport something to science?