Science is messy

Vaughan Bell in The brain isn’t going to take it lying down:
"Anger experiments that have measured electrical signals from the brain (using EEG) or that have altered neural activity with magnetic pulses (using TMS) have found that the left frontal lobe is more active than the right, but studies using fMRI functional brain scans have found no differences.

Psychologists Eddie Harmon-Jones and Carly Peterson wondered whether the brain might be working differently in EEG and TMS experiments because the participant is usually sitting upright, while in fMRI, the person is usually lying flat on their back.
........
Although this is only an initial study, it could be a major spanner in the works for cognitive science which often assumes that clumping together evidence from a whole range of techniques gives a better idea of what’s going on."

David H. Freedman in Why Scientific Studies Are So Often Wrong: The Streetlight Effect:
"Many, and possibly most, scientists spend their careers looking for answers where the light is better rather than where the truth is more likely to lie. They don’t always have much choice. It is often extremely difficult or even impossible to cleanly measure what is really important, so scientists instead cleanly measure what they can, hoping it turns out to be relevant. After all, we expect scientists to quantify their observations precisely. As Lord Kelvin put it more than a century ago, “When you can measure what you are speaking about, and express it in numbers, you know something about it.”

There is just one little problem. While these surrogate measurements yield clean numbers, they frequently throw off the results, sometimes dramatically so. This “streetlight effect,” as I call it in my new book, Wrong (Little, Brown), turns up in every field of science, filling research journals with experiments and studies that directly contradict previously published work."

See also Jonah Lehrer's The Mysterious Decline Effect

Related posts: Ashutosh Jogalekar's Aliens, arsenic and alternative peer-review: Has science publishing become too conservative? and Making speculation official: More on the conservatism of leading science journals. From the later post:

"There are rules for doing, interpreting and publishing science, just like there are rules for how to raise children. And just as the rules for raising children wonderfully break down in the face of reality, so do the rules of actual scientific research. Real science is as messy as real child rearing. It's only fair that the public knows about this process."
P.S. See also "The truth wears off: Is there something wrong with the scientific method?" by Andrew Gelman.