Whether and how much it depends on the IQs of individual members and the composition of the group like the presence of females and sensitivity of group members. In the discussion, some of the members of the original papers which did not agree with each other participate.
Perverse Incentives and Replication in Science a post of Steven HSU at information Processing.
See also the discussion at Economist's View http://economistsview.typepad.com/economistsview/2017/03/on-perverse-incentives-and-replication-in-science.html
"Stephen Hsu has a nice blog post on this topic. He writes about this common pattern:
#1 is spot-on for economics. Woe be to she who bucks the dominant narrative. In economics, something else happens. Following the study, there are 20 piggy-back papers which test for the same results on other data. The original authors typically get to referee these papers, so if you're a young researcher looking for a publication, look no further. You've just guaranteed yourself the rarest of gifts -- a friendly referee who will likely go to bat for you. Just make sure your results are similar to theirs. If not, you might want to shelve your project, or else try 100 other specifications until you get something that "works". One trick I learned: You can bury a robustness check which overturns the main results deep in the paper, and your referee who is emotionally invested in the benchmark result for sure won't read that far.
Hsu then writes:
Perverse Incentives and Replication in Science a post of Steven HSU at information Processing.
See also the discussion at Economist's View http://economistsview.typepad.com/economistsview/2017/03/on-perverse-incentives-and-replication-in-science.html
"Stephen Hsu has a nice blog post on this topic. He writes about this common pattern:
1. Study reports results which reinforce the dominant, politically correct, narrative.
2. Study is widely cited in other academic work, lionized in the popular press, and used to advance real world agendas.
3. Study fails to replicate, but no one (except a few careful and independent thinkers) notices.
#1 is spot-on for economics. Woe be to she who bucks the dominant narrative. In economics, something else happens. Following the study, there are 20 piggy-back papers which test for the same results on other data. The original authors typically get to referee these papers, so if you're a young researcher looking for a publication, look no further. You've just guaranteed yourself the rarest of gifts -- a friendly referee who will likely go to bat for you. Just make sure your results are similar to theirs. If not, you might want to shelve your project, or else try 100 other specifications until you get something that "works". One trick I learned: You can bury a robustness check which overturns the main results deep in the paper, and your referee who is emotionally invested in the benchmark result for sure won't read that far.
Hsu then writes:
"one should be highly skeptical of results in many areas of social science and even biomedical science (see link below). Serious researchers (i.e., those who actually aspire to participate in Science) in fields with low replication rates should (as a demonstration of collective intelligence!) do everything possible to improve the situation. Replication should be considered an important research activity, and should be taken seriously""
No comments:
Post a Comment