So we hear a lot about the need for ‘evidence-based policy’. But there is a huge elephant in the room:
The replication crisis (or replicability crisis or reproducibility crisis) is, as of 2019, an ongoing methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate or reproduce. The replication crisis affects the social and life sciences most severely.
That can’t be good. Mind you, I can’t complain, I’ve done well out of replicating empirical policy relevant analysis and found them wanting.
The (one-sided) Conversation has a piece justifying why evidence- based policy is good even in the absence of the very evidence the policy is being based upon.
The National Party, Queensland farming lobby group AgForce, and MP Bob Katter have banded together to propose an “independent science quality assurance agency”.
To justify their position, Liberal-National MP George Christensen and AgForce’s Michael Guerin specifically invoked the “replication crisis” in science, in which researchers in various fields have found it difficult or impossible to reproduce and validate original research findings. Their proposal, however, is not a good solution to the problem.
Now another review board may or may not be a good idea. But have a look at the arguments:
In various scientific areas, including psychology and preclinical medicine, large-scale replication projects have failed to reproduce the findings of many original studies. The rates of success differ between fields, but on average only half or fewer of published studies were successfully replicated. Clearly there is a problem.
So far, so good. But:
It is tempting to respond to politically motivated attacks on science by simply pointing to the excellent track record of scientific knowledge, or the good intentions of the vast majority of scientists.
How do you reconcile “excellent track record”, with “on average only half or fewer of published studies were successfully replicated”?
But moving along? What does The Conversation recommend?
Second, establish a national independent office of research integrity to allow errors in the scientific literature, whether deliberate or accidental, to be corrected in a fair, efficient, and systematic way.
Hmmmm. This differs from the “independent science quality assurance agency” in what way?