John Quiggin makes an interesting argument in a recent post.
if you can’t get basic stats right, you can’t get economics right either
I’d like to think that’s right, but I know it isn’t. First specialisation and the division of labour suggests that it would/should be possible to get economics right and not stats. Second, Deirdre McCloskey and Stephen Ziliak have tested that hypothesis. From the introduction of their 2003 Journal of Socio-Economics article.
Eight years ago, in “The Standard Error of Regressions,” we showed how significance testing was used during the 1980s in the leading general interest journal of the economics profession, the American Economic Review (McCloskey and Ziliak, 1996). The paper reported results from a 19-item “questionnaire” applied to all of the full-length papers using regression analysis. Of the 182 papers 70% did not distinguish statistical significance from policy or scientific significance—that is, from what we call “economic significance” (Question 16, Table 1, p. 105). And fully 96% misused a statistical test in some (shall we say) significant way or another. Of the 70% that flatly mistook statistical significance for economic significance, further, again about 70% failed to report even the magnitudes of influence between the economic variables they investigated (1996, p. 106). In other words, during the 1980s about one-half of the empirical papers published in the AER did not establish their claims as economically significant.
…
We are very willing to believe that our colleagues have since the 1980s stopped making an elementary error. But like them we are empirical scientists. And so we applied the same 19-item questionnaire of our 1996 paper to all the full-length empirical papers of the next decade of the AER, just finished, the 1990s. Significance testing violating the common sense of first-year statistics and the refined common sense of advanced decision theory, we find here, is not in fact getting better.
…
It is getting worse. Of the 137 relevant papers in the 1990s, 82% mistook statistically significant coefficients for economically significant coefficients (as against70%in the earlier decade). In the 1980s, 53% had relied exclusively on statistical significance as a criterion of importance at its first use; in the 1990s 64% did.
In their conclusion, this sentence sums it up.
The situation is strange: economic scientists, for example those who submit to and publish papers in the AER, or serve on hiring committees, routinely violate elementary standards of statistical cogency.
Does this make the AER authors bad economists? Not according to McCloskey and Ziliak.
The American Economic Review is filled with examples of superb economic science (in our opinion most of the papers can be described this way—even though most them, we have seen, make elementary mistakes in the use of statistical significance; in other words, we do not accept the opinion of one eminent econometrician we consulted, who dismissed our case by remarking cynically that after all such idiocy is to be expected in the AER).
Ziliak and McCloskey discuss this issue in book length too.
