John Quiggin makes an interesting argument in a recent post.
if you can’t get basic stats right, you can’t get economics right either
I’d like to think that’s right, but I know it isn’t. First specialisation and the division of labour suggests that it would/should be possible to get economics right and not stats. Second, Deirdre McCloskey and Stephen Ziliak have tested that hypothesis. From the introduction of their 2003 Journal of Socio-Economics article.
Eight years ago, in “The Standard Error of Regressions,” we showed how significance testing was used during the 1980s in the leading general interest journal of the economics profession, the American Economic Review (McCloskey and Ziliak, 1996). The paper reported results from a 19-item “questionnaire” applied to all of the full-length papers using regression analysis. Of the 182 papers 70% did not distinguish statistical significance from policy or scientific significance—that is, from what we call “economic significance” (Question 16, Table 1, p. 105). And fully 96% misused a statistical test in some (shall we say) significant way or another. Of the 70% that flatly mistook statistical significance for economic significance, further, again about 70% failed to report even the magnitudes of influence between the economic variables they investigated (1996, p. 106). In other words, during the 1980s about one-half of the empirical papers published in the AER did not establish their claims as economically significant.
…
We are very willing to believe that our colleagues have since the 1980s stopped making an elementary error. But like them we are empirical scientists. And so we applied the same 19-item questionnaire of our 1996 paper to all the full-length empirical papers of the next decade of the AER, just finished, the 1990s. Significance testing violating the common sense of first-year statistics and the refined common sense of advanced decision theory, we find here, is not in fact getting better.
…
It is getting worse. Of the 137 relevant papers in the 1990s, 82% mistook statistically significant coefficients for economically significant coefficients (as against70%in the earlier decade). In the 1980s, 53% had relied exclusively on statistical significance as a criterion of importance at its first use; in the 1990s 64% did.
In their conclusion, this sentence sums it up.
The situation is strange: economic scientists, for example those who submit to and publish papers in the AER, or serve on hiring committees, routinely violate elementary standards of statistical cogency.
Does this make the AER authors bad economists? Not according to McCloskey and Ziliak.
The American Economic Review is filled with examples of superb economic science (in our opinion most of the papers can be described this way—even though most them, we have seen, make elementary mistakes in the use of statistical significance; in other words, we do not accept the opinion of one eminent econometrician we consulted, who dismissed our case by remarking cynically that after all such idiocy is to be expected in the AER).
Ziliak and McCloskey discuss this issue in book length too.
It is getting worse. Of the 137 relevant papers in the 1990s, 82% mistook statistically significant coefficients for economically significant coefficients (as against70%in the earlier decade). In the 1980s, 53% had relied exclusively on statistical significance as a criterion of importance at its first use; in the 1990s 64% did.
Thus I can no longer trust an economist when he\she states, the data is clear …. .
John H.
6 Mar 10 at 6:47 pm
No amount of statistical savvy is any good unless you ask the right questions. First have a hypothesis that is worth testing.
Rafe
7 Mar 10 at 8:10 am
Seems like a bit of an own goal Sinclair; you have provided proof that in the main economists lack statistical cogency whilst the GFC proves it.
rog
7 Mar 10 at 5:57 pm
All the GFC proves is that economists can’t foresee the future, everyone knows that already.
Sinclair Davidson
7 Mar 10 at 6:01 pm
A nice point. But McCloskey is applying a much tighter standard than was violated in the present case, and I think one that is tighter than most econometricians would consider justified. Adrian Pagan, who knows more about econometrics than anyone was pretty scathing about the McCloskey paper when she presented it in Canberra a while back.
In her view simply reporting that there exists a statistically significant relationship without an analysis of the economic implications of the point estimate is an error. I don’t think this can be sustained. For (+ve) coefficients that are marginally significant (say t<3), you can't really say anything about the magnitude except that is positive, and that you have some kind of upper bound. In such cases, reporting the hypothesis test and not saying much about the estimated magnitude is perfectly reasonable. The exception would be if the effect is of a different order of magnitude than theory would predict, as for example, a tiny effect showing up as as significant in a gigantic sample.
At points, McCloskey comes close to saying that we should ignore statistical significance altogether – this is wrong in my view, though I'd prefer a Bayesian approach.
In any case, this is a far cry from translating "not statistically significant" as "insignificant" or "nonexistent" as has been done here. That really is an error so basic as to disqualify you from talking about data.
John Quiggin
8 Mar 10 at 9:48 am
I agree McCloskey’s overall standard is much higher than what we’re talking about here. I suspect that we’ll never really live up to her expectations – but that’s another point. The argument here, I think, fits into her ‘sign econometrics point’ – I’m very loathe to make a fuss about a coefficient that is either positive or negative if it’s not statistically significantly significant different from zero.
On the Bayesian issue, I don’t know enough about it to say one way or the other.
(On the point about Austrian economists should stop crapping on about meta issues and do economics, we are in total agreement.)
Sinclair Davidson
8 Mar 10 at 11:12 am