Over the past 12 years or so, I’ve had a nice side gig going where I have replicated and critiqued modelling that underpins government policy. To that end I have critiqued the FuelWatch scheme, the GFC stimulus, the mining tax, the carbon tax, Treasury revenue forecasts, the plain packaging policy, and various tax proposals. This keeps me in mischief, and can be a lot of fun.
One of the big challenges, however, is dealing with trust in government. Often I hear the words, “But the [insert government agency] would never publish faulty research”. Well, no. Not only do they publish it, they publicise it too, and many journalists and politicians believe them.
It can be an uphill battle. Speaking of uphill battles, here is a paper in Nature published just last week:
We uncover a substitution effect that is robust to different specifications and control countries. In response to the policy, smokers switched from more expensive to cheaper cigarettes and reduced their overall tobacco expenditure and expenditure intensity. However, as smoking became less costly, smokers consumed more cigarettes.
You read that here first – six years ago.
I digress. One of the problems is that many journalists and politicians don’t want to hear that government research is unreliable.
Luckily there is a growing body of research that investigates that very issue. The NBER has just published a very interesting paper looking at central bank research output (ungated version). From the introduction:
In this paper, we highlight a conflict of interests that has received little attention: that of central bank economists conducting research that evaluates central bank policies. Such research is common and widely cited.
Each of these five reasons could in principle induce a bias akin to the pro-sponsor bias in biomedical research. In fact, the bias could be even more severe for central bankers. While academic medical researchers are merely sponsored by industry, central bank economists are directly employed by central banks. Central banks evaluating their own policies is not unlike pharmaceutical firms evaluating their own drugs. Both have skin in the game. The conflict of interest is particularly acute for central banks that view their research output regarding their own policy as part of the policy itself, because by releasing a study supportive of this policy, they could potentially enhance the policy’s effectiveness.
Okay – so what did they find?
In this paper, we compare the findings of central bank researchers and academics regarding the effectiveness of unconventional monetary policy.
We find that central bank papers report systematically larger effects of QE on both output and inflation. Central bank papers are also more likely to report QE effects on output that are significant, both statistically and economically. For example, while all of the central bank papers report a statistically significant QE effect on output, only half of the academic papers do. Central bank papers are more likely to not disclose the width of the confidence interval, and also to use dynamic stochastic general equilibrium (DSGE) models rather than vector autoregression (VAR) models in their estimation.
Okay – so what?
We collect employment histories for all central bank authors and convert their job titles to numerical ranks on a sixpoint scale. For each author-paper pair, we measure the author’s subsequent career outcome by the first change in the author’s rank following the paper’s first public release. We find that authors whose papers report larger effects of QE on output experience more favorable career outcomes. A one standard deviation increase in the estimated effect is associated with a career improvement of about half a rank, such as moving halfway from Economist to Senior Economist. This evidence is consistent with career concerns.
Hmmmmm – so a willingness to produce dodgy research translates into career progression. To be fair – that isn’t quite how the authors of the paper interpret their result. But no matter how much lipstick you put on this pig, it is not a flattering finding for research that comes out of central banks.
The paper tries to sugar coat the result:
Importantly, we do not argue that central bank research should be discounted. In many ways, central banks are in an excellent position to provide accurate assessments of their own policies. Like pharmaceutical firms studying their own drugs, central banks have superior information about their own products, exceptionally strong expertise in the subject matter, and an intense desire to preserve their reputation. In addition, central banks are public institutions whose integrity is generally held in very high esteem. They understand that the effectiveness of their policy is predicated on their own credibility. We are not questioning that credibility. We simply highlight a previously unexplored conflict of interest that has the potential to influence a subset of bank research. We hope that our study will help central banks think through the implications of this conflict for their research processes.
It will be interesting observing the career path of two of the authors, who are central bank employees.
Update: While I’m thinking about it, here is a paper, published just two days ago, that finds that our very own Productivity Commission isn’t as impartial or apolitical as we’re often led to believe.