You can’t trust government research

Over the past 12 years or so, I’ve had a nice side gig going where I have replicated and critiqued modelling that underpins government policy. To that end I have critiqued the FuelWatch scheme, the GFC stimulus, the mining tax, the carbon tax, Treasury revenue forecasts, the plain packaging policy, and various tax proposals. This keeps me in mischief, and can be a lot of fun.

One of the big challenges, however, is dealing with trust in government. Often I hear the words, “But the [insert government agency] would never publish faulty research”. Well, no. Not only do they publish it, they publicise it too, and many journalists and politicians believe them.

It can be an uphill battle. Speaking of uphill battles, here is a paper in Nature published just last week:

We uncover a substitution effect that is robust to different specifications and control countries. In response to the policy, smokers switched from more expensive to cheaper cigarettes and reduced their overall tobacco expenditure and expenditure intensity. However, as smoking became less costly, smokers consumed more cigarettes. 

You read that here first – six years ago.

I digress. One of the problems is that many journalists and politicians don’t want to hear that government research is unreliable.

Luckily there is a growing body of research that investigates that very issue. The NBER has just published a very interesting paper looking at central bank research output (ungated version). From the introduction:

In this paper, we highlight a conflict of interests that has received little attention: that of central bank economists conducting research that evaluates central bank policies. Such research is common and widely cited.

Each of these five reasons could in principle induce a bias akin to the pro-sponsor bias in biomedical research. In fact, the bias could be even more severe for central bankers. While academic medical researchers are merely sponsored by industry, central bank economists are directly employed by central banks. Central banks evaluating their own policies is not unlike pharmaceutical firms evaluating their own drugs. Both have skin in the game. The conflict of interest is particularly acute for central banks that view their research output regarding their own policy as part of the policy itself, because by releasing a study supportive of this policy, they could potentially enhance the policy’s effectiveness.

Okay – so what did they find?

In this paper, we compare the findings of central bank researchers and academics regarding the effectiveness of unconventional monetary policy.

We find that central bank papers report systematically larger effects of QE on both output and inflation. Central bank papers are also more likely to report QE effects on output that are significant, both statistically and economically. For example, while all of the central bank papers report a statistically significant QE effect on output, only half of the academic papers do. Central bank papers are more likely to not disclose the width of the confidence interval, and also to use dynamic stochastic general equilibrium (DSGE) models rather than vector autoregression (VAR) models in their estimation.

Okay – so what?

We collect employment histories for all central bank authors and convert their job titles to numerical ranks on a sixpoint scale. For each author-paper pair, we measure the author’s subsequent career outcome by the first change in the author’s rank following the paper’s first public release. We find that authors whose papers report larger effects of QE on output experience more favorable career outcomes. A one standard deviation increase in the estimated effect is associated with a career improvement of about half a rank, such as moving halfway from Economist to Senior Economist. This evidence is consistent with career concerns.

Hmmmmm – so a willingness to produce dodgy research translates into career progression. To be fair – that isn’t quite how the authors of the paper interpret their result. But no matter how much lipstick you put on this pig, it is not a flattering finding for research that comes out of central banks.

The paper tries to sugar coat the result:

Importantly, we do not argue that central bank research should be discounted. In many ways, central banks are in an excellent position to provide accurate assessments of their own policies. Like pharmaceutical firms studying their own drugs, central banks have superior information about their own products, exceptionally strong expertise in the subject matter, and an intense desire to preserve their reputation. In addition, central banks are public institutions whose integrity is generally held in very high esteem. They understand that the effectiveness of their policy is predicated on their own credibility. We are not questioning that credibility. We simply highlight a previously unexplored conflict of interest that has the potential to influence a subset of bank research. We hope that our study will help central banks think through the implications of this conflict for their research processes.

It will be interesting observing the career path of two of the authors, who are central bank employees.

Update: While I’m thinking about it, here is a paper, published just two days ago, that finds that our very own Productivity Commission isn’t as impartial or apolitical as we’re often led to believe.

This entry was posted in Economics and economy, Libertarians don't live by argument alone, Plain Packaging, Politics. Bookmark the permalink.

26 Responses to You can’t trust government research

  1. Rex Anger

    It is shocking to believe, my Doomlord, that Vested Interests would not act like the impartial and honest brokers they so loudly proclaim to be…

    (Keep at them!)

  2. thefrollickingmole

    They had an interview on their abc with one of the tobacco papers writers. Almost 1st question was an attempted gotcha of “ who’s paying for your research”.
    Oh could almost hear the interviewer deflate as he was told no industry money.
    Of course they followed that up with a cancer council person who started her spiel with the same generalised slur.
    Awful people.

  3. Catlady

    I remember you unpacking the plain packaging policy. Excellent research! Refreshingly unbiased.

  4. Bruce of Newcastle

    The interesting thing is that the modelling they commission is usually massively wrong. Sometimes by orders of magnitude, as has notoriously been the case with the recent Covid-19 modelling. The same with climate change modelling. Yet they persist with this stuff, at great cost to Australians.

    You would think they would twig. The bureaucrats love models as a way to get pollies to do what the bureaucrats want, which continues the rule-by-bureaucrat which we saw in Yes Minister so long ago.

    I suppose that is why Trump gets so much hammering – he, nearly uniquely, goes from reason, long personal experience and won’t do what the elite bureaucratic class tells him to do. And has been remarkably successful by doing so too.

  5. Sinclair Davidson

    Thanks for the plug Catlady.

    That was a lot of fun.

  6. LBLoveday

    I hope occasional commenter Judith Sloan gives her thoughts on the Productivity Commission paper.

  7. NoFixedAddress

    Sinclair,

    with truly the deepest respect would you now consider that all Australian so called universities should be relocated to a central spot like, Alice Springs, and their former occupied lands in the cap city inner cities be used for something useful.. like growing flowers?

    The putrescence of wackademia will kill us all.

    So called scientists now need a bone through their nose to show their science wokeness.

    pathetic crap heads are now in charge of billions of dollars of other peoples money and some of them run the RBA and BHP!

  8. Sinclair Davidson

    NoFixedAddress – no. You’re being silly.

  9. Howard Hill

    Government lies. Well colour me shocked!

  10. NoFixedAddress

    whats wrong with growing flowers in inner cities?

  11. Some History

    Sinc, it’s not just government “research”. Authorities abusing their authority to exploit appeal to authority is a massive problem. Here’s a current example.

    UPDATE

    From Snowdon’s blog

    Smoking and COVID-19 – the evidence gets stronger

    I’ve found it impossible to keep up with all the research on smoking and COVID-19 recently. The tireless @phil_w888 has now catalogued over 700 studies of COVID-19 patients that have data on smoking prevalence.
    In the last week, the largest observational study yet conducted found that smokers (in Mexico) were 23 per cent less likely to test positive for COVID-19. This is in line with the results of an ongoing meta-analysis by some researchers who would clearly prefer the hypothesis to be disproved but who nevertheless have found a 26 per cent reduction in infection risk for current smokers.

    A study published in the Lancet a couple of weeks ago looked at the factors associated with COVID-19 caseloads at the national level. It found that countries with higher rates of smoking tended to have lower rates of Covid infection.

    And a newly published prospective study of nearly 20,000 Covid cases tells a familiar story. Your chances of ending up in intensive care with the virus are increased if you are male, non-white, from a low income area, obese … or a nonsmoker.

    https://velvetgloveironfist.blogspot.com/2020/08/smoking-and-covid-19-evidence-gets.html

    The last study referred to is a large UK prospective study.
    https://heart.bmj.com/content/heartjnl/early/2020/07/31/heartjnl-2020-317393.full.pdf

    It had good GP data on participants for multiple variables including sub-classifications of smoking status: Smoking status in five categories—never-smoker; exsmoker; light smoker (1–9 cigarettes/day); moderate smoker (10–19 cigarettes/day); heavy smoker (20+ cigarettes/day). From the study, smokers have a lower risk of contracting covid-19. Further, for those contracting covid-19, smokers had a lower risk of ending up in ICU than never-smokers and ex-smokers. There’s also a “dose/response” for the smokers group where heavy smokers had the lowest risk.

    There’s also the UK antibody study indicating that smokers are at lower risk of contracting covid-19 in the first place.

    https://www.imperial.ac.uk/media/imperial-college/institute-of-global-health-innovation/Ward-et-al-120820.pdf

  12. Some History

    How the World Health Organization will mess with your mind, if you let it

    There are around 800 studies on covid-19 that include smoking as a variable. The preponderance of case numbers from a variety of countries and research angles indicate that smokers are at lower risk of contracting covid-19 and severe covid-19 outcomes.

    What has the World Health Organization done with these findings?

    At the beginning of the pandemic, the WHO’s position on smoking was that the pandemic would be a good time to quit smoking. There was wild speculation that smokers would be at higher risk of covid-19 and worst outcomes.

    On the 4th May, the WHO released this antismoking propaganda piece:

    Tobacco control during the COVID-19 pandemic: how we can help

    The paradox is striking – the tobacco industry offering to help and support governments during this pandemic. Isn’t this the same industry that produces and aggressively markets an addictive product that kills up to half of its users? Isn’t this the same industry whose products lead to an increase in the incidence of NCDs that, in turn, worsen outcomes of COVID-19 patients?

    https://www.who.int/fctc/secretariat/head/statements/2020/tobacco-control-during-covid-19-pandemic/en/

    On the 11th May, the WHO releases

    WHO statement: Tobacco use and COVID-19

    WHO is constantly evaluating new research, including research that examines the link between tobacco use, nicotine use, and COVID-19. WHO urges researchers, scientists and the media to be cautious about amplifying unproven claims that tobacco or nicotine could reduce the risk of COVID-19. There is currently insufficient information to confirm any link between tobacco or nicotine in the prevention or treatment of COVID-19.

    https://www.who.int/news-room/detail/11-05-2020-who-statement-tobacco-use-and-covid-19

    On the 30th June, the WHO released its first “scientific brief”

    Smoking and COVID-19
    Scientific Brief

    Conclusions
    At the time of this review, the available evidence suggests that smoking is associated with increased severity of disease and death in hospitalized COVID-19 patients. Although likely related to severity, there is no evidence to quantify the risk to smokers of hospitalization with COVID-19 or of infection by SARS-CoV-2 was found in the peer-reviewed literature. Population-based studies are needed to address these questions.

    https://www.who.int/news-room/commentaries/detail/smoking-and-covid-19

    It may have been referred to as a “scientific brief” but it was simply a search of the available literature through particular filtering. Such a search was incapable of addressing the question that had become prominent – are smokers under-represented in the covid-19 data?

  13. Some History

    Having failed to address the fundamental issue of smoker under-representation in covid-19 data, the WHO released what appears to be its final position on covid-19 & smoking on the 10th July:

    WHO and partners to help more than 1 billion people quit tobacco to reduce risk of COVID-19

    The project gives people free access to nicotine replacement therapy and to Florence, a digital health worker, based on artificial intelligence that dispels myths around COVID-19 and tobacco and helps people develop a personalized plan to quit tobacco.

    https://www.who.int/news-room/detail/10-07-2020-who-and-partners-to-help-more-than-1-billion-people-quit-tobacco-to-reduce-risk-of-covid-19

    The WHO provided no evidence for its position on covid-19 & smoking. This baseless, one sentence about “dispelling” myths around covid-19 & tobacco was made in the midst of WHO’s launch of its latest “quit smoking” program that included Princess Dina of Jordan.

    WHO’s conduct is abominable, yet fully to be expected from a long-time leading promoter of prohibition of tobacco use. There will be many Public Health groups – government and NGOs – and individuals that will refer/link to the WHO position as definitive. These are suckers for “appeal to authority”. Authorities such as the WHO are well aware of this. They have been exploiting/abusing their authority to push ideological/commercial agenda for decades.

    What could now be a standard conversation –

    A: I think I heard something about smoking being protective against covid-19.
    B: No, that’s not right. I’ve seen the WHO website and the WHO even has an app that dispels this myth about smoking being protective, a myth probably created by the tobacco industry.

    A: But I don’t think the WHO addressed any actual data.
    B: No, that can’t be true. The WHO would have looked at all the available data and arrived at the conclusion that smoking is not protective for covid-19.

    A: I don’t know about that.
    B: Who are we going to believe – you or the WHO?

    Brainwashing accomplished.

  14. NoFixedAddress

    Sinclair

    “silly” is me being elected to the RMIT SRC back in the ’70s.

    “silly’ is my middle name.

    I went back to Melbourne in 2006 for a few years and did a job or 2.

    Went and had a look at RMIT and then had to track the Business Studies equivalent down in Burke Street (?) above Elizabeth.

    Amazed me.

    But I found the good coffee shop out the back lane.

  15. Some History

    I am not particularly familiar with the hydroxychloriquine (HCQ) data.

    This is a comment appearing on Catallaxy about a month ago.

    The author could have looked up HCQ research. He would have found, for example, that the WHO lists under mythbusters the fact that HCQ is of any use in COVID-19 infection. (https://www.who.int/emergencies/diseases/novel-coronavirus-2019/advice-for-public/myth-busters?gclid=Cj0KCQjwsuP5BRCoARIsAPtX_wG-LC67a-OmTNjGiF2GQBy2ANCROjZc-j1BLo4f3xzQqvEHCTCd1IIaAkboEALw_wcB). Johns Hopkins says there is no evidence of HCQ benefit in COVID (https://www.hopkinsmedicine.org/news/newsroom/news-releases/covid-19-story-tip-hydroxychloroquine-not-recommended-for-treatment-of-covid-19) Bonus quote :”“Patients infected with COVID-19 often have compromised heart and vascular systems, and receive other drugs that can interact with hydroxychloroquine and can put patients at increased risk of arrhythmias or irregular heartbeats,” says cardiologist Oscar Cingolani, M.D., associate professor of medicine at the Johns Hopkins University School of Medicine. “These cardiovascular side effects in some COVID patients aren’t often seen in patients receiving the drug for other purposes like autoimmune disorders, so therefore the safety observed in these other patients can’t be inferred for COVID-19 patients.”
    I suppose you could ask a Democrat or Republican or someone from CNN or Breitbart. Personally, I like to ask people with knowledge of health and science.

    Seeing how the WHO can use “mythbusting” as a propaganda tool, that it has a mythbusting section for HCQ should at least be ringing alarm bells. This is the WHO section at the first link:

    FACT: Studies show hydroxychloroquine does not have clinical benefits in treating COVID-19
    Hydroxychloroquine or chloroquine, a treatment for malaria, lupus erythematosus, and rheumatoid arthritis, has been under study as a possible treatment for COVID-19. Current data shows that this drug does not reduce deaths among hospitalised COVID-19 patients, nor help people with moderate disease.*
    The use of hydoxychloroquine and chloroquine is accepted as generally safe for patients with malaria and autoimmune diseases, but its use where not indicated and without medical supervision can cause serious side effects and should be avoided.
    * More decisive research is needed to assess its value in patients with mild disease or as pre- or post-exposure prophylaxis in patients exposed to COVID-19.

    Interesting, too, is how the commenter uses the information in appeal to authority –

    I suppose you could ask a Democrat or Republican or someone from CNN or Breitbart. Personally, I like to ask people with knowledge of health and science.

  16. stackja

    As I remember in the 1960s research was still respected. Then came Gough!

  17. NuThink

    But but but the independent tribunals that decide on pollie wages are truly independent! I would be shocked shocked shocked if they were not.
    Also I notice one TV news channel which says they are trusted.
    Just because someone is trusted does not necessarily make them trustworthy.

  18. Entropy

    Ahh, the sort of stuff that originally brought me to the Cat!
    Thank you Professor Doomlord! Thank you!

    See also:
    Regulation Impact Statements
    Rural and Regional Impact Statements
    Empolumnt Impact Statements
    Human Rights Impacts Statements
    Paris Agreement Impact Statements
    Ramsar Agreement Impact Statements
    Aaa did..FFS you get the drill.
    Any Explanatory Memorandums associated with any legislation these days is infested with this kind of self justifying shit up front, no doubt by design to ensure everyone is asleep before they get to the important stuff about what the legislation is actually supposed to do.

  19. Squirrel

    ” Often I hear the words, “But the [insert government agency] would never publish faulty research”. Well, no. Not only do they publish it, they publicise it too, and many journalists and politicians believe them.”

    Aside from instances of less-than-subtle political pressure (which is not, of course, what this post is primarily about), I think this generally happens because the people who commission and/or conduct the research will typically find (quelle surprise) that the research fairly neatly aligns with their world view and circumstances.

    I have yet to hear of economic research/modelling emanating from the federal econocracy which does not support policies that will improve the lot of 30/40-something, partnered with children (or intending to), upwardly-thrusting white collar workers – i.e. the typical Treasury, Finance, PM&C etc. mid and upper-level officials.

  20. Hmmmmm – so a willingness to produce dodgy research translates into career progression.

    Hmmmmm – sounds a lot like Climate Science to me.

  21. And a newly published prospective study of nearly 20,000 Covid cases tells a familiar story. Your chances of ending up in intensive care with the virus are increased if you are male, non-white, from a low income area, obese … or a nonsmoker.

    I’ll have a stab…

    MALE: Zinc seems to be essential in fighting the virus. Men have lower levels of zinc because semen is essentially zinc. Men drain their body of zinc by ejaculating.

    NON-WHITE: Vitamin D seems to be essential in fighting the virus. Non-whites have difficulty producing vitamin D due to skin colour. Darker people in Sweden, where the sun don’t shine too strongly seem to have been effected by COVID more than the general light skinned natives.

    NONSMOKERS: Smoking dries out the nose throat and lungs. Viruses need some moisture. Smoking also increases the temperature of those areas, albeit temporarily.
    Maybe smokers kill at least some of the virus before the darn thing makes it in sufficient numbers into the body.

    FAT BASTARDS: Say no more.

    POOR BASTARDS: lack of hygiene and poor diet?

    Send research money to Baa Humbug. c/o Cat Files.

  22. stackja

    Ronald Reagan is quoted as saying:
    Economists put end to end, don’t reach a conclusion. More OPM always required.

  23. DaveR

    I unfortunately caught malaria while working in rural Milne Bay province in PNG in the 1980s. Chloroquin and Malaprin were the drugs of choice then. It took 10 years to fully eradicate the disease, and the tin of 5,000 Chloroquin tablets were a central part of the medicine cabinet for quite a while. No side effects (so far), I have not caught SARS or COVID-19, and I have hardly caught a cold or a winter flu since that time.

  24. DaveR

    The obvious example of faulty government research is the output from the Climate Change Commission set up under Gillard. This group was hand picked from the ranks of green zealots, and produced “research” required for the emerging Carbon Tax legislation. Their faulty “research” and faulty recommendations went unchallenged, and was picked up by the institutions captured by the left such as CSIRO and BOM, and various Chief Scientists, who still support it today. It includes the Flannery howler that “rains will not fill our dams in the future” and other such absurd “scientific” forecasts.

    The CCC “science” formed the scientific basis for Turnbull’s policies in opposition, and then for Turnbull and Frydenberg’s NEG legislation attempt, with its concealed Carbon Tax, that destroyed Turnbull.

    It then formed the “scientific” basis for the Renewable Energy policies and massive public subsidies of the Coalition government, which have caused at least a doubling of electricity prices across the nation and destabilisation of the electricity grid. Still, the faulty “research” was not challenged.

    And today, despite its obvious erroneous analysis and conclusions, it still forms the basis of Coalition climate change and energy policy and is still accepted by institutions such as CSIRO and BOM. And still there is a refusal to re-evaluate the faulty “science” of the original CCC underlying the whole policy mess.

  25. John A

    For more on this and the various biases and faulty reasoning which go into statistical analyses, see
    Tim Harford “How to Make the World Add Up” available at all good bookstores (and possibly some bad ones as well 🙂 ) Here is the Amazon link.

Comments are closed.