Did the stimulus work II?

You’ll recall an earlier post where I questioned a graph in the budget papers.

In the budget papers they produce a graph showing a nice upward sloping relationship between the size of the stimulus in 2009 relative to IMF forecasting errors.

I had a go at the analysis saying that it was faulty. Treasury agrees with me.

Professor Sinclair Davidson of RMIT has pointed out that, when the dataset includes all 19 countries of the G-20 (rather than being restricted to the 11 countries used for the regression reported in the Budget), the estimated slope on the regression line is much flatter, and the coefficient becomes statistically insignificant (Chart B and Table A) ( http://catallaxyfiles.com/2010/05/13/did-the-stimulus-work/).

Professor Davidson is correct. Before publishing the results in the Budget, the regression result for the full sample of 19 countries was checked. Unfortunately, however, an error was made and the erroneous conclusion was drawn that the results for the restricted sample did not differ, to any material extent, from those for the full sample.

Treasury then redo the analysis with a different dataset and claim that they can get to the same result. But can we really believe them?

For completeness, note that including Greece, Hungary, Iceland and Ireland in the regression generates a statistically insignificant slope coefficient. This result is driven by Iceland. The Icelandic financial system collapsed in 2008 and economic output collapsed in 2009, falling by 6.5 per cent, though this outcome was better than had been expected by the IMF in April 2009. Given Iceland’s dire circumstances, it is possible that fiscal consolidation may have improved economic prospects somewhat, as suggested many years ago by Giavazzi and Pagano (NBER Macroeconomics Annual 1990). Whether or not it did so, the extreme circumstances faced by Greece, Hungary, Iceland and Ireland are not relevant to the question at hand.

That is a result that needs to be double checked. They seem to be suggesting that Iceland is an outlier, but have provided no evidence to suggest that it is.

This entry was posted in Uncategorized. Bookmark the permalink.

17 Responses to Did the stimulus work II?

  1. Samuel J

    This looks like choosing countries to meet the claim. And being misleading by not examining other explanations. And isn’t there an implicit assumption that every dollar spent was done efficiently?

  2. daddy dave

    Okay, so they admit you were right, but now they’re pointing at a paper from the US treasury that found the same thing.
    I couldn’t find the data that the American paper used, although I didn’t look very hard. It seems like the entire relationship is based on Korea, which (a) had a massive stimulus, and (b( outperformed expectations. Since the other countries are a wash, this makes it look like on average, countries with stimuli outperformed expectations. In other words, a single outlier is skewing the curve and creating a spurious relationship. I’d like to see what happens when you take out Korea.

  3. Samuel J

    I can just imagine the exchange in Treasury:

    General Manager running into Gruen’s office: “Sinclair Davidson found a hole in Box 4 of the Budget Paper – he says that we selected 11 countries to show statistical significance”

    Gruen “Damn – I thought that would slip through to the keeper. I suppose we can’t cover it up. So let’s go on the offensive and offer old Sincs a backhander. Go and find a country that could be an outlier.”

    GM: “ok I’ll be back shortly”

    GM to Manager: “that box 4 you were showing me – I want a country that’s an outlier. I don’t care how big or small or irrelevant. I want an outlier.

    Manager to Analyst: “crank up the excel spreadsheet and let’s feed in some DX data to get the outlier”

    Analyst standing over Graduate’s computer: “come on work more quickly. I’ve been told Dr Gruen wants this pronto”

    Graduate: “I’ve found it – what about Iceland?”

    Analyst to Manager: “Iceland”

    Manager to General Manager: “Iceland”

    GM to Gruen: “Iceland”

    Gruen: “Eureka – now I’ll pad around that in Estimates”.

  4. JC

    Korea would of course be the outlier.

  5. Boris

    I have to say I am a bit shocked by the shallowness of their analysis. At the very least, the regression should weighted by the GDP. Then Iceland won’t be a problem.

  6. Sinclair Davidson

    Boris – we don’t usually weighted regressions. OLS is the standard approach.

  7. pedro

    The quality of the claim would be visually apparent with names put on the second graph in your original post.

  8. Sinclair Davidson

    Pedro – see second graph hopefully by the end of today I’ll have an updated version.

  9. Capitalist Piggy

    The Treasury table shows the r-squared for the OECD (26 countries only, excludes Iceland, Ireland, Greece and Hungary) as 0.15. How is such a low number “statistically significant”?

  10. Seems to be one of the most common (human) errors with statistics – choosing to exclude data you don’t want because of what seems a “perfectly valid reason”

    I always thought the saying about “lies, damn lies, and statistics” was unfair to statistics. Numbers don’t lie, but even when people don’t make mistakes, they are prone to make things fit. Counting the hits, and ignoring (or finding a reason to eliminate) the misses is a standard part of post-hoc analysis.

  11. Skuter

    Of course Iceland is an outlier. The fact that Iceland performed better than expected given a fiscal consolidation is heresy. I mean it just doesn’t fit with what that nice fellow Mr Keynes said. He said that if we try to repair our bdugets in times of crisis we would fall into an abyss. We can’t have those rogue observations interefering with our theory now can we?

  12. daddy dave

    Counting the hits, and ignoring (or finding a reason to eliminate) the misses is a standard part of post-hoc analysis.
    True. That’s why I guess you can’t take out Korea or Iceland as they were in the original analysis. Still, it would be an interesting to find out how much of the effect is accounted for by one data point.

  13. http://www.news.com.au/business/breaking-news/economy-grew-by-05pc-in-march-quarter-gdp-data-shows/story-e6frfkur-1225874488223

    We spent 4.2% of GDP to get 2.7% growth. The recovery had already occured when the money began to slosh around and discouraged workers and labour force disattachments went through the roof.

    I’m inclined to say this stuff never works. It worked this time on paper.

  14. pedro

    LOL sinclair, my lyin eyes tell me that the claims can’t be sustained. Treasury has turned in to figure-fudging central.

  15. Boris

    “Boris – we don’t usually weighted regressions. OLS is the standard approach”

    But is this reasonable? If we compute, say, a number of olympic medals per population, one champion from Monaco wll skew the whole average enormously. The outcome in a big country is a lot more important because of the sample size, so to speak.

  16. Pingback: Club Troppo » How do we know if the stimulus worked?

  17. Sinclair Davidson

    Boris – economists do control for size effects, but don’t usually use weighted least squares.

Comments are closed.