Then there is the Tobacco PIR analysis.
It is contained in Appendix A to the PIR.
So first things first. What does the analysis report?
To measure the effect of the packaging changes on smoking prevalence, I adopt a widely-used approach in policy analysis often referred to as “before-after” regression analysis. My analysis relates an individual’s decision to smoke to a set of explanatory variables, including sociodemographic factors and controls for tobacco control policies (including the policies governing plain packaging and enlarged graphic health warnings) that are widely believed to influence individuals’ decisions to smoke. There are two important features of this analysis. First, it disentangles the effects of multiple factors that may simultaneously be influencing the observed outcome. Second, it identifies the effect of the packaging changes by comparing smoking behavior before the policy to smoking behavior after.
So far, so good.
The analysis makes use of Roy Morgan data and shows a time trend.
A bit dodgy – the analysis does not test to see if a linear trend is appropriate or not. These things are a function of beginning and end points, but okay.
Then the bottom line result!
Now I understand that not everyone can read an econometrics table – so let’s have the author of the report explain what that means:
Put differently, as shown in Table 1 smoking prevalence in Australia declined from an average of 19.4 percent in the 34 months before the 2012 packaging changes to an average of 17.2 percent in the 34 months after the 2012 packaging changes. Without the 2012 packaging changes, the model predicts that smoking prevalence would have still declined, but only to 17.77 percent. Thus, the packaging changes should be credited with about 0.55 percentage points (or about 25 percent) of the 2.2 percentage points of actual decline over this period.
So the difference between the two scenarios is about half of one percent – mind you nowhere near the 3.4% that Health claim Treasury told them. The joint effect of plain packaging and increased warning signs is half of one percent! Just to remind ourselves what the Public Health Association of Australia said about the policy:
Tobacco plain packaging has been a remarkable success, and has already saved tens of thousands of lives, according to the Public Health Association of Australia (PHAA).
Hyper-bowl as former prime minister Julia Gillard might have said.
Okay – so now let’s look at the problems.
First – We are told that the difference between the two estimates of smoking prevalence is statistically significantly different from zero. But that really tells us how precisely the model has estimated a difference. It doesn’t tell us how good the model is itself. If we have a bad model it doesn’t matter that the estimates are precisely estimated. The analysis is also based on survey data – what is the error rate in the survey? So we have a survey error rate, and a model error rate, and then a half of one percent estimate. The model error rate must be quite high – the pseudo-R square is less than 10% in all estimations of the model. In other words, the model presented cannot explain over 90% of the variation in the data – but we are invited to believe that the model can tell us that a half of one percent decline is due to the policy the government introduced.
Then there is something called Lindley’s paradox. As the number of observations increase so the probability of rejecting the null hypothesis rises even if it shouldn’t be rejected. This analysis has nearly 800,000 observations while the p-values for the policy intervention are between 1% and 3%. The author suggests that those p-values are appropriate the reject the null hypothesis at the 5% level (he is 95% confident that the null can be rejected). But not so fast – not with a regression that has nearly 800,000 observations and (a quick count) 52 variables. SO some quick back of the envelop calculations tell me that a coefficient from a regression with those characteristics needs a p-value of about 0.000117 to be confident of rejecting the null hypothesis.
Then we have the regression itself – the analysis consists of a probit with lots of dummy variables setting out socio-economic conditions. All good – but what is the base-case individual in the model. Who is everyone being compared to? Looking at the PIR – the base case person is an unmarried, male, Australian born, 14 – 17 year old, with a tertiary qualification, employed full time, but with an income less than $6000, and living in Victoria. Kind of tortured really. Strictly speaking the base shouldn’t really effect the overall results, but it is poor practice to have a non-existant entity as the base case – especially when the overall effect is so small. I’m thinking data snooping here.
So, in summary, survey data measured with error thrown into a regression analysis with nearly 800,000 observations and 52 variables and a pseudo-R square of less than 10% produces coefficients with relatively large p-values, and tells us a half of one percent difference is due to the policy?
Dodgy. Dodgy. Dodgy.
As we previously indicated this doesn’t even tell us whether plain packaging worked in isolation.
Then we read:
Because plain packaging is intended to deter smoking initiation, promote cessation, and deter relapse, the benefits of the packaging changes will likely grow over time.
But hang on … the analysis is assuming what it is meant to demonstrate!
I wonder how much the Health Department paid for this?
Update: The more I think about the dodgier I reckon the analysis is. I suspect he hasn’t even done an out of sample forecast.
Update II: It gets worse. The analysis only adjusts for major changes in excise policy – otherwise it does not include price effects in the modelling. This omitted variable bias means that the analysis assumes that cigarettes prices have been constant over the period of analysis.