I’m a tad suspicious of research undertaken by Pascal Diethelm and Timothy Farley – I have previously requested data to replicate their analysis of the Australian plain packaging policy and have never received any response; let alone the data.* So it was with very low expectations that I read their latest paper.
It is very typical of the anti-tobacco activist literature. Compare the Abstract with the actual results.
Our findings suggest a decline of smoking prevalence in minors following the introduction of plain packaging in Australia. They differ substantially from those presented in an industry-funded study on the effects of plain packaging on smoking prevalence in minors in Australia, which used the same data.
Actual Result – taken from the conclusion of the same study (emphasis added):
… our logistic regression analysis suggests a plain packaging effect in the expected direction, although this is not statistically significant, the data set on minors being too small and thus lacking the power needed to reach a firmer conclusion.
Just to confirm – the plain packaging effect in the Diethelm and Farley paper has a p-value of 0.15 – well above the generally accepted levels of 0.01, 0.05 or, if you are scraping the barrel, 0.1.
Now when undertaking statistical research one might set up the analysis as follows:
Null hypothesis: Policy had no impact on outcomes (for example, the introduction of plain packaging had no impact on smoking prevalence).
Alternative hypothesis: Policy had an impact (for example, the introduction of plain packaging reduced the prevalence of smoking).
What Diethelm and Farley find and report is that they were unable to reject the null hypothesis. That is, they cannot find evidence for the alternate hypothesis. Now I’m happy to accept their excuses for failing to reject the null – low power and small sample size. But the fact of the matter remains that they failed to reject the null.
This is important because they spend a great deal of time telling us that another paper by Kaul and Wolf is wrong. This is how they describe their results:
[Our] analyses fail to find any evidence for an actual plain packaging effect on Australians aged 14 to 17 years. Several reasonable variations to our methodology are discussed. All of these would only result in findings even more indicative of an absence of any plain packaging effect.
Yet ultimately from a policy perspective Diethelm and Farley came to precisely the same result – there is no evidence to support the hypothesis that plain packaging contributed to a decline in smoking prevalence amongst Australian youth. Rather the difference between the two sets of results is why that may be – either the policy is/was a dud or statistical techniques are too blunt to measure the true impact of the policy.
That in itself is an interesting argument – yet the fact remains that anti-tobacco activists are unable to deploy sufficiently powerful statistical techniques to demonstrate the policy actually worked. That means, in plain English, that there is no scientific evidence the policy worked.
I have a series of other quibbles about their study – for example, they date the policy has having started in November 2012 and not December – but that doesn’t really matter given their failure to reject the null hypothesis, that is fatal to their overall argument.
* email sent 28/11/2015:
I am interested in replicating the Deithelm and Farley results and wish to follow up on this statement in their paper:
“see description of method, Python program and reconstructed data points in on-line material”
Unfortunately I am unable to find any online material that has any such description etc.
Could you please point me in the right direction?
That email was sent to both the editor of the journal and Pascal Diethelm.