Failing to reject the null hypothesis

I’m a tad suspicious of research undertaken by Pascal Diethelm and Timothy Farley – I have previously requested data to replicate their analysis of the Australian plain packaging policy and have never received any response; let alone the data.* So it was with very low expectations that I read their latest paper.

It is very typical of the anti-tobacco activist literature. Compare the Abstract with the actual results.

Abstract:

Our findings suggest a decline of smoking prevalence in minors following the introduction of plain packaging in Australia. They differ substantially from those presented in an industry-funded study on the effects of plain packaging on smoking prevalence in minors in Australia, which used the same data.

Actual Result – taken from the conclusion of the same study (emphasis added):

… our logistic regression analysis suggests a plain packaging effect in the expected direction, although this is not statistically significant, the data set on minors being too small and thus lacking the power needed to reach a firmer conclusion.

Just to confirm – the plain packaging effect in the Diethelm and Farley paper has a p-value of 0.15 – well above the generally accepted levels of 0.01, 0.05 or, if you are scraping the barrel, 0.1.

Now when undertaking statistical research one might set up the analysis as follows:

Null hypothesis: Policy had no impact on outcomes (for example, the introduction of plain packaging had no impact on smoking prevalence).

Alternative hypothesis: Policy had an impact (for example, the introduction  of plain packaging reduced the prevalence of smoking).

What Diethelm and Farley find and report is that they were unable to reject the null hypothesis. That is, they cannot find evidence for the alternate hypothesis. Now I’m happy to accept their excuses for failing to reject the null – low power and small sample size. But the fact of the matter remains that they failed to reject the null.

This is important because they spend a great deal of time telling us that another paper by Kaul and Wolf is wrong. This is how they describe their results:

[Our] analyses fail to find any evidence for an actual plain packaging effect on Australians aged 14 to 17 years. Several reasonable variations to our methodology are discussed. All of these would only result in findings even more indicative of an absence of any plain packaging effect.

Yet ultimately from a policy perspective Diethelm and Farley came to precisely the same result – there is no evidence to support the hypothesis that plain packaging contributed to a decline in smoking prevalence amongst Australian youth. Rather the difference between the two sets of results is why that may be – either the policy is/was a dud or statistical techniques are too blunt to measure the true impact of the policy.

That in itself is an interesting argument – yet the fact remains that anti-tobacco activists are unable to deploy sufficiently powerful statistical techniques to demonstrate the policy actually worked.  That means, in plain English, that there is no scientific evidence the policy worked.

I have a series of other quibbles about their study – for example, they date the policy has having started in November 2012 and not December – but that doesn’t really matter given their failure to reject the null hypothesis, that is fatal to their overall argument.

* email sent 28/11/2015:

Hello
I am interested in replicating the Deithelm and Farley results and wish to follow up on this statement in their paper:
“see description of method, Python program and reconstructed data points in on-line material”
Unfortunately I am unable to find any online material that has any such description etc.

Could you please point me in the right direction?

That email was sent to both the editor of the journal and Pascal Diethelm.

This entry was posted in Plain Packaging. Bookmark the permalink.

17 Responses to Failing to reject the null hypothesis

  1. Up The Workers!

    Facts.

    They’re lost on the Left.

    (And THAT’S a fact!).

  2. rich

    When a study is funded, the desired result is funded…

  3. Mundi

    Why would they release their data, that would help others point out their mistakes and mistruths. It would force them to do real science.

  4. entropy

    It’s preety embarrassing though to make a definitive conclusion when your own paper points out the result is not significant. I find this a particular problem with modellers too.

  5. Nerblnob

    The rate of expansion of the anti smoking industry far exceeds the rate of decline in smoking which has been fairly steady and started before the whole circus really got going.

    And if these people are taking public money then they to are funded by the tobacco industry.

  6. John Bayley

    Those two sound like they learned their ways from the ‘climate scientists’.
    Wasn’t it the infamous Phil Jones, who when asked by Steve McIntyre to provide his data for verification, retorted along the lines of ‘why should I give it to you, if you’re just going to try to show that there’s something wrong with it?’

  7. Senile Old Guy

    Abstract:

    Our findings suggest a decline of smoking prevalence in minors following the introduction of plain packaging in Australia. They differ substantially from those presented in an industry-funded study on the effects of plain packaging on smoking prevalence in minors in Australia, which used the same data.

    Actual Result – taken from the conclusion of the same study (emphasis added):

    … our logistic regression analysis suggests a plain packaging effect in the expected direction, although this is not statistically significant, the data set on minors being too small and thus lacking the power needed to reach a firmer conclusion.

    Just to confirm – the plain packaging effect in the Diethelm and Farley paper has a p-value of 0.15 – well above the generally accepted levels of 0.01, 0.05 or, if you are scraping the barrel, 0.1.

    Only in very unusual circumstances would you use 0.1 as the cutoff for significance. At 0.15, there is nearly one chance in five that the results are due to chance rather than cause. That is simply useless.

    Now when undertaking statistical research one might set up the analysis as follows:

    Null hypothesis: Policy had no impact on outcomes (for example, the introduction of plain packaging had no impact on smoking prevalence).

    Alternative hypothesis: Policy had an impact (for example, the introduction of plain packaging reduced the prevalence of smoking).

    What Diethelm and Farley find and report is that they were unable to reject the null hypothesis. That is, they cannot find evidence for the alternate hypothesis. Now I’m happy to accept their excuses for failing to reject the null – low power and small sample size. But the fact of the matter remains that they failed to reject the null.

    Sinc, I am not happy to accept their excuses because “low power and small sample size” are evidence of poor planning. Yes, these excuses are commonly used but they are still the results of poor planning.

    People using the “low power; small sample size” argument are usually trying to argue that there was an effect which they would have detected with a more powerful study. Russell Lenth writes about this. As you say, “the fact of the matter remains that they failed to reject the null”. But you need to go further: therefore, at p = 0.15, their study supports the view that policy had no impact.

  8. NB

    @ John Bayley – Yes, poor dears. They have taken some lessons from the climate industry, but not enough. A few tweaks here and there in the data would have shown a much more acceptable result. And why on earth actually offer the data? Declare it copyright or something. And massage the media and your industry colleagues properly to get the appropriate hysterical support that indicates that any opposition will be met with crucifixion.

  9. H B Bear

    Keep whacking them Snic. So much Crapmannian “science” out there.

    A blast from the past too. I wonder who will be the new Nanny Roxon when the PeanutHead union puppet regime takes over from the Lieborals?

  10. Bruce of Newcastle

    What Diethelm and Farley find and report is that they were unable to reject the null hypothesis. That is, they cannot find evidence for the alternate hypothesis.

    I wonder if this is another omitted variable fallacy issue. These people are extremely sensitive to Roxon’s PPL but there has been little coverage of a related change which occurred at the same time: the requirement to hide all tobacco products from view by the retailer. So retailers were forced to install plain cupboards to hide cigarette packets in, out of sight of the little loves who might otherwise salivate at the sight of them.

    You can see the dates of comparable legislation brought in by all the states here:

    State and territory legislation

    Roxon’s PPL started on 1 Dec 2012. The state’s display legislation all came in around the same date:

    NSW 1 July 2010
    Vic 1 January 2011
    SA 1 January 2012
    QLD 18 November 2011
    NT 2 January 2011
    WA 22 September 2010
    Tas February 2011
    ACT 1 January 2011

    As you can see the display prohibition came in within about 2 years of the national PPL, which in terms of the coarse nature of the response data is effectively simultaneous timing. So it is entirely plausible that their tiny “reduction” in smoking rate is simply because of the inhibitory fear effect of forcing minors to ask retailers for cigs out of the cupboard.

    Did they control for this variable? I don’t see how they could’ve.

  11. Mundi
    #2569465, posted on November 29, 2017 at 6:44 am

    Why would they release their data, that would help others point out their mistakes and mistruths. It would force them to do real science.

    And

    John Bayley
    #2569488, posted on November 29, 2017 at 7:50 am

    Those two sound like they learned their ways from the ‘climate scientists’.
    Wasn’t it the infamous Phil Jones, who when asked by Steve McIntyre to provide his data for verification, retorted along the lines of ‘why should I give it to you, if you’re just going to try to show that there’s something wrong with it?’

    Here is Phil Jones direct quote from the East Anglia emails

    Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it.

    There was always bad science, some deliberate and some just bad science. However since the Global Warming crowd demonstrated unequivocally that fraudulent science pays off and pays off big time up to and including Nobel prizes, fraudulent science is the norm.
    That’s why I never, never ever take the word of a scientist. In fact, whenever I hear “scientists say” my default position is that I’m being misled.
    Sinc and others would be wise to do the same IMHO. You can believe me…I’m no scientist.

  12. Steve

    @Sinclair Davidson

    Just to pick up on your point about not getting access to their original data and python scripts, it can be found here:

    http://www.tobaccopreventioncessation.com/Refuting-tobacco-industry-funded-research-empirical-data-shows-decline-in-smoking-prevalence-following-introduction-of-plain-packaging-in-Australia,60650,0,2.html

    under the ‘Supplementary material’ heading.

    It’s a PDF outlining their reverse-engineering of the plot from an earlier paper by using Photoshop and some python image recognition techniques to recreate the data from the plot.

    My personal opinion is this methodology (rather than obtaining the raw original data) has the potential to lead to errors, but at least they’ve provided a pretty comprehensive description of how they undertook the reverse-engineering

  13. DM OF WA

    Davidson, from your email request to the journal should we conclude that the data and code relating to this paper has not been made available? If so I would be disappointed. The case for releasing this sort of related statistical material in due course is clear and I thought that researchers are required to do so these days. Maybe the government via the Australian Research Council needs to be pushing for this as best practice?

  14. .

    The ARC has very little to do with best practice.

    Don’t we all get sick of asking questions we already know the answer to?

  15. Sinclair Davidson

    from your email request to the journal should we conclude that the data and code relating to this paper has not been made available?

    Yep. Despite them claiming the data were available, they didn’t even email back to say “piss off”.

    Given that the Australian government itself refuses to release a very similar data set, I wouldn’t be holding my breath waiting for them to push for best practice.

  16. Chris

    Just reading Cialdini’s ‘Pre-suasion’, a followup to ‘Influence: Science and Practice’ which focuses on how to create the moment and mindset where the person will accede to what you want to ask them.
    On p72 they have a pic of scary cigarette package photos and asserts they have reduced smoking around the world. The author says that

    presenting the most frightening consequences of bad habits works better than showing the good consequences of good habits. In over a dozen countries, placing large scary images on cigarette packages has had the double-barrelled effect of convincing more nonsmokers to resist and more smokers to stop the practice.

  17. On p72 they have a pic of scary cigarette package photos and asserts they have reduced smoking around the world. The author says that

    presenting the most frightening consequences of bad habits works better than showing the good consequences of good habits. In over a dozen countries, placing large scary images on cigarette packages has had the double-barrelled effect of convincing more nonsmokers to resist and more smokers to stop the practice.

    Reduce the price of cigs back to $5 per packet and see how well those scary images work.

Comments are closed.