I am currently in the process of writing up a paper on Australian tobacco control policy – in particular the 2012 plain packaging/graphic health warning policy. In the excerpt below I summarise several of the arguments and many of the posts that have appeared here at the Cat.
Constructive feedback happily received. (For reasons I don’t know, the footnote function doesn’t work on the blog, so the footnotes are really endnotes).
Since December 2012 all tobacco products (legally) sold in Australia are required to be packaged in a standardised package. In Australia this is known as the “plain packaging” policy. Since then several countries, including the United Kingdom, Ireland and France, have announced that they too will adopt a plain packaging policy for tobacco products.
The objectives of the policy are set out in section 3 of the Tobacco Plain Packaging Act 2011, No. 148, as amended:
(1) The objects of this Act are:
(a) to improve public health by:
(i) discouraging people from taking up smoking, or using tobacco products; and
(ii) encouraging people to give up smoking, and to stop using tobacco products; and
(iii) discouraging people who have given up smoking, or who have stopped using tobacco products, from relapsing; and
(iv) reducing people’s exposure to smoke from tobacco products; and
(b) to give effect to certain obligations that Australia has as a party to the Convention on Tobacco Control.
(2) It is the intention of the Parliament to contribute to achieving the objects in subsection (1) by regulating the retail packaging and appearance of tobacco products in order to:
(a) reduce the appeal of tobacco products to consumers; and
(b) increase the effectiveness of health warnings on the retail packaging of tobacco products; and
(c) reduce the ability of the retail packaging of tobacco products to mislead consumers about the harmful effects of smoking or using tobacco products.
In short, the objective of the policy is to reduce the prevalence of tobacco consumption in Australia by reducing the appeal of tobacco products, and enhancing the health warnings associated with tobacco consumption. A test of the efficacy of the policy would demonstrate that the prevalence of smoking had declined due to the introduction of the policy.
Unfortunately, and despite assurances from the Australian government and the Australian public health lobby, that evidence is simply non-existent.
The Australian government and public health lobby have pointed to five pieces of evidence to support the notion that the plain packaging policy has been successful.
- A 3.4% reduction in tobacco clearances.
- A reduction in household expenditure on tobacco.
- Victorian Cancer Council Fact Sheets showing survey data of smoking prevalence.
- Surveys undertaken by Melanie Wakefield and her research team.
- Regression analysis presented in the Post-Implementation Review.
I discuss each of these claims in turn.
1. The 3.4% tobacco clearance myth
In June 2014, the Fairfax media claimed that the Australian government Treasury had “entered the debate over cigarette sales, publishing previously secret information that shows sales falling since the introduction of graphic health warnings and plain packaging”. In particular, Treasury was said to have made the claim: “3.4 per cent fewer cigarettes were sold last year than 2012”. The Health Department subsequently included that claim on its web based fact sheet. Originally it stated:
The Commonwealth Treasury has further advised that tobacco clearances (including excise and customs duty) fell by 3.4% in 2013 relative to 2012 when tobacco plain packaging was introduced.
That statement was subsequently amended:
Treasury has advised that tobacco clearances (including excise and customs duty) fell by 3.4% in 2013 relative to 2012 and fell a further 7.9% in 2014. Tobacco clearances have fallen a total of 11.0% since 2012 when tobacco plain packaging was introduced.
These growth rates do not take into account refunds of excise equivalent customs duty made under Customs’ plain packaging related Tobacco Refund Scheme between December 2012 and May 2013. These refunds cannot be related to annual net clearances on a comparable basis to other data used to derive these growth rates.
The addition of the second paragraph is very telling – and damning. The initial media coverage was very suspicious – no Treasury official was interviewed, no Treasury official was quoted, and no Treasury document was cited. That, however, did not stop the Australian government official news agency, the Australian Broadcasting Corporation from running with the story.
The Department of Treasury keeps records on the sales of cigarettes for taxation purposes, but has never before made the information publicly available.
The Treasury data reveals that 3.4 per cent fewer cigarettes were sold last year than 2012.
After the story went to air, the Australian Broadcasting Corporation admitted that the tobacco industry had challenged that 3.4% figure:
While we don’t know the full detail of Treasury’s tobacco clearances from their statement, from Philip Morris’ perspective, the final quarter of 2012 saw an artificially high rate of tobacco clearances due to our replacement of branded stock on retailers’ shelves with plain packaged stock. Whilst this was not double-counted from an industry sales perspective as it was replacement stock, it would have initially been double-counted from a 2012 tobacco clearances perspective as tax must be paid on every pack. Most claims for refunds of the excise paid on our recalled branded stock were not processed until the first quarter of 2013.
It turns out that the 3.4% figure – which has been a very influential statistic – is misleading. This only became apparent once Treasury was required by a Freedom of Information request to report the data for tobacco clearances.
What Treasury had done was as follows: it calculated total tobacco clearances for 2012 and then for 2013 and calculated the difference between the two. When that exercise is performed it is correct to say that tobacco clearances fell between 2012 and 2013 by 3.4%. There are, however, two complications:
- Plain packaging was introduced in December 2012 not January 2013.
- As the industry claimed, and the Health Department subsequently admitted, a large number of refunds were made in 2013 for excess payments made in 2012.
When the 12 month period before December 2012 is compared to the 12 month period after December 2013, then tobacco clearances fell by 0.8% – that is, less than one per cent.
When you also take into account the double counting of excise paid in 2012 and the refunds in 2013, it appears that tobacco clearances increased by 0.5%.
It is correct that tobacco clearances increased by less than 1% – but that remains a 3.9% turn around on the number initially quoted by Treasury and the Health Department.
New South Wales Senator David Leyonhjelm of the Liberal Democratic Party has been closely questioning both Treasury officials and Health Department officials over the claimed 3.4% decline in tobacco clearances. Having previously confirmed the 0.8% figure to be correct, excluding the refunds, he recently posed a question to the Health Department:
… today I received a reply to a question on notice from Treasury which advised that in period of the 12 months ended November 2013 and the 12 months ended 30 November 2012 there was a 0.8 per cent decline in tobacco clearances, excluding tobacco refund scheme refunds. … Do you intend to modify your website to say that, comparing like with like, the reduction was 0.8 per cent? It gives the impression that it had an immediate, substantial impact on clearance rates.
[Health Department Official]: We have no intention of suggesting that clearance rates are a direct measure of tobacco plain packaging effects. In fact, they are not designed to measure the effects of plain packaging or, indeed, any particular tobacco control measure.
In short – Treasury has abandoned the claim that tobacco clearances fell by 3.4%, but the Health Department will not withdraw their false claim; instead the Health Department now claims the Treasury data are not definitive.
2 Australian Bureau of Statistics Household Expenditure data
The Australian Bureau of Statistics provides an estimate of Household expenditure data on tobacco products. This forms part of their estimates of Gross Domestic Product. Figure 1 shows seasonally adjusted Household Final Expenditure on cigarettes and tobacco data.
Figure 1: Household Final Consumption Expenditure on Cigarettes and tobacco: Chain volume measure, seasonally adjusted.
Source: ABS Cat. 5206.0 Table 8.
It is important to note that Household expenditure on tobacco products has been falling since the early 1980s. Any evaluation of the plain packaging policy must be undertaken in the context of a long-term downward trend within household tobacco expenditure. Looking then at the period immediately before and subsequent to the introduction of plain packaging we see a very significant deviation from the declining long-term trend. In figure 2 below I have accentuated the graph in order to highlight the change in trend. In December 2012 – the month plain packaging was introduced in Australia – household expenditure on tobacco products was $4.134 billion. By December 2013 – exactly one year later and when the Australian government imposed a 12.5% increase in tobacco excise – household expenditure on tobacco products had increased by some 2.25% to $4.228 billion. Following the increase in excise and changes to excise indexation and then subsequent excise increases household expenditure on tobacco resumed its long-term decline. That increase in household expenditure over the year December 2012 through December 2013 is in marked contrast to what a Health Department official told the Australian Senate in February of this year:
They are one indication amongst many of whether plain packaging is working, including the ABS household expenditure data, which has, between September 2012 and September 2013, dropped by a total of 20 per cent.
It is not immediately clear why she chose September 2012 to September 2013 as her comparison period, but even then the decline in household expenditure on tobacco products, according to the Australian Bureau of Statistics was 1.8%.
Figure 2: Household Final Consumption Expenditure on Cigarettes and tobacco March 2012 – December 2015: Chain volume measure, seasonally adjusted.
Source: ABS Cat. 5206.0 Table 8.
3 Victorian Cancer Council Fact Sheets
The available evidence suggests that plain packaging is likely to be contributing along with other tobacco control policies to continuing reductions in the prevalence of smoking in Australia
The question, however, is not whether Australia’s tobacco control policies are successful – clearly they are – but rather whether the plain packaging policy is successful in reducing the prevalence of tobacco consumption.
On that latter question, the Victorian Cancer Council fails to produce any evidence in its factsheets. To the contrary it provides the following graph (their figure 5):
Figure 3: Victorian Cancer Council figure of smoking prevalence in Australian mainland states
Source: Fact sheet no. 4: What is happening to the prevalence of smoking in Australia?
The whiskers on the bar graphs represent the confidence intervals of each survey. The Victorian Cancer Council interpret over-lapping confidence intervals as representing no statistically significant different between two surveys. Whether or not that is an appropriate measure of statistical significance is debatable – more precise techniques exist. Nonetheless, two points are immediately obvious in figure 3.
- In four of the five Australian mainland states smoking prevalence increased in 2013 compared to 2012.
- None of the changes in smoking prevalence are statistically significantly different from zero if we accept overlapping confidence intervals to be an appropriate measure of statistical significance.
4 The Wakefield Tracking Survey
In 2012 Professor Melanie Wakefield of the Victorian Cancer Council was awarded a $3 million contract to conduct a national tracking survey of tobacco consumers (and recent “quitters”) immediately prior, during, and after the implementation of plain packaging. Professor Wakefield has previously been a member of the National Preventative Health Taskforce that had recommended the implementation of plain packaging, she was a member of the Federal Government’s Expert Advisory Group on plain packaging, and was then was commissioned by the Health Department, in the absence of a tender process, to investigate the efficacy of the very policy she had recommended, designed. and implemented. Unsurprisingly the results of her research (with several co-authors) supports the efficacy of plain packaging as a policy to reduce the prevalence of tobacco consumption.
The results of the National Tracking study have been reported in a special issue of Tobacco Control. Details as to the study itself are available from the Health Department. It is also possible to request copies of the data. The National Tracking Study canvassed many issues and consisted of 26 waves of approximately 400 interviews of current smokers and recent “quitters”. There were also follow up interviews a month after the initial interview.
What is remarkable about the special issue of Tobacco Control is that it claims that the plain packaging policy was successful. Yet not one of the papers demonstrates success as set out by section 3(1) of the Tobacco Plain Packaging Act. This has not gone unnoticed by Senator Leyonhjelm:
Senator LEYONHJELM: … Your department’s website says that the key findings of the survey were that the objectives of tobacco plain packaging were achieved. Given that is a departmental website—we are not referring to Professor Wakefield’s here—can you tell me: was there a key finding from the survey that plain packaging improved public health?
[Health Department Official]: The language on the website reflects the broad findings in the BMJ articles published on 19 March last year. They were referencing the proximal objectives as they are referred to in those articles. I think the department ordinarily now refers to them as the mechanisms, which are found in section 3(2) of the Tobacco Plain Packaging Act under the objects of the act.
Senator LEYONHJELM: Yes, that is why I am asking the questions. Was there a key finding that plain packaging improved public health? That is one of the objectives.
[Health Department Official]: The tracking survey and the BMJ articles that relate to the tracking survey were not designed to measure prevalence and cannot measure prevalence.
Senator LEYONHJELM: So it did not measure whether there was increased giving up of smoking?
[Health Department Official]: As I said, the design of the tracking survey and the articles in the BMJ that discuss it largely related to the section 3(2) mechanisms—so reducing the appeal of the packet, increasing the effectiveness of graphic health warnings and minimising the pack’s ability to mislead. In the long term, those three mechanisms work to reduce prevalence.
In short, the Health Department now argues that the reported results from the National Tracking survey do not establish whether the plain packaging policy actually achieved its goals as set out in section 3(1) of the Act. Rather they establish whether the policy reduces the appeal of tobacco products and/or enhances the impact of the health warnings. Furthermore the Health Department argues that the National Tracking survey cannot answer whether or not the objectives of the Act could be met.
In particular, none of the published studies investigates whether plain packaging per se met any of the objectives of the Act as opposed to the increased graphic warnings on the packaging. What is not fully appreciated is that two policies were introduced simultaneously: plain packaging and increased size and usage of graphic warnings. The empirical work published in the special issue does not differentiate between the two policies. Indeed, the author of the empirical work presented in the Post-Implementation review argues that it is impossible to separate the two effects.
Despite the flawed nature of the empirical work that has been published from the National Tracking study, two studies in particular are said to demonstrate the efficacy of the plain packaging policy:
- Durkin et al. concludes: “These findings provide some of the strongest evidence to date that implementation of PP with larger GHWs was associated with increased rates of quitting cognitions, microindicators of concern and quit attempts among adult cigarette smokers”.
- Brennan et al. reported: “In multivariable models, we found consistent evidence that several baseline measures of GHW effectiveness positively and significantly predicted the likelihood that smokers at follow-up reported thinking about quitting at least daily, intending to quit, having a firm date to quit, stubbing out cigarettes prematurely, stopping oneself from smoking and having attempted to quit”.
It is well worth emphasising that neither of these two studies investigated whether or not smokers had actually stopped smokers (become quitters). Rather they investigated whether smokers had considered “quitting”. What makes this oversight all the more remarkable is that the National Tracking survey contained information as to whether smokers had quit between their first interview (the so-called baseline survey) and the follow up survey one month later. If they did ever investigate the relationship, if any, between plain packaging and actually quitting smoking – an objective of the policy under section 3(1) of the Act – they have never reported it in a peer reviewed journal despite having the data available to do so.
It is possible, however, to reverse out quit rates from the Durkin et al. paper’s table 1. Durkin et al. have divided the 26 waves of interviews into 4 time periods:
- A pre-PP phase.
- An early transition phase.
- A late transition phase.
- PP year 1.
Over those periods smokers and recent quitters were interviewed as to their attitudes towards smoking in general and plain packaging and larger graphic health warnings in particular. The Durkin et al. study provides a series of summary statistics for the sample they employ in their analysis including whether or not a survey respondent is a “continuing smoker” at the follow up interview. See table 1 for an extract of the Durkin et al. table 1.
Table 1: Extract of Durkin et al. Table 1 with calculated % changes.
In the table CS is a continuing smoker and BS is a baseline smoker. It is possible to calculate the percentage difference between these two numbers in each time period and thus show an estimated quit rate. Over the entire 26 wave survey period the quit rate was 5.59%. In the pre-PP time period the quit rate was 5.90% which then fell in the PP year 1 period to 5.66%. If we were to accept these data at face value, the quit rate fell after the introduction of a policy specifically designed to increase the quit rate. Unfortunately that is not reported in the study and no test of statistical significance is reported either.
Looking at Durkin’s et al. other findings, the results are somewhat weaker than the advertised conclusions would suggest. For example in those few cases where they are able to show statistically significant differences between quitting related behaviour and cognitions many of those results are confined to the late transition period (i.e. when the plain packaging policy was being introduced). In the PP year 1 time period, only three results are statistically significant. They include “concealed or covered pack several or many times in past month”. That is hardly surprising – the plain packaged packs were specifically designed to be unattractive.
In short, the results from the Wakefield tracking survey do not actually test whether the plain packaging policy met its stated objectives, they merely assert that the objectives were met. The results do not investigate whether smokers did in fact quit, despite having the data to do so. It is possible to estimate a quit rate from the published papers and that estimation shows that the quit rate fell after the introduction of plain packaging compared to the pre-plain packaging period.
5 The Post-Implementation Review regression analysis
In February 2016 the Australian government released its Post-Implementation Review of the plain packaging policy. This review included an econometric analysis undertaken by Tasneem Chipty Ph.D (MIT) of the United States based Analysis Group, Inc. Dr Chipty has been previously employed by the Australian government to provide expert evidence to the World Trade Organization on matters relating to trade mark disputes resulting from the introduction of plain packaging in Australia.
Dr Chipty employed Roy Morgan Single Source Survey data over the period January 2001 to September 2015 to model smoking behaviour. That survey consists of monthly cross-sectional surveys of approximately 4,500 respondents. In total her sample includes 177 monthly surveys. Based on that data she is able to calculate smoking prevalence in the Australian population – results are summarised in her Figure 1 (Figure 4 below).
Figure 4: Smoking Prevalence reported by Post-Implementation Review.
Dr Chipty has inserted two trend lines into the data: the blue line is a “before” trend line and the green line is an after trend line. It is not clear, however, how those two lines were estimated. It does appear, however, that there is a structural break in the data – unfortunately that appearance may well be an optical illusion given the insertion of the trend lines. Nowhere in the study does Dr Chipty explain the origins of the trend lines, nor does she ever test for a structural break at that point.
That figures does explain a statement made by a Health Official to Senator David Leyonhjelm.
[Health Department Official]: … but our modelling suggests that it is a beyond trend drop, especially since plain packaging. As I said, since 2012 it has been the most substantial drop in 20 years. We do not say that is entirely attributable to plain packaging, but it is a beyond trend drop.
Trend lines are very sensitive to chosen start and end points. Looking at the blue line in Figure 4 there are very few data points above trend after 2011. With a linear trend line we would expect approximately half the data to plot above the line and half to plot below. That suggests that the blue line is too shallow to be a unbiased estimate of trend. Similarly, the green line appears to be a good estimate of trend if the appropriate start and end points are January 2013 to September 2015.
It is also worth pointing out that those estimated smoking prevalence rates are well above National Drug Strategy Household Survey results. By September 2015 Dr Chipty estimates a smoking prevalence of some 16%, while the National Drug Strategy Household Survey reported daily smoking prevalence rates to be 12.8% in 2013. At the very least smokers are likely to be over-represented in the Roy Morgan Single Source Survey data. Without access to the underlying data it is impossible to establish whether there are any other sources of bias in the data set.
It is possible to investigate the claim that the trend changes dramatically after the introduction of plain packaging. Using the Get Data Digitiser I was able to capture the underlying data from Figure 4 – but, unfortunately, at a very high level meaning that I have a lot more x-axis points than the original. For our purposes here that does not change any of the conclusions that I draw. I then reproduce as best I can the original two trend lines, and also insert a trend line that I’m comfortable with – I have used linear regression to estimate the trend lines (readers will notice that my estimate trend lines are not entirely straight – this is due to my estimate technique). My best guess of the original trend lines have adjusted R-squares of 0.85. My trend line has an adjusted R-square of 0.91. In Figure 5 below the original trend lines are shown as red dots and my trend line is dark blue.
Figure 5: Replication of Chipty Smoking Prevalence and Trend Lines
Plain packaging is introduced at observation 705. That is where the red dotted line deviates and the second original trend begins. Yet my trend lines show an on-trend decline in smoking prevalence for another year when a massive increase in tobacco excise was implemented. It appears then that the trend lines were deliberately engineered to mask the impact of the tobacco excise increase and attribute any decline in smoking prevalence to the plain packaging policy.
I now turn my attention to Dr Chipty’s actual econometric analysis. In order to estimate the effect of the plain packaging policy (and the increased graphic health warnings) Dr Chipty estimates a probit regression model where the dependent variable is the smoking status of respondents (= 1 if a smoker and 0 otherwise). She then includes a time trend, sociodemographic variables, tobacco control indicator variables such as changes to excise policy, and an indicator variable for the 2012 policy changes (plain packaging and increased graphic health warnings). Unfortunately she omits to include a price variable – the analysis implicitly assumes that tobacco prices play no role in determining whether or not individuals will consume tobacco. At the very least the analysis suffers from omitted variable bias.
She experiments with various start dates for the 2012 policy introduction but, quite correctly, prefers the December 2012 date as being the most appropriate start date for the policy. She estimates a coefficient for the 2012 policy of -0.0237 that is statistically significantly different from zero with a p-value of 0.017. As an aside, she also finds that the introduction of graphic health warnings in 2006 is not statistically significantly different from zero. Given that her model has 786,518 observations it is unsurprising that she finds statistically significant results.
Dr Chipty then provides an economic explanation for her results. Her model estimates that by September 2015 smoking prevalence was 17.21%. In the absence of the plain packaging policy (and increased graphic health warnings) that smoking prevalence would have been 17.77%. In other words, over a 34 month period after the introduction of plain packaging (and increased graphic health warnings) that policy had contributed to a 0.55% decline in the prevalence of tobacco consumption. That is just over half of one per cent.
She interprets that result as follows:
The evidence shows that 2012 packaging changes are succeeding in reducing smoking prevalence beyond trend. In terms of order of magnitude, smoking prevalence is 0.55 percentage points lower over the period December 2012 to September 2015 than it would have been without the packaging changes. For reasons I have explained, this effect is likely understated and is expected to grow over time. This evidence supports the conclusion that the TPP Act is having its intended effect.
That is one interpretation. In the first instance the model omits important variables such as price. It does include excise policy changes but not excise itself. Tobacco excise in Australia is indexed – originally to the consumer price index and currently to increases in average weekly earnings. In addition to any changes in policy, tobacco excise increases twice every year. Tobacco companies themselves may change the prices of their products. Furthermore she has not given careful thought to the base case tobacco consumer in her model. Given the nature of the probit regression and the large number of indicator variables, the model must be estimated with a base case tobacco consumer. Dr Chipty’s base case tobacco consumer is an unmarried, male, Australian born, 14 – 17 year old, with a tertiary qualification, employed full time, but with an income less than $6000, and living in Victoria.
Of particular concern is that the pseudo-R-squares of her models (the measure of how well her models explain the data in the sample) is only 0.091. That means that her modelling cannot explain 90.9% of the variation in her dependent variable (i.e. smoking status). Add to that the average sampling error of the Roy Morgan data (for 5000 respondents the margin of error is 0.6) and our confidence in the 0.55% estimate must fall. It is hard to accept a half of one per cent difference is important when the model estimating that difference cannot explain 90% of the variation in the data, and the data itself has a bigger sampling error than the reported effect.
 Peter Martin, 2015, “Plain packaging pushes cigarette sales down”, Sydney Morning Herald, 23 June.
 Strictly speaking that claim cannot be correct. Treasury do not track how many cigarettes (or tobacco products generally) are sold in Australia. Treasury tracks how much tax has been paid on tobacco products available for sale in Australia.
 Senate Hansard Estimates 10 February 2016, http://tinyurl.com/zpwcsga
 Sarah Durkin, Emily Brennan, Kerri Coomber, Meghan Zacher, Michelle Scollo, and Melanie Wakefield, 2015, Short-term changes in quitting-related cognitions and behaviours after the implementation of plain packaging with larger health warnings: findings from a national cohort study with Australian adult smokers, Tobacco Control, http://tobaccocontrol.bmj.com/content/24/Suppl_2/ii26.full.
 Emily Brennan, Sarah Durkin, Kerri Coomber, Meghan Zacher, Michelle Scollo, and Melanie Wakefield, 2015, Are quitting-related cognitions and behaviours predicted by proximal responses to plain packaging with larger health warnings? Findings from a national cohort study with Australian adult smokers, Tobacco Control, http://tobaccocontrol.bmj.com/content/24/Suppl_2/ii33.full.
 Tasneem Chipty, 2016, Study of the Impact of the Tobacco Plain Packaging Measure on Smoking Prevalence in Australia. In Post-Implementation Review: Tobacco Plain Packaging.