ABC Fact Check: Error-ridden and misleading IV

So let’s have a look at the quality control practiced at the RMIT-ABC Fact Check unit. This is how they self-describe their quality control:

Once the director approves a claim, one of our researchers contacts experts in the field to seek their opinion and guidance on available data.

We may also contact the claimant to ask for the basis of the claim.

The expert opinion and data is written into a draft, which is then reviewed by our chief fact checker, who identifies problems, and challenges the researcher on anything that they might have missed.

The chief fact checker also scrutinises all sources and makes sure the draft is consistent with what the data says.

The researcher continually reworks the draft based on this feedback, and once the chief fact checker is satisfied, the team discusses the final verdict

These discussions are rigorous and much thought is given to the verdict word and the colour that will be used, which is an important part of how we inject nuance into our verdicts.

Our online editor then prepares the final product, which is once again checked by the chief fact checker for any inaccuracies which may have crept up during the editing process.

Once the director signs off on the finished draft, it’s ready to be released to the world.

Sounds very impressive. The director has to give approval for a fact check and then the chief fact checker checks the facts not once but twice.

It obviously never once occurred to him that there might be conflict of interest problems with the RMIT-ABC Fact Check unit fact checking a criticism of the ABC by a pair of RMIT employees. Certainly that declaration is missing from the body of the fact check.  You’d also think that they do a better job. But no.

Apparently careful thought is given the whole process.

Yet none of this prevented the RMIT-ABC Fact Check unit from identifying Chris Berg as not being from RMIT, but rather an IPA employee.

Yet none of this prevented the RMIT-ABC Fact Check unit from not reporting that “appropriate statistical methods were used in testing for differences between sub-samples to take account of the smaller sample sizes” in the original study we rely on.

Yet none of this prevented the RMIT-ABC Fact Check unit from not reporting that their own commissioned research showed that our estimate of Greens voting relative to the general population was well within their 95% confidence interval.

Yet none of this prevented the RMIT-ABC Fact Check unit from criticising us for comparing survey data to Newspoll data when we did no such thing.

Yet none of this prevented the RMIT-ABC Fact Check unit from reporting that the original survey “showed no evidence that ABC journalists were five times more likely to vote for the Greens than the general public” when they knew full well we had compared the survey data to election data.

Or from reporting that the University of the Sunshine Coast had been rebranded as the Queensland University of Technology!

Really? Why anyone wouldn’t know about QUT and University of the Sunshine Coast being separate institutions let alone the author of the fact check, and the chief fact checker who had checked it twice!

Anyway – a nice correction at the bottom.

Editor’s note (13/07/2018): A previous version of this fact check contained a line which incorrectly said that the University of the Sunshine Coast had been rebranded as the Queensland University of Technology. The two are separate institutions which both still operate today.

This entry was posted in Books and writing, Media, Shut it down. Fire them all., Taking out the trash. Bookmark the permalink.

11 Responses to ABC Fact Check: Error-ridden and misleading IV

  1. Once the director approves a claim, one of our researchers contacts experts in the field to seek their opinion and guidance on available data.

    We may also contact the claimant to ask for the basis of the claim.

    The expert opinion and data is written into a draft, which is then reviewed by our chief fact checker, who identifies problems, and challenges the researcher on anything that they might have missed.

    Funniest thing I’ve read for a while. A case of the blind leading the blind and asking the blind for street directions.

  2. RobK

    ABC fact check has no credibility. It is as biased as the rest of the organization.

  3. Suburban Boy

    The “fact-checking” process is a media scam.
    The two most important ways in which media outlets distort stories are selection and framing. Of necessity they have to select not only which stories to report, but which elements of those stories. And the context in which the factual content is presented (framing) can radically change the perception and interpretation of the facts.
    So even if the media outlet gets the facts right (and obviously it doesn’t always do so), it can easily distort the reader’s (viewer’s, listener’s) understanding by the dishonest use of selection and framing.
    But none of that is picked up by the “fact-checking” sleight-of-hand so the story can still get a clean bill of health.

  4. Roger W

    Love it when the children think they can play with the adults and then get reminded of their place.

  5. Dr Fred Lenin

    Fact checking is leftese for ignore critics and obfuscate real truth with dogmatic dialogue,it’s an old communist trick to look legitimate .

  6. jock

    Stunned that you think this unit is anything other than a job creation effort for the husband of a certain “journalist” and other hasbeen abc employees. If they want to comply with the small budget saving then getting rid of this lot would have to save 1 or 2 mill.

  7. Ubique

    It reminds that what was to be the Tasmanian Institute of Technology (later becoming the Newnham Campus of the University of Tasmania) had to be named the Tasmanian State Institute of Technology for appropriate acronym purposes.

  8. old bloke

    jock
    #2766815, posted on July 18, 2018 at 5:21 pm

    Stunned that you think this unit is anything other than a job creation effort for the husband of a certain “journalist” and other hasbeen abc employees. If they want to comply with the small budget saving then getting rid of this lot would have to save 1 or 2 mill.

    Didn’t Gillard gift them an additional $10M to setup this fact checking unit?

  9. Robber Baron

    “I don’t agree with your facts Professor, I prefer these.”

  10. Mother Lode

    One of the problems would seem to be that they consult experts (presumably identifying them by having previously given ‘sound’ results) and then non-experts interpret and edit it according to their non-expert lights.

    With regards the 95% range of confidence – I will wager an ABC know-all would reckon that being near the upper end of the range means less likely. That if the 95% confidence range was between 2 and 4 then the middle of this dance, 3, was more probable that 2.5 or 3.5. Just average it out.

  11. Tony M

    My own recent feedback to the ABC (via their complaints page) on another recent Fact Check:

    “You recently published a “Fact Check” piece on Craig Kelly’s comparison of Tesla vs Corolla emissions, in the context of the debate over subsidies to electric vehicles. >(http://www.abc.net.au/news/2018-03-07/fact-check-does-corolla-emit-less-than-tesla/9461096).

    Your verdict – Mr Kelly is “Cherry-picking” (that is, according to Wikipedia, “the act of pointing to individual cases or data that seem to confirm a particular position while ignoring a significant portion of related cases or data that may contradict that position.”)

    From reading your article, it would seem, however, that Mr Kelly is using the publically available agreed dataset to make a valid and reasonable point that is broadly supported by that data. He’s not publishing a scientific study or paper – he’s using the data to formulate and ask reasonable questions.

    Mr Kelly:
    * Was clear where the data he had came from.
    * Used the currently available agreed database – whether the data there was completely current, it is considered an agreed source. And the new data does not vary markedly from the old data.
    * Used the agreed methodology for comparison – contrary to UWA Professor Thomas Bräunl, it is entirely irrelevant whether different cars create different emissions in different circumstances (the Continuum fallacy). The only agreed comparison is precisely the one that Mr Kelly used. You compare apples to apples.
    * Made the rational comparison of a Tesla (likely to be bought by a well-off inner city resident) with a Corolla (likely to be bought by a working family in an outer suburb). His point (as mentioned in the article) is low income earners (who overwhelmingly buy Corollas) subsidizing high income earners (who are buying Teslas, not Renault Zoes).
    * Is broadly correct in his analysis and his conclusion – you don’t at any point acknowledge that, rather than arguing about differences of +/-5% between the emissions of various Teslas and Corollas, the larger point is that they ARE broadly comparable. This would be genuinely surprising to most Australians, who would assume that the electric vehicle was MUCH lower than the petrol equivalent (e.g. https://www.theguardian.com/environment/2017/oct/25/electric-cars-emit-50-less-greenhouse-gas-than-diesel-study-finds).

    As such, this piece is clearly in breach of 4.1 (Gather and present news and information with due impartiality), and 4.5 (Do not unduly favour one perspective over another) of the Editorial Standards from the Code of Practice. It is not a fair representation of either Mr Kelly’s specific statements, or his overall questions and argument. He’s not cherry-picking, he’s correct. “

    No response.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.