Auditing the ERA

I have a paper that has just been published in Agenda: A Journal of Policy Analysis and Reform looking at the federal government’s audit of research quality in Australia. As luck would have it, the head of the ARC had an op-ed in The Australian defending the ERA process just a couple of weeks ago.

Given the size of this investment, it is important to ensure that university research is of the highest quality. Excellence in Research for Australia has broad acceptance, but some commentators continue to disparage it.

Really? Some people don’t trust the government audit of research quality?

Claims ERA is a blunt measure of research are simply wrong. Decisions are based on extensive information, including reports from peer reviewers or comprehensive citation analysis and benchmarking. Universities also provide contextual information about research income, patents, and esteem measures for their staff.

Unlike other national systems, which use citation data in an ad hoc way, the Australian Research Council has developed a world-leading citation methodology. The use of peer review or citation analysis for a particular discipline is strongly guided by detailed and ongoing consultation with the research community.

Well, yes. This is how I describe the ERA process.

The ERA 2012 Evaluation Handbook claims several objectives with the first being:
• Establish an evaluation framework that gives government, industry, business and the wider community assurance of the excellence of research conducted in Australia’s higher education institutions.
To that end the ERA defines a five point scale:
5. “… well above world standard …”
4. “… above world standard …”
3. “… at world standard …”
2. “… below world standard …”
1. “… well below world standard …”
It is quite remarkable that the ERA does not define what “world standard” is, but rather tells us what it is not. We are told, “’World Standard’ refers to a quality standard. It does not refer to the nature or geographical scope of particular subjects, or to the locus or research not its place of dissemination”. As reassuring as that is there is still no definition of the “quality standard” that “world standard” entails. Further, “The ERA evaluation measures research quality, not scale or productivity.” Indeed, “The methodology and rating scale allow for [units of evaluation] with different volumes of output to achieve the same rating.”

Universities submit data to the Australian Research Council (ARC) that administers the ERA. That data relate to so-called Units of Evaluation i.e. a research discipline within a university. That Unit of Evaluation need not be a single department or school within a university. The Australian Research Council recruits individuals for Research Evaluation Committees and individuals who undertake peer review. Evaluation takes place, outcomes determined, and results published. The peer reviewers and members of the Research Evaluation Committees are bound by very strict confidentiality clauses.

[Research Evaluation Committee] members and Peer Reviewers are required to sign a confidentiality agreement with the ARC prior to their participation in ERA. The agreement covers all aspects of their work with ERA, and the agreement survives the conclusion of their engagement for the purposes of ERA.
[Research Evaluation Committee] members and Peer Reviewers may not contact researchers and/or institutions under any circumstances in relation to material that has been submitted for evaluation in ERA, or seek additional information from any sources. [Research Evaluation Committee] members and Peer Reviewers may not reveal details about any evaluation at any time.

The most important consideration, however, is this (emphasis added):

[Research Evaluation Committee] members exercise their knowledge, judgement and expertise to reach a single rating for each [Unit of Evaluation]. … The rating for each [Unit of Evaluation] reflects the [Research Evaluation Committee] members’ expert and informed view of the characteristics of the [Unit of Evaluation] as a whole.

In other words, the “evaluation framework” that the ERA provides is not a transparent and replicable process; but rather an exercise where unknown individuals, acting in secret, selected by the government (or its agencies), express an opinion as to the quality of research relative to an undefined benchmark.

So this is what I do:

The reality is that the ERA report results rely on some courageous assumptions. First, that government (or its agencies) can define quality. Second, that government (or its agencies) can measure quality. Third, that quality can be sufficiently represented in a single number between 1 and 5.

In this critique I do not challenge these fundamental assumptions important as they are. Nor do I attempt to critique the ERA ranking process relative to the existing literature that attempts to provide similar rankings (for example see Rodgers and Valadkhanim 2006 for a recent Australian example of this literature). I simply investigate whether ERA rankings are consistent with objective information.

To that end I calculate the average citations for articles published between 2005 and 2010 for the 30 Australian universities that have a 2012 ERA ranking in the area of “Applied Economics”. I employ the same global database that the ERA use. I investigate that data along with an alternate measure of quality (the H-Index). Finally I relate the ERA rank to the objective data I have gathered and discuss the anomalous results.

So what is the bottom line?

Ultimately, it is not at all clear why there is such a large discrepancy between the actual ERA Ranks and objective information that can be derived from Scopus – the ERAs chosen dataset. It is not surprising that most Australian universities should score at the world standard (however defined). What is surprising is that the Australian government and its research agency would claim that 14 out of 30 Australian universities were below the world standard without bothering to define what that standard might be. It is even more surprising that several universities could be ranked at above the world standard (or even well above the world standard) without any reliable objective evidence to support that ranking except for the “expert and informed view” of the individuals making the ranking.

This entry was posted in Economics and economy, Education, Uncategorized. Bookmark the permalink.

14 Responses to Auditing the ERA

  1. YAHOO – best news for today.

    Sure, Australia’s research is world standard. We’re part of the world, ain’t we? Sure as hell ain’t on the Moon.

    [spits baccy]

  2. As reassuring as that is there is still no definition of the “quality standard” that “world standard” entails.

    World standard: Research that is produced at any point somewhere in the world.

    Further, “The ERA evaluation measures research quality, not scale or productivity.” Indeed, “The methodology and rating scale allow for [units of evaluation] with different volumes of output to achieve the same rating.”

    Quality: Research produced by the biggest universities with the most money over the longest period of time.

    There you go, Sinc. Happy Christmas.

  3. Bruce

    That looks like a worthy attempt Prof Davidson. I know from much experience that evaluation of research projects is very difficult.

    The current system favours clique formation and pal review. That goes for all subjects (psychology, economics, medical science to name a few), and therefore the consensus will tend to hold and gain the most support in prestige and ARC money just because the consensus (whatever it might be in any particular field) will have the numbers to get their favoured academics’ projects up.

    My feel is the best approach is to get rid of as much of the bureaucracy as possible, eg the ERA, and use the most transparent and competitive funding system possible, probably with an external open blog-style review and a for-and-against appeal mechanism. Then let the best ideas fly.

    Trying to evaluate research excellence is so much in the eye and tribalism of the beholder that it must be made as transparent and as open as possible. If there are problems with distribution of cash between different fields (eg sexy vs non-sexy) then do a high level proportional allocation to each field.

    When I did postgrad several decades the untenured supervisors were pushing hard to publish as much as possible even back then. Turning the handle on the sausage machine. Bad, bad stuff. Its a big reason I escaped out to industry. Now, with more money floating around, my impression is its much worse.

  4. Stateless, free and happy

    Nice paper Sinclair.
    These assessors come with priors. They are human after all. The priors must surely be that Go8 MUST be above world standard and the others (RMIT, Deakin and the like) must be ranked below the Go8. This is so anti intellectual and prone to confirmation bias. The current ERA process is very poor science.

    UTS result was surely driven by 2-3 big names.

  5. Stateless, free and happy

    Sinclair, there is another consequence of the ERA rankings. UTS score of 5 has given them greater influence, e.g. in the ABDC ranking exercise, the UTS submission was given greater weight.

    Real consequences from poor science.

  6. cohenite

    While ever people like Lewandowsky and Cook are given peer review status something is wrong.

    The emails reveal the extent of the cronyism and intent to preserve the status quo; if it happens in climate science then I suppose it must happen generally, although AGW ‘research’ is currently where the bucks are.

    As a layman I am absolutely unsurprised that scientists should be as venal, vain, money and power hungry and hypocritical as the hoi polio. Being smarter their failings are just better and more pronounced.

  7. dd

    What is surprising is that the Australian government and its research agency would claim that 14 out of 30 Australian universities were below the world standard without bothering to define what that standard might be.

    What is a ‘world standard’ anyway?

  8. sabrina

    An excellent analysis Sinclair. I hope this article gets all the visibility it deserves. Ahead of the next ERA, jockeying for prolifically publishing academic authors (with high citations) by universities having began.
    This citations centric exercise has given rise to an undesired explosion of journal papers. I do not track your area, but majority of the recent journal papers in the energy and fuels area are of questionable quality, and have dubious prospects of any real life application. I also notice the rapid rise of self-citation as eveident from Google Scholar (freely accesible) and of course Scopus. This indirectly increases the journal’s impact factor, and then the author’s H-index

    I note your H-index is low Sinclair for reasons I think I understand, but the type of your papers have good policy implication. Keep up the good work. Merry Christmas.

  9. .

    I have had a really bad run with peer review.

    One paper got published nearly unchanged from the original after we were told to totally change our methodology after an underling promoted above their ability overstepped their authority and deluded themselves as to their ability and self importance. In essence we had to write the same paper twice, with a stop at writing a bogus paper no one actually wanted along the way.

    Another paper was rejected, with some helpful comments about publishing generally prior to that. I took their recommendations on board. The other reviewer of this earlier paper spouted nonsense about variables and was totally wrong, did not understand the theory (they could have at least read a textbook or another paper if they did not know) yet was totally condesending. Some of their criticism was contradicted by the better reviewer.

  10. Jim Rose

    The best papers take time to be recognized. Few top papers receive immediate aclaim

  11. I get the impression that a lot of researchers, especially in CSIRO, do not even have PhDs. Anyone seen any breakdown on Masters vs Doctorates etc in Australian research?

  12. World

    World Standard is usually used as a marketing tool to describe and empty vessel. Its a bit like an imaginary cherry in the middle of an oasis which disappears the closer you get to it.
    For example, the standard of Englands bowling attack was world class until it was belted all over the place,looked very ordinary at bes and appeared to be running on empty.

  13. World

    Sorry about that . I wanted to end the comment on English hyperbowers but I bowled a world class wide.

  14. James Hargrave

    Of course, ERA rhymes with error; and ARC, the ‘c’ is soft’ rhymes with…

    This concern with world standards might best be applied to the ARC itself. Is it, by its own convoluted non-definition, a world standard research council, or merely a collection of provincial hicks on a trading estate near Canberra airport? From experience, the national research councils in parts of the former Soviet Union are run with more directness and efficiency.

Comments are closed.