I have a paper that has just been published in Agenda: A Journal of Policy Analysis and Reform looking at the federal government’s audit of research quality in Australia. As luck would have it, the head of the ARC had an op-ed in The Australian defending the ERA process just a couple of weeks ago.
Given the size of this investment, it is important to ensure that university research is of the highest quality. Excellence in Research for Australia has broad acceptance, but some commentators continue to disparage it.
Really? Some people don’t trust the government audit of research quality?
Claims ERA is a blunt measure of research are simply wrong. Decisions are based on extensive information, including reports from peer reviewers or comprehensive citation analysis and benchmarking. Universities also provide contextual information about research income, patents, and esteem measures for their staff.
Unlike other national systems, which use citation data in an ad hoc way, the Australian Research Council has developed a world-leading citation methodology. The use of peer review or citation analysis for a particular discipline is strongly guided by detailed and ongoing consultation with the research community.
Well, yes. This is how I describe the ERA process.
The ERA 2012 Evaluation Handbook claims several objectives with the first being:
• Establish an evaluation framework that gives government, industry, business and the wider community assurance of the excellence of research conducted in Australia’s higher education institutions.
To that end the ERA defines a five point scale:
5. “… well above world standard …”
4. “… above world standard …”
3. “… at world standard …”
2. “… below world standard …”
1. “… well below world standard …”
It is quite remarkable that the ERA does not define what “world standard” is, but rather tells us what it is not. We are told, “’World Standard’ refers to a quality standard. It does not refer to the nature or geographical scope of particular subjects, or to the locus or research not its place of dissemination”. As reassuring as that is there is still no definition of the “quality standard” that “world standard” entails. Further, “The ERA evaluation measures research quality, not scale or productivity.” Indeed, “The methodology and rating scale allow for [units of evaluation] with different volumes of output to achieve the same rating.”
Universities submit data to the Australian Research Council (ARC) that administers the ERA. That data relate to so-called Units of Evaluation i.e. a research discipline within a university. That Unit of Evaluation need not be a single department or school within a university. The Australian Research Council recruits individuals for Research Evaluation Committees and individuals who undertake peer review. Evaluation takes place, outcomes determined, and results published. The peer reviewers and members of the Research Evaluation Committees are bound by very strict confidentiality clauses.
[Research Evaluation Committee] members and Peer Reviewers are required to sign a confidentiality agreement with the ARC prior to their participation in ERA. The agreement covers all aspects of their work with ERA, and the agreement survives the conclusion of their engagement for the purposes of ERA.
[Research Evaluation Committee] members and Peer Reviewers may not contact researchers and/or institutions under any circumstances in relation to material that has been submitted for evaluation in ERA, or seek additional information from any sources. [Research Evaluation Committee] members and Peer Reviewers may not reveal details about any evaluation at any time.
The most important consideration, however, is this (emphasis added):
[Research Evaluation Committee] members exercise their knowledge, judgement and expertise to reach a single rating for each [Unit of Evaluation]. … The rating for each [Unit of Evaluation] reflects the [Research Evaluation Committee] members’ expert and informed view of the characteristics of the [Unit of Evaluation] as a whole.
In other words, the “evaluation framework” that the ERA provides is not a transparent and replicable process; but rather an exercise where unknown individuals, acting in secret, selected by the government (or its agencies), express an opinion as to the quality of research relative to an undefined benchmark.
So this is what I do:
The reality is that the ERA report results rely on some courageous assumptions. First, that government (or its agencies) can define quality. Second, that government (or its agencies) can measure quality. Third, that quality can be sufficiently represented in a single number between 1 and 5.
In this critique I do not challenge these fundamental assumptions important as they are. Nor do I attempt to critique the ERA ranking process relative to the existing literature that attempts to provide similar rankings (for example see Rodgers and Valadkhanim 2006 for a recent Australian example of this literature). I simply investigate whether ERA rankings are consistent with objective information.
To that end I calculate the average citations for articles published between 2005 and 2010 for the 30 Australian universities that have a 2012 ERA ranking in the area of “Applied Economics”. I employ the same global database that the ERA use. I investigate that data along with an alternate measure of quality (the H-Index). Finally I relate the ERA rank to the objective data I have gathered and discuss the anomalous results.
So what is the bottom line?
Ultimately, it is not at all clear why there is such a large discrepancy between the actual ERA Ranks and objective information that can be derived from Scopus – the ERAs chosen dataset. It is not surprising that most Australian universities should score at the world standard (however defined). What is surprising is that the Australian government and its research agency would claim that 14 out of 30 Australian universities were below the world standard without bothering to define what that standard might be. It is even more surprising that several universities could be ranked at above the world standard (or even well above the world standard) without any reliable objective evidence to support that ranking except for the “expert and informed view” of the individuals making the ranking.