I absolutely believe it is appropriate to assess the quality of publicly funded research. In fact, I think there is a case that there should be more assessment not less. (Do I need to disclose here that if nothing else it helps keep me employed?)
When we measure things we change the way they behave - well that is at least true when we measure people and they know we are measuring them - and academics are exquisitely sensitive and responsive to measurement. Even universities can sometimes be fairly responsive if they believe money or reputation might be at stake. If we are going to measure something we value, we probably don't want to use a destructive measuring system. With this mind there is one aspect of research assessment and ERA in particular that does worry me a great deal.
ERA, unlike some other research assessment systems, tries to measure across the whole quantum of research. Institutions submit all their work from all their researchers in all their disciplines. This does allow the ARC to do some very neat and robust citation analysis.
So whats the down side?
In an ERA assessment you definitely want to minimize the number of low citing or otherwise unexciting research publications but you have to report them all. So it's better not to publish them at all! Right? Well no, categorically NO! Science relies on the publication of all research. If we aren't publishing all our work (or at least trying to) we aren't doing science.
Its hard work and fairly unrewarding work to get negative results published anyway but it's necessary to limit the number of times scarce public funding gets spent going down the same dead end. And no, you generally can't see the dead ends coming.
I think ERA needs to develop citation metrics that have the robustness and discipline sensitivity of the current system but are applicable where only a subset of publications is assessed. Ideally it would be left to the institution to decide how many publications (beyond a minimum of 50) should be submitted for assessment. If an institution does not want to be assessed in a particular discipline then allow a 'Not Submitted' result.
Lets fix it before it breaks something!
PS Michelle Duryea has a piece in Research Whisperer on ERA game playing including a nice explanation of some of the rule intricacies.
No comments:
Post a Comment