Apparently we should check the website at 3pm (28/2)
http://arc.gov.au/era/ERA_2015_reference_periods.htm
Friday, 28 February 2014
Monday, 17 February 2014
Crowd Sourcing - but whats in it for us?
Having all dutifully submitted our responses to the draft specifications now it's time to knuckle down and work on the journal and conference lists.
The journal list fair enough - it's pretty complete and we will want to tweak the FOR codes.
But why would we add conferences to the conference list; other than in the 08 codes? This is our chance not to do anything and get rid of all those pesky conferences we'd really prefer not to deal with isn't it?
If no one populated the list - which is pretty incomplete in most codes we could get rid of lots and lots of crappy conference papers - and send a strong message to the ARC about the pointless inclusion of these outputs (08 and maybe 09 aside).
Seriously I'd like to hear what others think about this. If you would prefer to comment anonymously you can email era.whisperer[@]gmail.com and I'll post the comment to the blog.
The journal list fair enough - it's pretty complete and we will want to tweak the FOR codes.
But why would we add conferences to the conference list; other than in the 08 codes? This is our chance not to do anything and get rid of all those pesky conferences we'd really prefer not to deal with isn't it?
If no one populated the list - which is pretty incomplete in most codes we could get rid of lots and lots of crappy conference papers - and send a strong message to the ARC about the pointless inclusion of these outputs (08 and maybe 09 aside).
Seriously I'd like to hear what others think about this. If you would prefer to comment anonymously you can email era.whisperer[@]gmail.com and I'll post the comment to the blog.
Saturday, 15 February 2014
Opening Black Boxes
Those of us that have been lucky enough to see presentations from Tim Cahill will have been quietly pleased to hear that the ARC has plans to improve the feedback institutions get from the peer review process. Apparently we can expect to see some nifty metrics showing how each reviewer characterized our peer review selections.
This is a great start towards opening the impenetrable black boxes that are peer review disciplines.
Realistically it may be the best we can get from this part of the process. It's never been clear in any case that the panels give much weight to the opinions of the peer reviewers - why would they - they can and do read the peer review material themselves and no doubt form their own opinions. Regardless we will start to understand how the material we nominate is regarded.
Now what do the panels think?
And do context statements make any difference? I'm sure we have all heard wildly varying views on that.
Oh and how many minutes did they spend discussing your institution. That can't be secret can it?
This is a great start towards opening the impenetrable black boxes that are peer review disciplines.
Realistically it may be the best we can get from this part of the process. It's never been clear in any case that the panels give much weight to the opinions of the peer reviewers - why would they - they can and do read the peer review material themselves and no doubt form their own opinions. Regardless we will start to understand how the material we nominate is regarded.
Now what do the panels think?
And do context statements make any difference? I'm sure we have all heard wildly varying views on that.
Oh and how many minutes did they spend discussing your institution. That can't be secret can it?
Monday, 10 February 2014
To do list - Feb
- Be seen to have a thorough internal consultation process about the draft ERA docs - maybe a quick chat with the DVC(R) might not cut it.
- Write an absolutely compelling response for the ARC so they can ignore it.
- Waste a week or so finding and submitting 4th rate journals to the ERA list so the uncited papers published in them can bite you in the arse later.
- Explain to academics again why submitting papers in the wrong discipline might not be a good idea.
- Do 4 again.
- Try and work out which adjuncts need to be official by the unknown census date
- Try to work out if the university actually has a process for appointing adjuncts or if the pixies do it.
- Spam academics with requests to add relevant FOR codes to the journals in the ARC list.
- Do 4 again.
Monday, 3 February 2014
Lets fix it before it breaks something!
I absolutely believe it is appropriate to assess the quality of publicly funded research. In fact, I think there is a case that there should be more assessment not less. (Do I need to disclose here that if nothing else it helps keep me employed?)
When we measure things we change the way they behave - well that is at least true when we measure people and they know we are measuring them - and academics are exquisitely sensitive and responsive to measurement. Even universities can sometimes be fairly responsive if they believe money or reputation might be at stake. If we are going to measure something we value, we probably don't want to use a destructive measuring system. With this mind there is one aspect of research assessment and ERA in particular that does worry me a great deal.
ERA, unlike some other research assessment systems, tries to measure across the whole quantum of research. Institutions submit all their work from all their researchers in all their disciplines. This does allow the ARC to do some very neat and robust citation analysis.
So whats the down side?
In an ERA assessment you definitely want to minimize the number of low citing or otherwise unexciting research publications but you have to report them all. So it's better not to publish them at all! Right? Well no, categorically NO! Science relies on the publication of all research. If we aren't publishing all our work (or at least trying to) we aren't doing science.
Its hard work and fairly unrewarding work to get negative results published anyway but it's necessary to limit the number of times scarce public funding gets spent going down the same dead end. And no, you generally can't see the dead ends coming.
I think ERA needs to develop citation metrics that have the robustness and discipline sensitivity of the current system but are applicable where only a subset of publications is assessed. Ideally it would be left to the institution to decide how many publications (beyond a minimum of 50) should be submitted for assessment. If an institution does not want to be assessed in a particular discipline then allow a 'Not Submitted' result.
Lets fix it before it breaks something!
PS Michelle Duryea has a piece in Research Whisperer on ERA game playing including a nice explanation of some of the rule intricacies.
When we measure things we change the way they behave - well that is at least true when we measure people and they know we are measuring them - and academics are exquisitely sensitive and responsive to measurement. Even universities can sometimes be fairly responsive if they believe money or reputation might be at stake. If we are going to measure something we value, we probably don't want to use a destructive measuring system. With this mind there is one aspect of research assessment and ERA in particular that does worry me a great deal.
ERA, unlike some other research assessment systems, tries to measure across the whole quantum of research. Institutions submit all their work from all their researchers in all their disciplines. This does allow the ARC to do some very neat and robust citation analysis.
So whats the down side?
In an ERA assessment you definitely want to minimize the number of low citing or otherwise unexciting research publications but you have to report them all. So it's better not to publish them at all! Right? Well no, categorically NO! Science relies on the publication of all research. If we aren't publishing all our work (or at least trying to) we aren't doing science.
Its hard work and fairly unrewarding work to get negative results published anyway but it's necessary to limit the number of times scarce public funding gets spent going down the same dead end. And no, you generally can't see the dead ends coming.
I think ERA needs to develop citation metrics that have the robustness and discipline sensitivity of the current system but are applicable where only a subset of publications is assessed. Ideally it would be left to the institution to decide how many publications (beyond a minimum of 50) should be submitted for assessment. If an institution does not want to be assessed in a particular discipline then allow a 'Not Submitted' result.
Lets fix it before it breaks something!
PS Michelle Duryea has a piece in Research Whisperer on ERA game playing including a nice explanation of some of the rule intricacies.
Subscribe to:
Posts (Atom)