Wednesday, 9 July 2014

I see that the egregious business rules 131 and 132 made it through the final specifications

Thursday, 12 June 2014

Monday, 12 May 2014

Budget Blues

Anyone like to lay odds on whether ERA will survive tomorrows budget?

Seems like we could achieve the same end much more cheaply with a little bibliometric analysis. 

Bit then we wouldn't need this blog!  I predict that the survival of the ERA Whisperer blog will be seen as too important, and ERA will survive.

Wednesday, 19 March 2014

ERA conference list

As of this afternoon a little under 1000 submissions seem to have been made through JACCI regarding conferences (I'm assuming that the recommendation ID is a counter).

So 40 universities, that's what, 25 each - maybe even the same 25.  Either we are all planning to get very busy in the next few days, or we are going to making very heavy use of 'other'.

Similarly there have been around 400 comments made on the 2000 or conferences currently in the list.

Let just say that this conference thingy might be a debacle! 

Maybe the ARC wants to get rid of them too?

Friday, 14 March 2014

Predatory Journals

The issue of predatory journals on the ERA list was raised with me recently.  I haven't had time to check how many are on the list but it sounds entirely plausible that there is overlap between Beall's list (http://scholarlyoa.com/publishers/) and the ERA list.

Does it matter?  I'm not sure although it could be a juicy tabloid story if spun the right way.

"PYNE'S SUPER EXPENSIVE AUS  RESEARCH ASSESSMENT EXERCISE WANK RELIES ON DISCREDITED DODGY FRAUDULENT JOURNALS"

(I did enjoy adding Pyne to my headline :>) )

Impact in the citation disciplines is negligible unless the  journals are indexed by the citation supplier - and if they are, they really predatory.  Might be more of a problem in the peer-review disciplines.  Hopefully not too many get nominated for peer review.

Could the ARC to a bulk remove of the Beall's list journals or maybe flag them for the 'peak bodies' to pay special attention to phase 2 of the consultation.

btw if anyone knows of a Beall's list version with ISSN's that could help us all out.

Monday, 3 March 2014

Conference Submission Process

Those of you who subscribe to the Google group will have seen that the ARC has clarified the question as to whether a conference can be submitted if it isn't on the ERA list.  Apparently there will be an 'Other' option - so everyone who thought we could lose all those manky conference publications by collectively not submitting the conferences will need to rethink their strategy.  I would think it will be much preferable to supply the conferences with the correct identity rather than as an amorphous lump of bad and sometimes good conferences.

I've finally had a bit of a squiz at the JACCI website for new conference submission.  You can expect to need to supply
  • Conference name
  • Acronym
  • Title of last Proceedings
  • ISBN of last Proceedings
  • Start and End dates of last conference
  • Venue of last conference
  • Website of last conference
Interestingly there doesn't seem to any mention of a full paper refereeing process although clearly we won't be able to submit conferences papers without this.

Anyone who has had the dubious pleasure of verifying/auditing conference publications for the HERDC or previous ERA processes will immediately realize that unless we have documentation to hand this is going to be a time consuming process.  And we all get to do it separately and severally since we don't know which conferences others have already done the work for.

It really is a shame that ERA has retained conference papers across the board rather than accepting them only in those disciplines where they meaning and respect.


Monday, 17 February 2014

Crowd Sourcing - but whats in it for us?

Having all dutifully submitted our responses to the draft specifications now it's time to knuckle down and work on the journal and conference lists.

The journal list fair enough - it's pretty complete and we will want to tweak the FOR codes.

But why would we add conferences to the conference list; other than in the 08 codes?  This is our chance not to do anything and get rid of all those pesky conferences we'd really prefer not to deal with isn't it?

If no one populated the list - which is pretty incomplete in most codes we could get rid of lots and lots of crappy conference papers - and send a strong message to the ARC about the pointless inclusion of these outputs (08 and maybe 09 aside).

Seriously I'd like to hear what others think about this.  If you would prefer to comment anonymously you can email era.whisperer[@]gmail.com and I'll post the comment to the blog.

Saturday, 15 February 2014

Opening Black Boxes

Those of us that have been lucky enough to see presentations from Tim Cahill will have been quietly pleased to hear that the ARC has plans to improve the feedback institutions get from the peer review process. Apparently we can expect to see some nifty metrics showing how each reviewer characterized our peer review selections.

This is a great start towards opening the impenetrable black boxes that are peer review disciplines. 

Realistically it may be the best we can get from this part of the process.  It's never been clear in any case that the panels give much weight to the opinions of the peer reviewers - why would they - they can and do read the peer review material themselves and no doubt form their own opinions.  Regardless we will start to understand how the material we nominate is regarded.

Now what do the panels think? 

And do context statements make any difference?  I'm sure we have all heard wildly varying views on that.

Oh and how many minutes did they spend discussing your institution.  That can't be secret can it?


Monday, 10 February 2014

To do list - Feb

  1. Be seen to have a thorough internal consultation process about the draft ERA docs - maybe a quick chat with the DVC(R) might not cut it.
  2. Write an absolutely compelling response for the ARC so they can ignore it.
  3. Waste a week or so finding and submitting 4th rate journals to the ERA list so the uncited papers published in them can bite you in the arse later.
  4. Explain to academics again why submitting papers in the wrong discipline might not be a good idea.
  5. Do 4 again.
  6. Try and work out which adjuncts need to be official by the unknown census date
  7. Try to work out if the university actually has a process for appointing adjuncts or if the pixies do it.
  8. Spam academics with requests to add relevant FOR codes to the journals in the ARC list.
  9. Do 4 again.

Monday, 3 February 2014

Lets fix it before it breaks something!

I absolutely believe it is appropriate to assess the quality of publicly funded research.  In fact, I think there is a case that there should be more assessment not less.  (Do I need to disclose here that if nothing else it helps keep me employed?)

When we measure things we change the way they behave - well that is at least true when we measure people and they know we are measuring them - and academics are exquisitely sensitive and responsive to measurement.  Even universities can sometimes be fairly responsive if they believe money or reputation might be at stake.  If we are going to measure something we value, we probably don't want to use a destructive measuring system.  With this mind there is one aspect of research assessment and ERA in particular that does worry me a great deal.

ERA, unlike some other research assessment systems, tries to measure across the whole quantum of research.  Institutions submit all their work from all their researchers in all their disciplines.  This does allow the ARC to do some very neat and robust citation analysis. 

So whats the down side?

In an ERA assessment you definitely want to minimize the number of low citing or otherwise unexciting research publications but you have to report them all.  So it's better not to publish them at all! Right?  Well no, categorically NO! Science relies on the publication of all research.  If we aren't publishing all our work (or at least trying to) we aren't doing science.

Its hard work and fairly unrewarding work to get negative results published anyway but it's necessary to limit the number of times scarce public funding gets spent going down the same dead end.  And no, you generally can't see the dead ends coming.

I think ERA needs to develop citation metrics that have the robustness and discipline sensitivity of the current system but are applicable where only a subset of publications is assessed.  Ideally it would be left to the institution to decide how many publications (beyond a minimum of 50) should be submitted for assessment.  If an institution does not want to be assessed in a particular discipline then allow a 'Not Submitted' result.

Lets fix it before it breaks something!

PS Michelle Duryea has a piece in Research Whisperer on ERA game playing including a nice explanation of some of the rule intricacies.

Thursday, 30 January 2014

Hidden Rule Changes



The ARC has recently released consultation documents for the 2015 ERA submission. The process is largely unchanged from the 2012 rules. However sharp eyed readers will have noticed business rules 131 and 132. These rules place a limit on the percentage of publication outputs that can be recoded from the journal discipline codes assigned by the ARC. The recoding percentage is capped at 40%. This presumably has been introduced to limit gaming by institutions trying to hide supposedly poor quality or low citing publications. It could potentially have major impacts for smaller institutions doing multidisciplinary research who may struggle to research assessment thresholds. Its hard to believe this important change was accidentally omitted from the change summary in the submission guidelines – perhaps the rule changes were considered and discarded but accidentally left in the business rules document.

I suspect most Australian universities did their very best to present the best possible submissions in 2010 and 2012 but its not clear that gaming significantly changed outcomes. Its interesting to look at the 2012 outcomes in the 15xx codes. A number of universities submitted assessable numbers of publications in 1599, the 'not else where classified' subdivision of the Commerce, Management, Tourism and Services (CMTS) group. Of the 15xx codes submitted by these institutions, who were presumably hiding publications in 1599, only one of the defined 15xx codes scored a 4 ('above world standard'), and handful scored 3 ('world standard'). The only success from this strategy was probably to divert and consolidate some poor outcomes into a single code rather than advertising apparent shortcomings in the economically important CMTS disciplines.

ERA Whisperer



This blog is intended to provide an ongoing commentary on the Australian research assessment process – ERA. I hope to make useful suggestions about the ERA process and occasionally make comment about odious or laudable aspects of research assessment.