Using metrics responsibly
In 2012 the Declaration on Research Assessment (DORA) illuminated issues related to the excessive attention paid to journal metrics in research metrics use. It suggested ways in which all stakeholders involved in evaluating academic research and scholarship could mitigate errors in perception and reporting of research metrics. The principles within the declaration set out the ways in which institutions can ensure they are using research metrics responsibly.
It has attracted over 13,000 individual and 880 organizational signatories since then. Researchers, funders, universities and research institutes, publishers and metrics providers have all committed – at a minimum – not to “use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions”.
UHI has now developed a policy for the responsible use of research metrics based on the principles of DORA, and other similar best practice policy papers. Our policy aligns so well with the DORA principles the institution took the decision in 2019 to show it's committment by becoming a supporting signatory.
Internal measures that all researchers should aspire to follow when compiling research reports:
1- Academic culture and behavioural change to develop credible alternatives to the inappropriate uses of metrics in research assessment.
2 – Consider all types of research outputs (including datasets, portfolio work and software) in addition to research publications, and consider a broad range of impact measures in qualitative indicators of research impact, such as influence on policy and practice.
3 – Transparent approach; consider the composition of the committees that perform evaluations and whether a more diverse team would assist in achieving fairer outcomes.
4 – Researcher actions; Challenge research assessment practices that rely inappropriately on Journal Impact Factors and promote and teach best practice that focuses on the value and influence of specific research outputs
5 – Continue to develop a best practice approach with regular review and reflection
External measures that DORA are implementing to influence stakeholders:
Publishers: DORA aim to encourage greatly reduced emphasis on the journal impact factor as a promotional tool, ideally by ceasing to promote the impact factor or by presenting the metric in the context of a variety of journal-based metrics (e.g. 5-year impact factor, EigenFactor, SCImago, review and publication times) that provide a richer view of journal performance. (DORA Recommendations no. 6)
Metrics providers: Encouragement to be open and transparent by providing data and methods used to calculate all metrics (DORA Recommendation no. 11).
Ultimately, given the international nature of research and scholarship, effective culture change will only take root if reform of evaluation methods occurs on a global scale. Establishing a globally supportive environment for effective reform of research evaluation will take time.
Particular considerations for social/applied subjects
For social science and applied subjects the collection or even identification of appropriate metrics can be very difficult. This is one area where qualitative narrative has increased importance but there are other ways to 'measure' through non numeric means. The below academic paper written on this subject has numerous interesting ideas and comments:
[Journal of Documentation, vol. 71, no. 4 (July 2015) 817-833];
http://dx.doi.org/10.1108/JD-02-2015-0028