You have javascript disabled in your browser. We recommend turning it on for a better experience on this site.


Researching the bias in journal ratings

Researching the bias in journal ratings

A trusted system used to measure the quality of academic research produced at many of the world's leading universities is highly problematic, a study has found.

A team of organisational psychologists at the universities of Exeter, Leeds and Loughborough looked at how 168 business and management academics rank the specialist journals in which they publish research and found widespread bias is inherent in the system.

The study, 'Experts judgements of management journal quality: the role of identity concerns in a contested domain', raises questions about the role of journal lists as a measure of academic excellence.

Based on a well-established body of knowledge in social psychology known as "social identity theory", the team predicted and found that subject experts' evaluations of research in business and management are swayed heavily by personal biases.

Dr Kim Peters of the University of Exeter's School of Psychology, who led the study, said: "Academics rated the quality of journals more favourably when they had published more papers in that journal, when they were a member of the journal's board, when the journal reflected their academic interests and their geographical location. We found a bias from start to finish."

Professor Gerard Hodgkinson of Leeds University Business School, who co-authored the study. said: "Our findings turned out entirely as expected on the basis of our theoretical predictions.  For instance, a qualitatively-oriented scholar with a background in sociology will tend to rate journals that favour sociologically-oriented and qualitative studies more highly than a colleague with a background in psychology and a leaning toward quantitative methods.

"There is a rational explanation for the pattern of findings we observed - such judgements serve to bolster the experts' sense of selfhood and belonging.  Personal and social identity concerns are the primary driving force behind how we evaluate research quality, but our judgements are also constrained to some extent by the objective reality of the context in which they occur.  

"So, for example we found widespread agreement among our participants that North American journals are generally of a higher quality than UK/European ones.  Nevertheless, even here there was a tendency toward a bias of geography, with North American scholars tending to rate UK/European journals more harshly than UK/European scholars."  

Although the study focused on how business and management journals are rated, the authors stress that the approach they used means the results could be extrapolated across other academic disciplines and fields of study.  

Professor Kevin Daniels of Loughborough University said: "The study raises serious questions concerning the use of journal rankings in judging some aspects of research performance.

"At the very least, compilers of such lists may need to make their methods as transparent and as rigorous as might be expected in a peer-reviewed article, so that the lists themselves can be judged. However, a serious rethink may be needed on alternative or supplementary approaches that are rigorous, fair and also seen to be rigorous and fair."

For further information:

Please contact the University of Leeds Press Office on +44 (0)113 343 4031 or email pressoffice@leeds.ac.uk

Notes to Editors

The study 'Experts judgements of management journal quality: the role of identity concerns in a contested domain' by Kim .O Peters, Kevin Daniels, Gerard P. Hodgkinson and S. Alexander Haslam' was presented at the prestigious 2009 Academy of Management Meeting, Chicago, IL in August.

The 2008 Research Assessment Exercise showed the University of Leeds to be the UK's eighth biggest research powerhouse. The University is one of the largest higher education institutions in the UK and a member of the Russell Group of research-intensive universities. The University's vision is to secure a place among the world's top 50 by 2015.

Leeds University Business School is among the leading UK university-based business schools with a high reputation for research and a history of successful partnerships with industry and commerce at local, national and international levels.

Key facts include:

  • Ranked 11th in the UK's top MBA programmes (Financial Times Global MBA 2009)
  • Ranked 3rd in the world for value for money (Financial Times Global MBA 2009)
  • Leeds University Business School Ranked in UK Top Ten for Research (RAE 2008)
  • International Business at LUBS ranked 9th in the world (Financial Times Global MBA 2009)
  • EQUIS and AMBA accredited
  • CIPD accredited and ESRC recognised

Members of the Faculty are at the forefront of major developments in basic and applied research across the fields of Business, Management, Accounting, Finance and Economics. Senior staff have held leadership positions and committee membership and advised major policy-making bodies and learned societies. These include the Academy of International Business, the Academy of Management, the Economic and Social Research Council's Training and Development and Research Grants Boards and the British Academy of Management.

Ranked 57th in the world's top MBA programmes (Financial Times Global MBA 2009)

http://www.leeds.ac.uk/

Loughborough University

Loughborough is one of the country's leading universities, with an international reputation for research that matters, excellence in teaching, strong links with industry, and unrivalled achievement in sport and its underpinning academic disciplines.

It was awarded the coveted Sunday Times University of the Year 2008-09 title, and is consistently ranked in the top twenty of UK universities in national newspaper league tables. In the 2009 National Student Survey, Loughborough was voted one of the top five universities in the UK, and has topped the Times Higher Education league for the UK's Best Student Experience every year since the poll's inception in 2006.

In recognition of its contribution to the sector, the University has been awarded six Queen's Anniversary Prizes. Its Business School is recognised internationally by EQUIS accreditation for the excellence of its activities and is among just 5% of business schools in the world to have achieved this prestigious accreditation.

The Business School is frequently regarded as one of the top-ten business schools in the UK; its teaching is consistently rated as 'excellent' and its programmes are informed by world-class research, with 60% of research rated as 'world leading' or 'internationally excellent' by the 2008 RAE. The 2008 National Students Survey listed us at 4th in the UK for Management Studies, and joint 5th for Accounting. http://www.lboro.ac.uk/

Back to the top