Education-line Home Page

Achievement or performance: observation of productivity of educational research by bibliometric tools. A state-of-the-art-report

Alexander Botte
Deutsches Institut für Internationale Pädagogische Forschung (DIPF)

Paper presented at the European Conference on Educational Research, University of Crete, 22-25 September 2004

Preamble (before starting): In the subtitle of my paper presentation it says: ‘A state-of-the-art-report’. I’d like to explain that it is not possible to present a complete state-of-the-art of bibliometric projects and tools within this presentation. The aim of this presentation is to give reason for a proposal of a bibliometric project within the EERA context. And this intended project is the frame in which I want to include my state-of-the-art.

Evaluation by bibliometric measurement is not a very popular issue. Scholars normally do not feel good when they are put on achievement scales. Bibliometric tools are definitely not undisputed, the overall majority of comments on bibliometrics is very critical in general and in detail, even if the authors of these critical comments are very often insiders who use these methods themselves. A brilliant exponent and detractor of bibliometrics at the same time is Peter Weingart1.

However, the truth is that bibliometrics are still – and more and more - on the agenda and they have future perspectives. And it is worth while to look at bibliometrics in the EERA context, i.e. in an European context.

Let me show you at a glance what I want to address in a short overview today

  1. Research policy and administration promoting bibliometrics
  2. Central elements of bibliometric analyses
  3. Disadvantages of European and educational research publications
  4. Opportunities and advantages of bibliometrics: what could be the fruits of new publication standards?
  5. Activities and projects in the field at European level
  6. Proposal for a new project in EERA context

1 Research policy and administration promoting bibliometrics

  1. When I said bibliometrics are still on the agenda they are put there first by political and administrative bodies who foster the evaluation and ranking of achievement of research institutes or university departments by bibliometric measures; in Finnland financial support by the government seems to be directly connected to the performance in the American Science Citation Index, the most prominent and exclusive bibliometric instrument worldwide. We can say such administrative acts are inadequate but we should also try to discover the concern behind this as a challenge for the scientific community:
  2. There are two reasons to do so: if a certain discipline like educational sciences neglects this challenge it will later have to react to administrative activities. It seems to be wise to try to actively influence the ways and methods of evaluating the productivity of research.
    But it is not only the administrative threat which should move us. Besides this ‘sink or swim’ argument there are also internal reasons for the scientific community to take bibliometric representations and questions of the quality of the publishing process more seriously. Especially in a European and international cooperation perspective we should agree on certain common standards which make our work better visible, understandable and, yes, comparable in a productive way. And this is more or less the ratio behind administrative usage of bibliometric methods.

2 Central elements of bibliometric analysis

The American Institute for Scientific Information (ISI), now part of the Thomson company offers several high prized information products which build the basis for most of all bibliometric approaches worldwide. Through the so called ‘Web of Knowledge’ you can get access to the three core citation indexes: The Science Citation Index, the Social SCI and the Arts and Humanities CI (SCI, SSCI, AHCI). Besides you will find a lot of other tools like the Journal Citation Report (JCR), a rating of scientific journals according to numbers of citations. Some of these services are even free of charge, but the core indexes are very expensive.

(ISI maintains a specially structured file - the Science Indicators Database - of publication and citation data, which includes ALL items indexed by ISI and all the references listed in those items.)

A ‘citation index’ is a bibliographic database on journal articles combined with different analyses based on cited references.

The selection process for journals consists of the evaluation of many criteria such as

  • Basic Journal Publishing Standards: including timeliness of publication, adherence to international editorial conventions, English language bibliographic information (including English article titles, keywords, author abstracts) and cited references.
  • ISI also examines the journal's editorial content,
  • the international diversity of its authors and editors.
  • Citation analysis using ISI data is applied to determine the journal's citation history and/or the citation history of its authors and editors.
  • The main concepts of analysis are based on a set of standardized basic indicators evaluating publications on one side and journals on the other: The impact factor of publications as well as of journals are equally calculated by the number of citations received by a single authors’ articles resp. by all articles of a specific journal.

    Generally, bibliometric analysis is not targeted at evaluating one person as an author, but bigger groups, institutions or even regions and countries. A high level of aggregation is a pre-condition of seriousness.

    Bibliometric analysis can make visible research cooperation, thematic relations, centers of excellence and can - possibly – detect errors of peer evaluation.

    BUT: ‘Bibliometric assessment of research performance is based on one central assumption: scientists who have to say something important (do) publish their findings vigorously in the open, international journal literature’ (This was a quotation of Anthony van Raan, 2003). If this assumption is fulfilled for a certain discipline you ‘only’ need the complete database on all these journals, all their citations and the up-to-date institutional allocation of all the authors.

    If you examine these conditions carefully you will not be surprised that there is no competitor for the Thomson ISI databases and that their prizes are very high.

    3 Disadvantages of European and educational research publications

    If we look at the above mentioned journal selection criteria for citation index publications we will soon realize the severe disadvantaged position in which they put the publications of European and especially educational researchers. The SCI or better the SSCI does not represent European educational research properly! The reasons are obvious and widely known: The specific publication traditions of educational researchers do not meet the ISI standards. Euopean educational researchers, even if very important, do not publish all their relevant work in international, i.e. English language articles. The fact about European educational research publications is:

    1. Even very relevant publications are still published in the native language of the researcher.
    2. Even high quality national journals are not necessarily peer-reviewed; there is only the beginning of a new development of international publication standards under the political influence mentioned above.
    3. Citation traditions in education publications are different. It is not yet common to use quotation as representation of an ongoing interactive research communication.
    4. Scientists like the natural scientists who are used to publish in certain journals referenced in the SCI since years are more easily measured by bibliometric instruments, because you can be sure that relevant research is most probably represented in these journals and that most of their quotations relate to SCI journals. Here, the SCI is somehow a valuable tool, even though the focusing on numbers of articles in referenced journals causes new problems which are also discussed in the literature and which I will not address here, because these are (not yet) the problems of publications in the educational sciences.

    4 Opportunities and advantages of bibliometric methods: possible fruits of bibliometric standards

    Basically all bibliometrics go back to counting numbers of publications and counting indicators of relevance. We can say they aim at rankings and competitive aspects. But you could also look at it in a different perspective and say they aim at a complete survey of high quality research (which is a documentary aspect) and bibliometrics support the easy communication of research reports on an international level (by English as lingua franca).

    In this perspective some advantages come into view:

    1. The potential reach of bibliometric instruments is better visibility and accessibility of valuable research.
    2. A necessary element of bibliometric tools are publication standards which are designed to represent high quality research
    3. As instruments which are based on modern electronic database techniques a desirable combination is possible with the new scientific publishing initiatives in the context of digital libraries (OAI-DOAI and so on).

    And a perspective for fruitful products opens:

    i. Reliable catalogue of sources (journals, series, archives, editors) for research relevant documents

    ii. Monitor of the development of intensity of research for specific research areas in time (thematic mapping, rate of interdisciplinary research)

    And if bibliometric indicators are sophisticated enough:

    iii. Statistical data on institutes, regions, research cooperation and rate of international research, funding structures, centers of excellence for certain research areas

    iv. Deeper analysis of networks and biographies (researcher careers) is possible.

    The practical outcome of these data collections and instruments for EERA and national associations could be a sound support for

    1. all reports and descriptions of the achievement and development of European educational research,
    2. for planning future focal points of research,
    3. detecting connections between up to now isolated research fields and so on.

    5 Activities and projects in the field on European level

    If we want to catch hold of such an instrument for the scientific infrastructure and look at activities and projects on a European level we do not find very much preparatory work:

    1. There is more or less only one European center of bibliometric science. It is The Centre for Science and Technology Studies (CWTS) in Leiden (NL) with its director Anthony von Raan who is the most active promoter of bibliometric analysis in Europe. As far as I know the only bibliometric project funded by the EU is conducted by the CWTS and is called ‘Mapping Excellence in Science and Technology across Europe’. This pilot project is going to test the quality and utility of a bibliometric-mapping tool to identify and locate centres of excellence in science fields. This tool is applied in four fields in the life sciences: Bioinformatics; Genetics and Heredity; Immunology; and Neuroscience. This focus on natural sciences is typical for all bibliometric projects. There have only been a few projects cautiously involving social sciences. But as the American SSCI stays background tool the so-called social sciences were restricted to some research fields of ‘psychology’, which are represented relatively good in the SSCI. So, the CWTS is not of much help when it comes to analyze educational sciences, but we can make use of their experience when it comes to methodological problems.
    2. More often than bibliometric projects there are EU-projects on publishing systems with focus on e-documents, e.g. the new study ‘An effective scientific publishing system for European Research (EU study)’ which has been launched by the Commission ‘in order to determine the conditions required for optimum operation of the sector and to assess the extent to which the Commission can help to meet those conditions’. So, this study could be the starting point of a strong involvement of the EU in the scientific publishing system.
    3. The European Science Foundation supports the development of a European Science Citation Index in Humanities, which – surprisingly - includes pedagogy! It is very difficult to find reliable information about this project. There have been initiatives since 2000, but there are no results to be seen. Papers look like they have a very naïve access to the problem, because they seem to underestimate the extent of selective and database work that has to be done for the several fields of humanities on a European level. It looks like they want to do it on the basis of volunteer contributions. (Angel Garcia del Dujo seemed to know more about it, but I could not make contact with him.) Nevertheless, the ESF could also be an institution which would take an application for funding such a project.
    4. I would like to mention the EERA project Design:CRISS containing some of the principle elements of new and modern publication and cooperation strategies for European educational research. The CRISS application was not accepted by the EU, possibly because it was very much focused on serving EERA researchers specially, but we could take up some of it’s goals for a new proposal.
    5. Our project PERINE can provide contributions to the integration of e-documents and the problem of multilingual sources. A follow-up proposal to PERINE did just not pass the second stage of the selection process. This follow-up project was designed to explore the conditions for a European bibliographic database. PERINE partners are now thinking about a new and bigger proposal. The extension of PERINE could also comprise elements or the whole design of the following layout of a bibliometric project as a special workpackage.

    6 Proposal for a bibliometric and publication project in EERA context

    Let me start with principle considerations for the design of a European project by asking: How to overcome dangers and pitfalls of an projected European Educational Science Citation Index?

    The answer: We should not aim at a tool offering robust evaluation scales for European Research output. There is no need to build up a European Citation Index, which would be too expensive, too time-consuming, and too controversial!

    Instead, we should aim at an instrument for better visibility, accessibility, and comparability of European research as a special contribution to the creation of a European research area.

    The project will reach this goal by

    Let me stress that this concept has two crucial focuses which will be the basis of success:

    1. First element is an integrated bibliographic database on European printed and digital research documents. This task faces special challenges caused by the multilingualism of relevant sources (all European languages desirable, the main four or five languages necessary) and the essential cooperation of – until now – separately working information providers.
    2. Second element is the selection of highly relevant research documents according to criteria which describe the editorial and reviewing process on the basis of internationally accepted indicators. This task means the involvement of national scientific/educational research associations, because quality evaluation of relevant publication bodies (commercial publishers, universities, scientific associations and so on) can only take place on national or language level.

    I.e. crucial partners of the consortium must be big providers of digital bibliographic tools (some of which we find in the PERINE project) and confidential representatives of the national educational research communities (most of which we find in the EERA’s national Associations).

    The outcome of the project is not only addressed to researchers, it is also useful for administrators as well as educational practitioners and journalists who need information on the structure and development of European educational research.

    The design of the project should consider the following:

    1. Pragmatic reasons lead to restriction on strongly selected journal articles, series and proceedings: because only periodicals can be generally judged by national associations. Even though we know that in educational sciences books are a very relevant form of publication the following assumption could be pragmatic: nearly every important research result will be reflected in a journal article or a congress paper, even if a book publication is the final outcome! Selecting relevant journals and serial works we should give respect to the international relevance of publication sources.
    2. Instead of evaluation criteria based on citation indexing the database will provide clearly delineated descriptions of the publication standards: like ‘peer reviewed’ ‘blind peer reviewed’ ‘internationally peer reviewed’. These criteria can be applied to journal articles, conference papers, book series and other periodicals.
    3. The instrument must be multilingual : like in the PERINE catalogue we need multilingual keywords. Perspective aim should be as many English translations of title and abstracts as possible.
    4. The movements to e-documents and to open-access-publishing should be especially fostered by the project; we need reliable repositories. All chances to provide access to the documents’ full-text should be used: if full-text-databases are nationally available an integration of their relevant parts should be reached for. The availability of more and more high quality e-documents would stipulate the goals of the project.
    5. For as many research areas as possible the instrument (database) should provide items for thematic and cooperation mappings in order to offer information on national and international focuses of research, cooperation networks, and possible ‘processes of knowledge dissemination’ (van Raan), i.e. communication linkages between different subfields. The challenges for databases which contain items and indicators for such a qualitative analysis are high, and the project has to examine whether it is really possible to carry these achievements into effect.
    6. Citation analysis of a limited number of documents can be part of the project. It can focus on special thematic areas or periods of time in order to establish or prove the reliability of relevance constructions or cooperation structures and so on.
    7. It is a routine part of the work of bibliographic information providers for educational research to spread their screening of literature on certain social and psychological research sources in order to be complete for educational sciences. Nevertheless, on the long run it sounds reasonable to plan this European instrument as a broad Social Science Index, because the research areas are overlapping.
    8. The project will also have to face technical aspects like automatic machinery indexing of documents. Otherwise a representative set of languages and document types will not be reachable..
    9. Publishing guidelines should go with the goals of the project. They should include traceable review procedures and the demand for English titles and abstracts or keywords in order to make the database internationally communicable. It is expected that the existence of a European catalogue of high quality research literature, maintained by the national educational research associations, will support the movement to international standards for quality control.

    Let me sum up crucial elements of such a project are: a consortium of powerful bibliographic information providers and the assistance of possibly all national associations of educational researchers. The core group of the bibliographic information providers could be the partners of PERINE, who mostly maintain bibliographic databases. The national associations of educational researchers should be contacted via EERA.

    The project will be a feasibility study and a developmental study at the same time. It will be a derivative aim of the project to influence the European publishing process of educational research literature and to form an essential tool for the European research area.


    1. One of the latest publications: Peter Weingart: Evaluation of research performance: the danger of numbers. In: Bibliometric Analysis in Science and Research. Applications, Benefits and Limitations. Forschungszentrum Jülich, Conference proceedings of the second conference of the Central Library, Jülich 5-7 November 2003, Jülich 2003, p. 7-19

    This document was added to the Education-line database on 14 October 2004