Introduction: The content of this paper goes back to reflections presented in a former paper at ECER 2004 under the sub-title 'Observation of productivity of educational research by bibliometric tools'. ( http://www.leeds.ac.uk/educol/documents/00003714.htm ) This paper will analyse existing services, instruments and new ideas for scientometric and evaluative purposes in the field of educational research.

The bibliometric instruments per se are the citation indexes of the Institute for Scientific Information: the SSI and the SSCI. They still have a very high reputation for the evaluation of research publications. As first step I would like to present some data giving evidence that the Social Science Citation Index does not at all represent European educational research.

In the second part I will look at innovative approaches and projects aiming at better visibility and comparability of publications, whereas the different target groups and aims connected with ambitions to advance the visibility of educational research have to be considered.

Derived from this assemblage of approaches the third step will be the presentation of a general concept of a multi-attributive approach to the evaluative analysis of publications.

In the focus of the final part will be a tangible project proposal in the context of EERA which shall be submitted to the EU 7^th FP in a few months.

There is a very strong resentment against the Institute for Scientific Information (ISI) product "SSCI" among European social and humanities scientists – and – moreover – among the publishers of their publications. Thomson Scientific people keep on reporting that a growing number of non US journals and non-English language journals are covered in their indexes. This may be true for the hard sciences (SMT) which have submitted long ago to the dominance of the SSI, and very often publish their work results according to the guidelines of ISI.

But if you look at the European representation of educational sciences in the Thomson Indexes you get a picture which is very much in line with the most negative prejudice. (In the Science Citation Index there are only a few journals concerning medical and science instruction.) Educational Science and Research is part of the Social Science Citation Index. I want to present just a few facts based on the most up-to-date data.

1. First information basis is the Journal Citation Report (JCR), Social Sciences Edition with 1.700 journals at all, i.e. for all disciplines and all countries.

2. Secondly, on the basis of a topical search for "educational research" in the SSCI I just want to look at the authors by country (Appendix table 3):

The amount of US authors is less than 50%, is proven by several other search topics and seems to be representative. This percentage is still a US dominance, but – compared to the amount of US journals in the journal selection – it is a reduced rate. This shows that there is one way for European authors to be better represented in the SSCI. This is when they publish in US journals or at least English language journals. But, there is no reason why American educational journals should be more internationally oriented than European ones and the judgement that the SSCI is much too US biased and generally neglects non-US journals and their authors is nevertheless true.

You could also show that typical European topics of educational research – like the international recognition or accreditation of degrees – are not adequately represented. But I just wanted to underline by some accurate examples that the SSCI is imbalanced as far as it concerns the educational sciences outside of the USA.

Non-journal literature was until recently not part of the Web of Science, the web portal which features the ISI citation indexes. Very soon there will be a new service called "Web Citation Index" including documents from institutional repositories (IR) as strictly selected as the journal indexes. (The WCI was not yet online when I prepared for this paper. So I cannot tell about the representation of educational documents in that future index.) Even though, at the moment, there is still a weak quotation of online documents in educational publications the move to digital documents shows that Thomson ISI realized the urgency of this amendment, if they want to cope with future challenges.

But, there is very much evidence that for the time being books and book articles are the main gap in the ISI indexes, because they still seem to be very important publication types within the educational field: and this evidence can partly be taken from the analyses of journal article citations. We did it on a non-representative basis in Germany analyzing citations of the two leading German journals for educational research (one you can do using SSCI). And we did it by looking for prominent German educational scientists in the SSCI. The publications of these scientists you will mainly not find in the SSCI journals but as citations. All of these smaller surveys show the significance of books.

(Apology for the German bias in this, but I think it is quite representative for others, too).

Overall Findings:

1. Non-US journals and especially non-English language journals are a clear minority in the SSCI.
2. Conclusion: If European authors want to be represented in the SSCI they should preferably publish in US journals or at least English language journals!
3. The SSCI does not at all cover the literature which is quoted in its non-US journals. Most of the citations go to publications and authors which are not (adequately) represented in the SSCI and which have consequently no impact factor!

There seem to be different strategies to cope with this situation:

1. Aiming for better representation in SSCI
2. To create a new instrument of a classified or rated presentation of scientific publications

The structural analysis of tremendous misrepresentation of European educational research in SSCI shows that the first strategy will be a way paved with obstacles. European journals will have to adapt the ISI standards of international relevance. And the European scientific community has to draw a lot more attention to their publications than it does now: The European journals will only be accepted for the SSCI if European journal articles are more often cited in US journals. And this makes all efforts very incalculable and it is a dubious aim anyway.

I would suggest the third solution as preferable, even though it will not be easy to make it work. Later I will outline an alternative way to select and categorize publications which are relevant for the educational scholarship.

For several years different approaches for new methods of selective or measuring processes have been developed and tested. Behind these efforts different factions and perspectives are detectable.

1. Libraries

Libraries wish to measure usage of their resources and to identify user needs for prospective decision making: EU projects like COUNTER (Counting online usage of Networked Electronic Resources) are to be named in this context. User needs and topics are researched by biblioming projects. This is very closely connected to the second approach I want to mention - coming from the

Driven by the idea that more and more relevant literature will be available digitally these concepts are based on the assessment of links, citations or on lock-file analyses of internet documents. A short term for concepts of that sort is webometrics. Many research projects funded by governments and other agencies, like DRIVER ( http://www.driver-repository.eu/ ).

The broad digitisation planned by google (google science, google books) and others will soon enhance the usefulness of approaches based on online availability of documents.

Scientific associations – like the EERA – are interested in enforcing the role of peer-review in the field. In this context I want to introduce one European project a little bit in detail, as this project of the European Science Foundation and the European Union also addresses the educational sciences. The project is entitled ERIH, European Reference Index for the Humanities. It is part of a bigger EU programme called HERA Humanities in the European Research Area carried out since March 2005 by 14 national research funding organizations and the European Science Foundation with the objective of firmly establishing the humanities in the European Research Area and in the 6/7th Framework Programmes.

It is interesting that Pedagogical and Educational Research is one of the 15 disciplines covered by ERIH.

ERIH is based on a peer review process. 15 expert groups delegated by national scientific associations compiled a selection of highly and internationally relevant journals in their discipline. The journals are selected by strong criteria of international relevance into three categories(1).

The ERIH journal lists are not yet published, but insider information says that for educational research there are about 400 journals in the lists. I think these are too many according to the strong criteria of international recognition they made up, but it will be far too few to represent all worldwide high level research in education. And I fear this project does not address the language barrier sufficiently. Publication of the lists is scheduled for October 2006.

The big disadvantage of this (interesting) project is that the crucial parts of the development of the European Reference Index for the Humanities are not yet detailed as far as I can see: e.g. it is also planned to add books (monographs and collections of papers) to the index, but there is no word to find about the very problematic process of a peer-reviewed compilation of books.

There is no information about the gathering of single publication entries for the planned reference database. And no precise concept of announced ‘citation calculations’ is available. I am sure that analyzing the references of the compiled journals and books (citation analyses) will show up mostly publications which are not in the index, because the selection criteria are very strong.

Usage of automated revealing and assessment of attributes signalling relevance based on metadata and fulltext analyses of documents. This relatively new line of information research created only intermediary results: Spanish publication on educational research, approaches based on a set of attributes which are derived from expert rankings. I will explain this approach in a little more detail in the following part.

The idea is to use a combination of the above mentioned new ways of assessment of research relevance of publications as an alternative to the citation analysis process. The combination of these ways and metrics provides advantages:

There are two anchors where you can attach an attribute of relevance. One is to measure or rate well defined characteristics of the specific publication itself, the other one is to measure characteristics of the editorial process, i.e. to base the measurement on the affiliation of this specific publication to a certain journal, publisher or editor.

A first group of document characteristics derive from more traditional descriptions of literature often available in bibliographic databases: content, target group, purpose. The data for these indicators should be generated automatically, i.e. ICT programmes run across metadata and detect valuable information. Standardization and broad usage (!) of formats and of metadata will significantly improve the outcome of these procedures. These programmes have to be developed for each scientific discipline specifically, because content or methods will probably differ. In this context of standardized meta-information it is also important to name translations into the English language. Only if titles, keywords and abstracts are provided also in English machine based selection procedures will be successful.

In a German project we are now applying for we want to try out if special metadata for (social) science publications could be generated which characterize the status of the document in the scientific process: research report, state-of-the-art report, summary, commentary, handbook article, theory discussion. Maybe there is useful information in such standardized characterizations for selective purposes.

As already mentioned many projects now focus on the new facilities provided by digitisation. Digital documents and libraries (Open access movement) create new ways and strategies of selection procedures and quality analysis. Relevance is measured or assessed by log file analyses preferably supported by link servers. Attributes for relevance are numbers of (once again) citations, usage/access/downloads and linkings (a popular example is Google). The more literature will be available online in the future the more we will be able to represent these assessments in our databases, e.g. for sorting search results by relevance ranking.

The second line of rating scientific publications is based on the reliability of the source respectively on the evaluation of the editorial process. Not only can journals be peer-reviewed or not, so can all kind of serial editions of books and papers. In the near future, repositories of online documents will be peer-reviewed or a repository will be singled out by the reputation of the institution that stands behind it.

But peer review or peer questioning can also be applied as a judgement of rankings which have been the result of the above mentioned methods. They can also be applied to judge the need for refreshment of older rankings.

As I already indicated we will try to prove most of these methodological approaches on national level, if we get approval of our proposal. But there is also a project plan on European level which I want to present finally.

Some of you may know that the EERA Council wish to promote a proposal for an EU project which aims at better visibility of European educational research. There has already been one attempt through a proposal in the 6^th framework programme which was unsuccessful. Now we are planning a new go within the 7^th FP. The draft proposal will be discussed here at the conference and I would like to outline the main aspects of this project draft.

The consortium will consist of universities, bibliographic database providers, and journal publishers. At the moment the DIPF and BEI are addressed to be involved from network 12. But there is a need and a will to extend the number of collaborators. At the moment only German, British and Dutch, perhaps Finish partners are in the boat. There is an invited meeting here at the conference on Friday to discuss this project and to extend participation.

Aim: The project will determine and enhance the framework for relevance assessment for research documents in the field of educational research (as an example for particularly socially- and politically-embedded research fields within the broader area of the humanities and social sciences), based on formal mechanisms including references(2), citation linking and co-occurrence of items in open access and non-open access repositories. In order to optimize the possibilities of webometrics a cooperation with Google Scholar and Google Book would be a central part of the technological infrastructure of the project. The aspect of the unique quality of specific terminology embedded in the individual country/language context will be approached by different methods and tools: existing multilingual thesauri in the field, semantic analysis and comparison, ontologies, and search engine technology.

The project will constitute 3 work packages:

1. a feasibility study including analysis of the relevance assessment criteria, prevalence of these criteria,
2. development of a prototype European relevance assessment database including Open Access materials and publishers’ metadata
3. development of a viable business model for maintenance and financial sustainability of such a product (an essential work package of every EU project, now).

The feasibility study will aim at the following results:

The prototype database will be based on journal article contributions of the involved publishers and will moreover assemble documents from many sources resp. databases as Google Scholar, data from PERINE partners, from OAI-compatible repositories and data of other cooperating projects. On the basis of the feasibility study metadata and structuring elements should be added to allow new assessment methods and criteria. Multilingual functionalities should be installed.

This central part of the project, the establishment of an integrated bibliographic database, stands in close vicinity of another proposal we will discuss at this conference. In the succession of PERINE we plan to send in a new proposal to the EU, a proposal which will envisage also a bibliographic database for European Educational Research Publications. As I already mentioned PERINE partners are supposed to be involved in this scientometric project. This means we need to tune both proposals as good as possible.

The development of a Business Model to run such a database as competitor to the SSCI shall be a joint venture of universities, learned societies, publishers, private corporations.

The plan is to send in the proposal for the first call of the 7^th FP of the EU.

As a final remark I want to mention the following: All efforts of developing improved methods to make European educational research internationally better visible are quite limited, if the predominant practice of scientific publication stays as it is. Every scholar and every scientific publisher who wants to receive supranational perception has to adapt his ways of publication to certain standards. And indeed: scientists in the disciplines of Psychology and Medicine are probably much more aware and can take advantage of foreign research than educational scholars do at the moment.

In order to reach a better international awareness of European educational research publications it seems to be essential for the educational publication process to

3. It would be arguable if it does make sense to force educational researchers to publish a complex research report in a series of six or seven journal articles instead of putting it comprehensively in a book. But this is what many researchers in the natural sciences do nowadays

Subdisciplines:

Table 1

Table 2

Table 3

Result: European authors are much better represented in the SSCI when they publish in US journals or at least English language journals.