Developing evidence-informed policy and practice in education
Standards and Effectiveness Unit, Dept. for Education & Employment
Paper presented at the British Educational Research Association Conference, University of Sussex, at Brighton, 2-5 September, 1999.
At last years BERA conference two reviews of educational research had been published in the previous month and much debate about the validity and implications of their findings ensued. This short paper attempts to provide an update on progress and issues in one specific but major area of the findings from the Hillage et al (1998) review, that of the need to improve the relationship between research evidence, policy and practice. This relationship underlies many of the conclusions in the review and much of the debate about educational research that has gone before it. Discussion of the quality and impact of educational research is covered more fully in a number of sources including Furlong (1998); Gray (1998); Hargreaves (1994; 1996;1998); Mortimore and Mortimore (1999) and Rudduck and McIntyre (1998).
Criticisms have been made of the quality of existing research and Hargreavess (1996) often cited comparison to medicine suggests that educational research is of relatively poor quality. Yet, in the press over a year ago (Guardian 24 June 1998), the editor of the BMJ suggested that only 5% of scientific papers in medicine come up to scratch and that research evidence has little or no impact on the patient sitting in front of the doctor. As many of you will have experienced personally, it is far from apparent when seeking advice on medical treatment that reliable and consistent evidence is available and used by most doctors. While debate about which profession has the best evidence may be unhelpful, we are learning a great deal from attempts by those involved in health care intervention about improving access to research.
There are two distinct but overlapping issues discussed in a recent paper in the British Journal of Educational Studies by Phil Davies (1999). The first is making better use of existing evidence which includes ensuring access, determining its relevance and judging its quality. The second is establishing better quality evidence in the future. The first should influence the second more extensively than at present, by ensuring that researchers and funders have access to all current and previous research findings and the problems and issues - both substantive and methodological - that they pose, prior to undertaking further research. Other important factors such as the adequate supply and training of high quality educational researchers and more investment in longer term research and larger comparative studies are acknowledged.
It should also be acknowledged that not all research can, or should have an immediate impact on policy and practice. In a helpful chapter reviewing the impact of research on policy and practice in key areas of education, Mortimore and Mortimore (1999) note that the school effectiveness research took a long time to impact on policy. There are likely to be many reasons for this but some areas of research will require replications, extensions and long term outcomes before it is appropriate for changes in policy and practice to be made. The focus on informing policy and practice should not be interpreted as suggesting that the role of longer term exploratory research is undesirable or unnecessary. It is only through this exploratory research that fundamental concepts are challenged and new connections made between underlying theories and frameworks. The development of better databases and systematic reviews in education should assist rather than impede this process.
The rest of this paper is concerned with improving the use made of current evidence. Primarily users of research are being defined here as other researchers, teachers, LEA staff and policy makers at national, regional and local levels. However, as with the Cochrane Collaboration in health care which has inspired this work, other users such as parents, governors and pupils could become involved. Some of the areas of research prioritised in medicine for systematic reviews have been initiated by patients seeking better quality information on diagnosis and treatment.
The Cochrane Collaboration
The Cochrane Collaboration in health care intervention prepares, maintains and disseminates systematic reviews of the effects of health care interventions. There are 15 Cochrane Centres and 50 review groups each with members from many countries. Each group is responsible for undertaking, maintaining and updating a review in their area of interest supported by the centres. Review groups are mainly focused on an area of health care e.g. pregnancy and childbirth, some focus on methodological issues e.g. qualitative methods group and others span disciplines e.g. developmental, psychosocial and learning problems group. Each group has 4-5 editors and a much larger number of reviewers from different countries representing the range of perspectives on that topic. The co-ordinating editor typically provides 20% of their time to the groups work.
The unit for evidence-informed policy and practice in education
We are currently commissioning a team to set up a unit to develop a collaboration in education similar to Cochrane. The unit will have two main functions:
1. Setting up a database of recently published, less recent but seminal and ongoing research which will be accessible at a variety of levels e.g:
comprehensive list of references on a topic which have not been subjected to quality checks, which another researcher or Phd student requires;
full systematic review of evidence in an area where each study has been subjected to quality criteria by the review group which another researcher, funder or practitioner with dedicated time may require;
half a page of key points arising from the systematic review that might be used as a basis for discussion in a school staff meeting.
These are merely possible examples to provide a flavour of what is needed. The team commissioned will develop their own suggestions in response to perceived needs.
2. Registering research review groups who provide a proposal for their work including membership, scope, time commitment and support from their institutions.
The proposed review groups will need to demonstrate that they represent international interests, research, practitioner and policy inputs, and are committed to generating relevant, reliable and up-to-date evidence which minimises bias. The DfEE will be providing some funding to the groups but the unit staff will be expected to assist the group by hand-searching of journals and grey literature, electronic searching and development opportunities such as discussions with other review groups.
NfER have been conducting a small pilot on undertaking a research synthesis along similar lines to that used in Cochrane and are providing feedback on this which will help inform future developments.
The unit team will need to work closely with the co-ordinating centre being set up by ESRC to cover evidence-informed policy developments across the whole of the social sciences. It will also need to collaborate with colleagues in the United States who are setting up an international collaboration focusing on randomly controlled trials in social, criminological and educational interventions. These developments are seen as broader but complementary to the setting up of a unit specifically to focus on education initially and with a view to expanding to further and higher education and employment later. We are working together on ensuring that the systems used are compatible and accessible to one another.
1.Developing quality criteria
The Cochrane Collaboration has developed a set of criteria for judging studies which focuses mainly, although not exclusively on randomly controlled trials. As Davies (1999) notes some kinds of research questions (perhaps those likely to begin with what or whether) can be best addressed through larger quantitative studies which may involve randomly controlled trials. Other types of research questions (the how ones) which focus on the processes in education or those requiring rich descriptions of the consequences of interventions on pupils, parents or teachers, such as studies of the consequences of school exclusion, require more qualitative research. While we would wish to encourage more large-scale quantitative studies and randomly controlled trials, high quality research in education that is predominantly qualitative will need to be included in the reviews and common criteria for judging them need to be established. Robson (1993) has argued for the strengths in combining quantitative and qualitative methods in order to complement one another and it is hoped that the research review groups will demonstrate the benefits of bringing together studies using differing methodologies.
The need to develop criteria that are commonly accepted by researchers, funders and practitioners will be an important first task for the unit. However, there are useful bases for this work such as the hierarchy of evidence provided by Harlen (1997). Furthermore, the unit will be well supported in this task by the National Educational Research Forum (membership to be announced later this month), which will provide a steer on the criteria as part of its work on developing an overall framework for research. Existing Cochrane review groups, in particular those focusing on qualitative methodology, will also be a useful source.
2. Establishing support from researchers
In order to become effective, the database and synthesis groups will need the cooperation and support of most educational researchers. Some are understandably suspicious that these developments are based on the Governments desire for greater control or power over educational research. After many years in which little notice was taken of evidence in policy-making (Hargreaves, 1994) and the Department of Education was known by some as a knowledge - free zone, it is unsurprising that some researchers are finding it hard to enter into a dialogue with policy-makers about research. The role of the Government in this initiative is to lead and establish national coherence in the approach, not to assume control of the research agenda or the overall research programme. Establishing the database and research synthesis groups require researchers to share information and collaborate which they may have reservations about doing in the context of competition for research funding.
The commitment to funding longer term projects, longitudinal studies and dedicated research centres poses another challenge for researchers. During a 3 or 5 year programme of research the researchers will need to share interim findings with the user communities. Decisions cannot always wait for a project to be completed, yet historically researchers have been reticent to reveal interim findings. Those of us who have conducted research over many years may find if we revisited, let alone reanalysed our earlier work, that our interpretations or conclusions may be different. If research is to build on earlier findings it is inevitable that our conclusions in any given area change over time whether within one longer study or over many studies in the same area. The proposed changes require more open, ongoing dialogue and greater trust between researchers and between researchers and funders.
3. Establishing support from policy-makers
We have a considerable challenge in the Department to create a culture of expectation that policy-makers will ask for, consider and use research findings in their work. This needs to happen both in formulating policy and in evaluating policy implementation. Recent comments from ministers in speeches and the commitment to developing the unit to promote better use of evidence suggests that they would rather have the evidence than be protected from it even when it conflicts with policy or is inconclusive.
There have been notable recent examples of policy being informed by research, although the link has not always been made explicit. For example, the report by Maurice Galton, John Gray and Jean Rudduck (1999) on transition and transfer which has informed guidance on the Standards Fund, the implementation of a common transfer form and QCAs work on bridging units that pupils begin in year 6 and complete in the new school in year 7. Carol McGuinnesss (1999) review of thinking skills has informed the National Curriculum review and the model schemes of work. Both these studies have identified important areas for future research. The University of Manchester study (Farrell, Balshaw and Polat, 1999) of the career structure, training opportunities and effective use of classroom assistants reported at this conference and being published later this month is informing guidance, developments in training and establishing greater consistency in career structures.
Research commissioned specifically to evaluate current policies include evaluations of the literacy and numeracy strategies, Beacon schools, EAZs and provision for pupils from ethnic minorities all of which are providing ongoing formative feedback for policy development. The target should be to make explicit in every policy or guidance document, the research evidence that informed the content.
4. Improving the quality and availability of evidence
Evidence-informed policy or practice does not mean picking up research findings and automatically translating them into policy or practice. Hence, our choice of the term evidence-informed rather than evidence-based policy and practice. As David Hargreaves (1998) and others have suggested teachers do not pick up research findings and implement them directly into the classroom. Presented with evidence, some teachers will mediate or process the findings according to the specific context (pupils, resources, staffing) in which they are teaching. Similarly, policy-makers translate evidence in the context of the policy process taking into account resource and political implications.
In some areas the research evidence is inconclusive. For example, research on pupil grouping does not provide clear overall messages about how to group pupils to maximise outcomes. Ideological positions on setting and mixed-ability teaching have created a false dichotomy which is unhelpful. Researchers need to help others clarify the combination of approaches to pupil grouping which is most effective in each context. Research on the benefits of school-community links comes mainly from studies that are characterised by small-scale, localised projects evaluated by those funding or providing them, in other words those with a strong vested interest.
In other areas the research evidence is unavailable either because it does not exist or occasionally because it has not been published or only appears in a unknown source. The cost effectiveness of educational interventions is an example of an area in which until very recently there was no research evidence at all and those working in the area are grappling with the need to adapt a methodology developed in economics to education. The DfEE is commissioning a dedicated research centre in the economics of education which we hope will stimulate further work in this important field.
Some research findings are impractical to implement. For example, research over many years has established the benefits of immersion schemes for learning foreign modern languages in the primary years, but the shortage of specialist language teachers in secondary schools makes it impractical to attempt to provide this opportunity in every primary school.
The research synthesis groups in developing and updating their reviews will make a significant contribution to tackling these issues. The reviews will identify key findings and any truly inconclusive areas. They will assist in identifying priorities for extensions and replications of existing work as well as for new areas. Crucially, they will ensure better access to existing findings and provide assurances of minimising bias for those who need the information.
5. Development opportunities in the skills of using and doing research
Practitioners and policy-makers may get access to research and not feel able to make sense of it. Researchers employed in government departments help policy-makers use research by interpreting findings, relating them to connected policies, judging findings in terms of significance and helping draw out practical implications. Some teachers who have been involved in action research, undertaken higher degrees or more recently have had TTA teacher research grants or have been part of one of the school-based research consortia have had the opportunity to develop their research skills. The Governments green paper Teachers meeting the challenge of change (DfEE, 1998) provides the basis for exploring more opportunities for staff development for teachers including those involving research. Policy-makers may need similar opportunities. Researchers themselves need to look critically at the way they communicate and decide whether they should work collaboratively with others to communicate more accessibly.
Recently we have been involved with the Teacher Training Agency in setting up a teacher research panel of teachers with substantial research experience who can assist the government agencies and others to identify appropriate priorities, provide a user perspective on research applications, take part in research project steering groups and provide feedback on how research is influencing practice. This will provide an important development opportunity for the teachers involved but could, through the dialogue generated, provide an opportunity for researchers, funders and policy-makers to develop a better overview of what is required. This involvement of users is also a feature of the ESRC Teaching and Learning Initiative and the Education Panel for the next Research Assessment Exercise.
There are a few isolated examples of pupils being actively involved in research. Fielding and Sims-Fielding (1998) describe an initiative at Sharnbrook Upper School in which students have been actively involved in the school improvement process as researchers. With some support on research skills, they have undertaken surveys, interviews and other types of data collection in order to investigate an issue identified by the student body. They then present their findings to staff and students with recommendations of changes needed. One recent example of their impact on school improvement was in the area of personal, social and health education in which as a result of the data collected they redesigned the syllabus and this was adopted by the school.
An invitation to contribute
This paper has outlined briefly some of the current national developments that will contribute to improvements in evidence-informed policy and practice. If they are successful, further challenges will emerge. For example, the ultimate success criteria for the database and review groups might be to find pupils accessing the best evidence on classroom practice in order to challenge their teachers. The same information that gives rise to challenge should provide greater confidence, but not complacency, that decision-making at every level can be done in the knowledge of the best possible evidence. This requires an ever-expanding range of research syntheses to be undertaken and updated and everyone to be committed to using the information.
If you would like to contribute to this process there are a number of ways in which you can do so. These are some suggestions made at the meeting held on evidence-informed policy and practice in July 1999 at the School of Public Policy:
Identify a group of key researchers (international) who are working in your area of interest;
Volunteer to be a co-ordinating editor, editor or reviewer in your specific area of expertise;
Offer to assist in searches for reviews by hand or electronically;
Offer to read drafts or copy edit and comment on accessibility;
Organise study leave for practitioners with whom you work to enable them to be involved;
Develop and participate in training programmes on research skills;
Comment on the forthcoming consultation from HEFCE on the criteria for the Research Assessment Exercise to ensure reviews are adequately valued;
Explore cross-disciplinary developments - social work, criminology and others are involved in similar initiatives;
Participate in advisory committees;
Offer moral and intellectual support to your colleagues who invest time in these developments.
Davies, P. (1999) What is evidence-based education? British Journal of Educational Studies 47 108-121.
DfEE (1998) Teachers Meeting the Challenge of Change London: DfEE
Farrell, P., Balshaw, M. and Politz, F. (1999) The Management, Role and Training of Learning Support Assistants (LSAs). Research Report 161 London: DfEE
Fielding, M. and Sims-Fielding, E. (1998) Evaluation of the students as researchers initiative: Interim Report, Sharnbrook Upper School, Bedford
Furlong, J. (1998) Educational research: meeting the challenge. An inaugural lecture, University of Bristol, 30 April 1998
Galton, M., Gray, J. and Rudduck, J. (1999) The Impact of School Transitions and Transfers on Pupil Progress and Attainment Research Report 131 London: DfEE
Gray, J. (1998) The contribution of educational research to the cause ofschool improvement. Professorial Lecture, Institute of Education, University of London, 29 April 1998
Harlen, W. (1997) Educational research and educational reform In: S.Hegarty (ed.) The Role of Research in Mature Education Systems Slough: NfER
Hargreaves, D. (1994) The Mosaic of Learning: Schools and Teachers for the Next Century London: DEMOS
Hargreaves, D. (1996) Teaching as a research-based profession: possibilities and prospects. The Teacher Training Agency Annual Lecture 1996
Hargreaves, D. (1998) Creating Professionalism: the Role of Teachers in the Knowledge Society London: DEMOS
Hillage, J., Pearson, R., Anderson, A. and Tamkin, P. (1998) Excellence in Research on Schools London: DfEE
McGuinness, C. (1999) From Thinking Skills to Thinking Classrooms: a review and evaluation of approaches for developing pupils thinking skills Research Report 115 London: DfEE
Mortimore, P. and Mortimore, J. (1999) Does educational research influence policy or practice? In: I. Abbott (ed.) The Future of Education Research London: Falmer
Robson, C. (1993) Real World Research Oxford: Blackwell
Rudduck, J. and McIntyre, D. (eds.) (1998) Challenges for Educational Research London: Paul Chapman
This document was added to the Education-line database 09 September 1999