Accumulating the evidence base for educational practice: our respective responsibilities
Head of Effective Practices and Research Dissemination, Teacher Training Agency
Paper presented at the British Educational Research Association Annual conference, 16-18 September 2004
During consultations with UCET in Spring 2002, we were asked to undertake a wide range of work to support teacher trainers. The following is a report on what has been done.
Developments both nationally and internationally are providing the teacher educator community with new challenges and opportunities. I am going to focus on what we can do together to build and make accessible the evidence base for practice in ITE.
I will start by giving the rationale for TTA interest in building the evidence base for practice, and then report on new ITE support work related to the research agenda which has been undertaken by the TTA since the consultation in 2002. Following this I shall outline international developments and initiatives in other professions related to building an evidence base for practice, which I think you will find interesting.
Moving on, I will explore the contribution that research synthesis methods or systematic review methods, as they are generally called in the UK, may make in achieving the goal of providing a more accessible evidence base for practice.
Finally, I would like you to consider your role, as educational researchers and teacher educators, in taking forward the systematic review process.
TTA remit, resources, national policy
The TTA remit in short, is to work with providers to improve the quality of training and to ensure a sufficient supply of qualified teachers.
OFSTED data tells us that the quality of teacher training is now higher than ever, and that the quality of educational practice of newly qualified teachers is close to that of experienced teachers. Despite this we cannot become complacent. We have an increasingly complex and diverse society, which requires members of our communities to be highly educated.
We must also be aware of the changing context of teachers. The government now focuses on personalised learning. This is a recognition that the under achievement of individual children is an unacceptable loss to our society. The remodelled workforce agenda places the teacher at the heart of a multidisciplinary team of experts. Both these initiatives require teachers to have access to finely tuned pedagogical knowledge, diagnostic tools and subject specific specialist advice.
The TTA can argue for resources and influence national policy but we rely on you to ensure that new knowledge about effective practice in schools and HEIs is researched, documented and published.Colleagues with contracts which include an obligation to research are those whom our society is paying to identify effective practice in schools and to validate and publish that practice. Those involved in teacher education have the role of inducting new teachers into the most effective ways of teaching that we know about.
It is important to our society that people like you, teacher educators, researchers and teachers, are at the cutting edge of knowledge about effective practice, working with our new and existing teachers on the implementation of new practice. Your role is critical in moving the profession forwards, towards evidence informed practice.
21C educational practice
I want to take a little time to describe the changes in the context in which teacher educators are now working, which support the move to building an evidence base for practice. Developments in technology offer our profession great potential support for practice in ITE.
In the last decade for instance, teacher educators have moved from a position of having little access to electronic networks, to having them available on a daily basis.
We have moved from a historical oral tradition, where accessing evidence about what worked in education and up to date knowledge was done through word of mouth, learning from colleagues, journals and texts and attending conferences (if you knew about them), to a position where electronic networking supports new ways of working.
This new technology is supporting ways of codifying or recording the knowledge base, of supporting increased coherence in Research and Development and of developing and accessing professional knowledge that we could not access before.
It is the maximising of this potential that I invite you to work with us on. In teacher education, we are just at the beginning of harnessing this technology to support us in moving from the historical oral tradition, to a position where we can claim that we have evidence informed policy and practice.
Putting this goal into practice is by no means easy. The question of which evidence, whose evidence, and the quality of the evidence, coupled with the massive numbers of research papers available which may or may not be relevant, make this for many policy makers and practitioners an unachievable goal. We recently contracted teacher educators to undertake a systematic review of the evidence base for practice in primary Modern Foreign Languages. They identified over 5,000 studies of possible relevance. Think about the cost of these studies, the duplication, and the inaccessibility of the gems of effective practice that there must be there within.
Technology supports us in managing the provisionality of professional knowledge, for example with the rapid updating of files. It also has the potential to support us much more if we work together in gathering what we know.
All of us already use this technology to help our professional practice. Different institutions have taken on the challenge of using the technology to support teacher education in various ways. What I am tasked to do is to work with interested groups in harnessing the potential of the technology to support ITE at a national level. Building an open access evidence base is a task where if we work collectively, we are capable of creating something far more substantial than if we work alone.
There is an ongoing professional debate about the forms of knowledge involved in teacher education and I acknowledge that any electronic evidence base will be limited in what it can provide. But rather than concentrate on difficulties, we should focus on what we know the technology is particularly useful for.
It is from this positive standpoint that I invite you to tackle this possibility: How could such a resource be structured? What do we already have that partly meets the needs I have identified? How can we move forward to maximize the potential of the technology to support our profession?
TTA Support over the last couple of years
I now want to move to the progress report for the TTA Support Group work over the last couple of years.
The TTA Support Group has implemented a number of measures over the last two years. At an organisational level, forms of post OFSTED support, support for new providers and school based support are available which were not before. There has been a focus on supporting the implementation of the standards with workshops and a-v- material distributed. In addition, there has been allocation of funds to ICT equipment and workshops to share effective practices.
Then there have been subject focused and cross-curricular initiatives, constructed around theories of how professionals learn; the IPRN initiatives and the induction packs and programmes. Information about much of this work is already on the TTA website.
To bring together resources for teacher education, we have advertised for a database to be developed to hold high quality ITE resources a teacher education resource bank. The development of this resource will be overseen by a project management board with stakeholder representatives, and I would like to invite UCET to be a partner with us and others to build this resource, which will not only house and reference text-based materials, but also do the same for audiovisual materials. The aim, broadly, is to improve access to the knowledge base.
Using this resource, someone new to Teacher Education could find a system where detailed questions can be answered through phone or email, where quality assured text and A V resources are either available or referenced, and where users will be provided with syntheses of the research which show an audit trail from synthesis to the relevant studies identified.
The intention of the TTA is to focus the content of the resource bank sharply on what are particularly high quality and relevant resources. Current TTA projects with outcomes which will be accessible through the resource bank include:
- ITT professional resource network – collaborations of ITE professionals bringing together what is known about effective practice and commissioning new work to fill in gaps in knowledge.
- R&D projects – up to £10,000 for innovative projects – better than 1 in three chance of being selected. Also regional projects for partnership.
- Subject induction packs and programmes: supported by expert input across a series of seminars
- Systematic Reviews
I would now like to focus on systematic reviews.
The DFES and TTA systematic review programme is managed by the Evidence Based Policy and Practice Institute (EPPI) at the Institute of Education in London following a competititive tendering process across Europe. EPPI has a formal link with the international initiatives in systematic reviewing thorugh the Campbell and Cochrane Collaborations.
Following a recent DFES review of the first phase of EPPI centre work, it is likely that it will continue to be funded at least partly by the DFES, and that there will be more stress on evaluating the usefulness of the outcomes.
How is systematic reviewing relevant to ITE?
Building the skill base in ITE
Following a pilot phase testing the relevance of processes for ITE staff, and to build knowledge quickly in key policy areas, we have funded 8 review groups to work over three years covering a range of questions. (other organisations are also funding such reviews).
We are supporting this initiative for particular strategic reasons:
- the areas of work are considered important by those consulted
- training, development and capacity building: we wanted to ensure there was a large body of ITE staff who were familiar with the process as it is in 2003/4 in order to help shape the processes which will be used in 2004/5 onwards
- we wanted to experiment with the knowledge building opportunities created by cross-institutional working groups. They are also tasked with building the capacity of less experienced staff
- we want to see if the process has positive outcomes in building knowledge about effective practice:
The systematic reviewing process has the potential of providing access to quality assured summaries of what the best state of knowledge is in a particular area, at a given point in time. The outcomes support evidence-informed practice in a unique way. New teacher educators will be able to access the accumulated knowledge of the professor much more easily.
Where teacher educators take part in review groups, they themselves can expect to become deeply knowledgeable in specific areas. As they understand the process, they can be expected to understand how the outcomes of other reviews can be used to inform their own practice.
Processes of systematic reviewing are, in some institutions, superceding the traditional forms of literature review that students undertake in MA, MPhil and PhD programmes.
Accessing the giants of educational thinking
In the seventeenth century, Sir Issac Newton, known for his discovery of gravity, made a statement which I feel is relevant to what we need to achieve in education today;
'If I have seen further than others, it is because I have stood on the shoulders of giants'.
I would now like you to consider how your staff, new to the training of teachers, know who the giants providing evidence for educational practice are today. (various routes to ITE teaching mean ITE staff come with various backgrounds). I have received feedback about staff taking up new posts and finding nothing to guide them in their new work. This is across old and new universities alike, and many ITE staff are sole practitioners in their subject area.
So, how do our colleagues new to teacher training access the accumulated knowledge we have as an education community? And how do ITE staff keep up to date across areas that they wish to know about, but which are not the main focus of their research and reading? Furthermore, what can we do collectively to improve the situation for new staff in particular?
Having explained briefly what has been done so far and the challenges that face us, I would now like to turn to what is happening in other professions and national and international developments.
We have a model from health professionals showing how they are going about bringing together evidence about practice:
Dentistry: an unlikely choice. My dentist recently said to me that if I wanted to review the evidence about different types of fillings, then I should just visit the British Dental Association’s website. In my community, I was asked to chair a debate about fluoridation, and I was astonished to find the dentists were citing the outcomes in the systematic review of evidence about fluoridation. I predict that in 10 years, teachers will be able to do the equivalent.
In the NHS, there is a national electronic library of health, a research news service (bandolier), telephone support for clients, systematic review centre contracts being let now.
In addition, there is the international Cochrane Collaboration: a group of 10,000 professionals from over 89 countries, who have worked over 10 years in building the Cochrane Library. The library contains thousands of reviews of the evidence from research about various types of medical practice. We in the UK, have free access to this service, paid for by the National Electronic Library of Health. The money provides a revenue stream for the independent Cochrane Collaboration.
The ‘Cochrane Collaboration' is organised into 50 centres, each of which receives support for administration. As I mentioned above, funds are provided to support the central administration but practitioners of all types, nurses, doctors and para-professionals take part in reviewing and updating the professional knowledge base as part of their professional work. As well as a secretariat in Oxford (funded through publications sales) there are sub-groups dealing with membership and publishing policy.
It should be a comfort to us all that our health professionals are so organised. The thought about being operated on by someone who is not up-to-date with recent practice, or who considers intuitive practice to be sufficient is not one any of us would welcome.
But is the position about the need for evidence any different in education? Think about the type of teacher you wish your children had. We want their teachers to have a range of qualities, for them to be kind and to understand the fragility of a child’s self esteem. But don’t wealso want them to be up-to-date? To base their decisions about how to help children learn on the best evidence we have?
Key messages from healh care professionals
The key message from the Cochrane Collaboration is that professionals can work together to manage the establishment of a library of professional evidence, and that there are various funding models which can support such an initiative.
We certainly would not accept from medicine, the level of intuitive practice that we currently accept in education.
You might argue that medicine is dealing with the body, and the mind is different. I would agree, but only in some respects. In medicine, mistakes in everyday practice are readily detectable, and can be very serious, leading to death or physical injury. In education, the impact of poor practice is hidden, but it can have a devastating and life long impact. Less well educated people tend to have a lower life expectancy.
So I ask you, is building processes for accumulating and accessing knowledge from research an optional activity for academics? For Government? Those who are sceptical that we can achieve anything are welcome to watch from the sidelines. I am looking for people who are willing to take a risk, to build a big vision for the future.
The Campbell Collaboration builds on the foundations laid by the Cochrane Collaboration. (The EPPI centre has a formal relationship with the Campbell Collaboration).
The Campbell Collaboration is an international professional organisation developing standards and methods for synthesising the outcomes of research studies to support accumulation of knowledge from research. Their reviews focus on establishing what works. Areas covered are crime and justice, social welfare and education.
The fourth conference of the Campbell Collaboration was in February this year, about 20 countries were represented including 24 government departments, 26 research centres and 50 universities. The next conference will be in February 2005 in Lisbon.
I went to the Campbell Collaboration Conference because I wanted to:
- #9; understand fully, international work on establishing the evidence base for educational practice;
- #9; find other colleagues working at the policy level with an interest in the area;
- #9; identify UK academics with interest in research synthesis;
- #9; benchmark the TTA systematic review strategy against international thinking in education and in other disciplines.
The Campbell Collaboration’s vision for the future knowledge base in education is that research syntheses/systematic reviews will be developed in many areas, will be regularly updated by academics who claim expertise in the area, and will support evidence informed practice and policy making. We therefore need to focus on how to collectively meet this challenge, and develop national processes that will enable us to contribute to this international initiative in order that the outcomes will benefit us nationally.
I believe that we need to support the development of a UK strategy in this area through consultation withthose currently undertaking reviews – probably several hundred of our colleagues, and with BERA, UCET, DFES, GTCE and the Educational Research Funders Forum.
We should for example, be aiming, over a ten year period, to achieve what the Cochrane Collaboration has achieved. A visit to the Cochrane Collaboration in Oxford with DFES, GTCE, BERA and UCET colleagues is planned for October 2004.
But how does all of this fit in with the RAE? At the Campbell Collaboration conference I asked colleagues from different countries how systematic reviewing or research synthesis counted in their research assessment of exercises. The answers varied.
In some instances it is highly rated, in others less highly rated. What does seem to be important for the academics concerned is to publish the outcomes of the synthesis through the traditional routes i.e. journals. We all have an opportunity to influence the structure of the next RAE and we will certainly be stressing the national importance of systematic reviews in our response. We will also be working with interested colleagues and DFES colleagues on how to ensure the outputs of systematic reviews are usable and accessible by the different groups to whom the outcomes are relevant.
Michael Fullan, who has written so much on managing change and raising quality in education, provides a Canadian perspective. Writing in 1993, he challenged the teacher education community to identify the evidence underpinning their practice.
At the same time, in the US, the ERIC database was being set up. As you are probably aware, this work was stopped last year, the Government withdrew the funding and a new approach is being developed including the ‘What works in education’ database which focuses on quantitative data.
The Campbell Collaboration has been established since Fullan wrote.
Let me now run over what has been happening nationally.
Since the publication of the Hillage report, reviewing the impact and extent of educational research, there have been a number of developments which are intended to increase the value of educational research in influencing practice. I would argue that the practice we have developed is in advance of what has so far been achieved by the Campbell Collaboration in Education. But we still have a long way to travel.
One of the outcomes of this report was the establishing of the National Educational Research Forum.Establishing the Forum has been a major step forward in bringing new coherence to the use of resources for educational research. All major funders, government and charities, sit on this body and recent discussions have resulted in support for the systematic reviewing process and discussions about how to support this process nationally.
With regard to the RAE, the Funders’ Forum and other national supporters will work hard to ensure recognition of systematic reviews as a high level research activity.
Systematic reviews and other methods of reviewing the evidence base for practice
I shall now explain what I mean by systematic review and the difference between this approach and the traditional ways we have generated evidence for practice. Traditionally we have used research and literature reviews to produce evidence for practice and we have experimented with various forms of teacher researcher approaches.
Emerging models of work in this area are providing much wider access to evidence than was possible before. The stress now is making our research findings available to the user.
I mentioned earlier that the systematic review for Modern Foreign Languages primary, found 5000 articles. No single professional can make sense of that in the time they have available. Clearly we need some sort of synthesising process to support practitioners to make sense of this research.
Traditional reviews have a number of limitations when compared with the systematic review process.
We can no longer justify spending scarce research resources on a multitude of small studies. It just doesn’t make sense for academics who wish to deepen their expertise or for funders who want to ensure that new knowledge is being created.
But in order to take on the challenge of setting up structures like the Cochrane Collaboration; to involve a wide range of interested staff in systematic review groups, and to build a library of evidence or effective practice, we require the development of new processes. These processes involve dividing up areas for review. I think these processes will have significant benefits for teacher educators and educational researchers alike.
Other professions have much more sophisticated expert structures ours in education. We have a more haphazard structure. Schools may or may not find experts, HEI tutors may or may not be experts, and they in turn may or may not be able to find experts. The models we could adopt following Cochrane would mean that finding the experts and indeed publicising your own expert status would be supported.
What I call ‘fundamental reviews’ are major studies undertaken in areas where there are no existing reviews. These are costly and time consuming.
However, once these are done, different models of review emerge that are ‘thin slice’ or ‘top up’ reviews. This I would see in the future as being done by academic review groups, consisting of people who have a claim for expertise in the area, and through their normal academic work in keeping up with their subject, they will, collaboratively, keep reviews up to date.
Teachers who are interested in being research active would also undoubtedly be interested in involvement in ongoing systematic review groups. This approach may provide an interesting framework for research engagement by teachers and schools which is currently missing.
We are at a time where the future is being shaped. Right now, processes and procedures can be shaped. We are not expecting that all aspects of practice can be covered, but we are prepared to try out the processes to see what is valuable about this initiative. Let me also make this quite clear. Qualitative studies and quantitative studies are all able to be included. There is no return to paradigm wars.
Key Players: their roles and responsibilities.
The TTA may be able, with others, to find resources, we then need the help of professionals to turn the vision into reality. Your role is crucial. Here now then is a summary of my vision of where we might be in 10 years time.
It is recognised across disciplines, that it is time to change. Now that we are in the 21st century, professional practice and policy making should be based on the best evidence available. There is a need to take stock of what evidence exists, and what the quality of that evidence is. There is a plethora of individual studies, and now a synthesis is necessary to extrapolate the evidence provided from these which should inform practice. In addition, the gaps in knowledge need to be more easily identified.
User involvement is essential in achieving a transformation of practice. A model of systematic review groups taking responsibility for particular areas is actually based on theories of expert professional learning: the communal construction of knowledge.
The methods used for research synthesis do encompass evidence from qualitative and quantitative data. There are misunderstandings amongst academics that not all forms of research are included in reviews. In fact, the choice of what types of evidence to include is down to the review group.
There are quality issues. All disciplines report that it can be difficult to establish the quality of the research process from articles and abstracts. Improvements in the ways research is reported are needed. (The EPPI centre has been commissioned to produce guidelines for journal editors and article writers in the UK. These will be ready by summer 2004. The February 2004 Educational Researcher, the AREA journal contains a publication about article abstracts for the same purpose in mind).
I also anticipate in years to come, funding more systematic reviews, particularly in specific subject areas. We need giants in every subject area, every cross curricular theme, if we are to claim a profession working from an evidence base. The approaches I am supporting, to set up a national network of systematic review groups, will, I think, enable new ITE staff to find an ‘expert niche’ for themselves and will enable existing experts to have their expertise more widely recognised.
So then, is demonstrating the evidence base for practice going to remain optional for educators? I do not think that educational professionals can expect to be exempt from the requirement to demonstrate evidence informed practice. If we are to have a national electronic library providing evidence for practice, then who will take it forward? We need one group, or a collaboration between interested parties to do this.
In conclusion, the key messages I want you to take away are
The challenge for each one of us is whether we stay on the footpath and let the bus pass by or whether we jump on the bus even though we are not entirely sure what the destination is.
Marilyn Leask brought to the TTA,
- her experience in teacher education – across primary and secondary programmes, short and long courses, as a lecturer, researcher and manager;
- Her experience in LEAs and schools – both rural and inner London.
She was a teacher- researcher in the Enfield/Cambridge Institute TVEI evaluation scheme in the 80s which she went on to run for the LEA.
Since then, she has held research posts in old and new universities and at NFER. Her MPhil is based on an analysis of the reliability and validity of teacher-researcher approach and her PhD provides an analysis of strategies for whole system change in education.
Some of you will be aware of the Routledge Learning to Teach in the Secondary School series of texts which she initiated and now edits.
Much of her research in the years before going to the TTA was about the use of ICT in supporting professional development of educators. Her role at the TTA is focused on establishing the knowledge base for educational practice and using ICT to disseminate this.