In 1995 the Scottish Council for Research in Education was commissioned by the Scottish Office Education and Industry Departments (which have since amalgamated) to evaluate the National Record of Achievement (NRA) in Scotland. The research we undertook was mainly quantitative (including for example a census of all secondary schools), but we were asked to identify models of good practice and this was addressed by carrying out a small number of interviews with staff and pupils in four schools recommended as sites of good practice. This paper describes and analyses our experiences of addressing this aspect of the evaluation, which proved to be contentious with our sponsors and those responsible for developing the NRA in Scotland. We also include a discussion of the concept of 'good practice' in a more general sense.
The NRA and the evaluation
The NRA is a UK-wide initiative introduced jointly by the Employment Department and Department for Education and Science, together with the Scottish Office and Welsh Office, in February 1991. The aim of the NRA is to provide a standard presentational format to summarise an individual's achievements in education, training and throughout life. A considerable amount of development work and research in profiling and records of achievement had been undertaken with government funding during the 1980s (Brown and Black, 1988; PRAISE, 1988, RANSC, 1988). When the NRA was launched, in policy terms it did not build as closely as might have been expected on this previous work (James, 1995). However, in practice, within the schools, it was seen as an extension of what had gone before rather than a new departure. The difference between the new policy and continuing practice lay mainly in a difference in emphasis. Policy shifted to a greater emphasis on the NRA as a record of achievement to be used at transition points in the education system or job market, while teachers and educationists in general continued to value the NRA mainly as a tool to support the learning process.
The aim of the NRA is to provide a record of achievement throughout life. However, it was introduced first with 14-16 year old pupils in secondary schools and at the time of our evaluation was only just beginning to 'grow through' to FE, HE and employment. For this reason the evaluation only attempted to identify examples of 'good practice' in secondary schools and this is the focus of our paper. Practice in schools, nevertheless, was influenced by teachers' perceptions of a rather low take-up of the NRA post-school, particularly among employers. Our survey data showed that the majority of FE colleges were using the NRA in selection. However, few employers, training providers or institutions of higher education were using it in this way and only a very small number of employers, training providers and institutions of further and higher education were encouraging or supporting young people to continue to use the NRA for their own personal development (Somekh, Tinklin, Edwards and Mackay, 1996).
Identifying examples of good practice
We approached key individuals in different regions and asked them to suggest schools that they considered to represent examples of good practice in their use of the NRA. At this stage, we had accepted the term unquestioningly. We had not clarified with the funders how they defined good practice and we did not clarify with our informants what we considered to constitute good practice. A number of schools were suggested to us and a short telephone interview was carried out with each of them to find out about their use of the NRA. On the basis of this information, we chose four schools in different regions which were using interesting and disparate approaches to the NRA. Each school was visited. Staff and pupils were interviewed over the course of a day and documentation relevant to the NRA was collected. On the basis of this information, we produced descriptions of practices relevant to the NRA and the views of staff and pupils in the four schools.
Initially, we did not expect to run into problems in identifying models of good practice. In retrospect it is difficult to understand why this was the case. It is certainly true to say that we were not surprised that this task was built into the specification of the evaluation. We accepted it as part of our brief because we knew it was important to our sponsors. Perhaps we were guilty of believing that we could provide short descriptions and 'keep things simple.' Probably we were lulled into acceptance simply because the term 'good practice' is so commonly used. It has become a central part of the discourse of education policy-makers and has therefore acquired some of the meaninglessness of a cliché. For whatever reason, throughout this data collection stage there was a tacit acceptance on the part of the research team that these schools provided examples of good practice.
It was only when we took our initial descriptions of practice in these schools to our advisory committee for comment that the problems with the term good practice really started to emerge. The advisory committee for the project was made up of Scottish Office representatives and representatives from industry, teaching and other bodies with an interest in the NRA (TVEI, HMI).
Several members of the committee were unhappy when we presented one of these descriptions to them as an 'example of good practice'. These schools had all been recommended to us as examples of good practice, yet some members of our advisory committee did not recognise good practice in our accounts. The schools themselves had also seemed uncomfortable with the term. They were initially pleased that what they were doing had been recognised as good practice. However, they were quick to point out the flaws in what they were doing and to talk about the gap between their ideal aims and what they could actually manage given the limitations of time and resources.
The problem lay, perhaps, in the title we had given to our accounts. 'Example of good practice in the NRA' was likely to suggest to readers something rather different from a research study of a school recommended as a site of good practice in the NRA. We had written full descriptions of NRA practices in the schools. We had not isolated certain practices and labelled them good. This meant that areas of difficulty and problems were also described. All four of the schools were anxious to point out that they were still developing the NRA and what they were doing at that time was a compromise between their own vision for it and what they could practically accomplish given the limitations of time, resources and the competing demands of other requirements. Members of our advisory committee were possibly reacting to our labelling these 'warts and all' accounts as examples of good practice. With hindsight, we were unwise to give this label to our accounts. Nevertheless, as educational researchers we had very different expectations of what good practice might look like. No school would be likely to match an ideal type. We would argue that there were elements of good practice in each of the accounts. In all four schools, there was a high degree of commitment to the aims of the NRA and the staff had a very good understanding of the purpose of the NRA and had found ways of integrating its practice with other on-going work.
The problems arising at this stage were twofold: there were clearly differing viewpoints on what constituted good practice in the NRA; and there was a tension between the way in which policy-makers wanted to use examples of good practice and the aims of the researchers. The researchers were trying to present faithful representations of what was happening in a small number of schools that had been identified as sites of good practice. The policy-makers perhaps were hoping that we would present more of a blueprint that could be copied by other schools. The question here is whether it is appropriate for researchers to be requested to do this given that it is getting away from the realms of research and into the realms of promotion. We were also clear that presenting concrete examples of good practice in the NRA for schools to copy was unrealistic since schools were clearly, wherever possible, integrating the NRA with other existing and new initiatives.
After considerable discussion the difficulty was partially resolved by re-titling the accounts 'Case studies of schools'. We also included a paragraph in the report which explained the purposes of accounts such as these in an evaluation, so that readers would not be misled. However, the tension manifested itself once more when we produced the draft of our final report. There was some resistance to including our descriptions of practice in these schools in the main body of the report. We felt strongly, however, that these accounts were useful as they 'put flesh on the bones' of the quantitative data, and raised issues that were concealed by the quantitative data.
We were still left with a difficulty, however. We had agreed in our initial contract to identify examples of good practice. Once we had agreed with our sponsors that the accounts of practice in schools would be inappropriately labelled as such we felt bound to address the issue of good practice in a different way. We did this within the final report in two ways: we produced a short checklist of preliminary criteria aimed at helping schools to develop shared understanding and plan action strategies during the initial stages of implementing the NRA; and we produced a set of questions to develop understanding of the NRA. The questions were designed as a focus for staff development and were aimed at generating discussion to further the development of the NRA.
The notion of good practice
This account of our experience of addressing this aspect of the evaluation raises several issues about the concept of good practice. Our aim here is to encourage discussion of the term 'good practice' and questioning of the assumptions underlying its use. We argue that the term 'good practice' is becoming a cliché which is being used without sufficient questioning of the assumptions underlying its use. We discuss the difficulties of defining it, assert that it is always context-specific and that different stakeholders may not agree on what constitutes good practice. Finally we discuss the usefulness of identifying and disseminating examples of good practice.
The term 'good practice' is becoming a cliché. In our own case it was used without clarification of its meaning on either side. There was an assumption of a common understanding which, it became very apparent, was not the case. Papers addressing good practice in other areas also use it without a clear explanation of how exactly the author's conception of good practice has been arrived at (eg Taylor (1994), Hill (1993)). If the notion of good practice is not questioned and used more carefully, there is a danger that good practice is arrived at simply by the process of someone making a value judgement without questioning the assumptions they have made in arriving at that judgement. In simply using the term 'good practice' and requesting an evaluation team to identify examples of it there is an assumption that such a thing as good practice exists and that it is possible to define and identify it. We would like to encourage more questioning of the assumptions underlying the use of the term.
It is very difficult to define good practice. Fenstermacher (1986) in his discussion of research on teaching discusses the concept of good teaching. He makes a distinction between good and successful teaching which is taken further by Smith (1993). The basic notion is that good practice should be successful in that it meets the aims embodied in it, but also be exemplary or commendable in some respect when judged by a relevant standard. This is a useful starting point but still leaves questions unanswered. When considering whether a particular NRA practice is meeting its aims, for example, whose aims are important? Should a practice be judged relative to the aims of the policy-maker's vision for the NRA or relative to the aims of the individual practitioner? Furthermore how should relevant standards against which to judge good practice be determined? Simply defining good practice in the NRA is a huge task in itself. It requires knowledge of current practice in the NRA since we would argue that good practice can only be judged relative to existing practice and this is particularly so in the case of the NRA which is a partially implemented initiative. It also requires some mechanism for deciding on relevant criteria against which to judge practice.
Further difficulties with the notion of good practice emerge. Knight and Smith (1989) argue that good practice is always context-specific: specific to the subject being taught and specific to the current paradigm of good practice, since what was considered good practice in a subject 15 years ago will not necessarily be considered good practice today. We would go further and argue that a particular practice in the context of one classroom may not be considered good in the context of another. We argue this on the basis of our understanding of the nature of teaching practice. Teaching is not a set of techniques which can be learnt and reproduced according to set procedures. Rather, it consists of multiple interactions between a teacher and learners in a particular context (usually a particular classroom) with the teacher's intention of enabling, supporting and monitoring their learning. Teaching begins from the teacher's plans and intentions, but is framed by the learners' expectations and needs, and the 'politics' of motivation and control in the classroom. What takes place in these interactions is the result of complex teacher-pupil and pupil-pupil negotiations played out within a frame of the values and needs of teacher and pupils, who themselves mirror the values and expectations of society. The work of Doyle (1979) demonstrates that, in this context, orderly, well-organised classrooms may be places where teachers and pupils collude to exchange good behaviour for good grades and where, as a result, very little learning may be taking place. On the other hand, Doyle associates 'learning tasks' with learners experiencing ambiguity (eg they will not be sure how to tackle a problem) and risk (eg they do not know when they begin if they will be able to complete the task satisfactorily). Doyle's work suggests that we need to take great care in evaluating the quality of classroom practice, since what can be shown to be good practice in one classroom may look very similar to what can be shown to be bad practice in another.
A further problem with the notion of good practice is that different stakeholders may not agree on what constitutes good practice. What is considered good practice by teachers, for example, may not be considered good practice by their pupils. In one school that we visited the NRA process was based largely on the updating of a workbook which was started in S3 and updated through into S6 if the pupil stayed on at school. The idea was that reviewing and recording achievements should be a cyclical process which occurred several times throughout a young person's school career. On the surface, this practice seemed to be achieving the aims of the NRA: young people were being required to review their achievements, assess their own strengths and weaknesses and set themselves targets which were then reviewed at a later date as part of a continuous cycle. To policy-makers and teachers this may appear to constitute good practice in the NRA. However, when we asked a group of S5 pupils at the school to comment on this exercise, they said they had found it repetitive and they felt they had been asked the same things three times. Clearly in the eyes of these pupils this was not good practice.
In using the term good practice there is an assumption that identifying and disseminating good practice are useful exercises. The arguments for using it are:
that it raises standards;
that it provides models of practice that inspire and educate other practitioners;
that it is useful in promoting and disseminating the ideals of policy-makers.
It is easy to understand why there has been so much interest in providing 'models of good practice'. The innovation literature provides many examples to show that individuals cannot easily change their practice on the basis of a theoretical explanation of how and why to do things in a different way. The experience of trying to do something in a new way is much more powerful than an explanation in the abstract. Case studies of other people's experience can provide a vicarious experience and practical ideas for trying something out oneself. However, the problem is that examples of good practice invite no discussion or questioning. They are presented as absolutes. The new approach to police training, implemented after the MacDonald Report (Home Office, 1987), includes role play and discussion on the basis of accounts of problematic cases from police practice. This requires the teacher to have good skills in facilitation in order to support the development of understanding from a wide-ranging discussion. Exemplars of good practice, such as we were perhaps expected to produce, have the apparent advantage of presenting no mixed messages; but they have the disadvantage that they do not invite discussion and their claims to represent the ideal are nearly always contentious.
We have discussed some of the difficulties inherent in using the term good practice and in trying to identify and disseminate examples of good practice. We have come to understand that good practice in the NRA, is like good practice in any other field of human activity: it varies considerably according to context. It is, therefore, not possible to provide examples of good practice for others to copy. In his novel 'Changing Places' David Lodge makes this point in relation to 'good practice' in writing a novel. Faced with the demand that he should teach a creative writing course at Euphoria State University, Philip Swallow writes home to his wife at Rummage (Birmingham) and asks for a copy of a forgotten book on the top shelf in his office, called 'How to Write a Novel'. Every subsequent chapter of Changing Places begins with a quotation of a recommendation from this book and is then written as an example of not following the recommendation. Changing Places is a novel which obeys none of the rules of good practice in novel writing, but it is a very good novel.
Brown, S & Black, H (1988) Profiles and Records of Achievement. In: S Brown (ed) Assessment: A Changing Practice. Edinburgh: Scottish Academic Press.
Doyle, W (1979) Classroom tasks and student abilities. In: P L Peterson & H J Walberg (eds) Research on Teaching: concepts, findings and implications. National Society for the Study of Education. California: McCutchan.
Home Office (1987) Police Probationer Training: The Final Report of the Stage II Review (MacDonald, Argent, Elliott, May, Miller, Norris) London: HMSO.
Hill, F (1993) Looking for Good Practice - Case Studies of Approaches to HIV and Aids Education for 16-19 Year-Olds in Further Education. Health Education Journal, 52, 1, pp 28-33.
James, M (1995) Evaluation for Policy: Rationality and Political Reality: The Paradigm Case of PRAISE? In: R G Burgess, Educational Research and Evaluation for Policy and Practice? London and Washington: Falmer Press.
PRAISE (Broadfoot, P, James, M, McMeeking, S, Nuttall, D and Stierer, B) (1988) Records of Achievement: Report of the National Evaluation of Pilot Schemes. London: HMSO.
RANSC (1988) Records of Achievement: Report of the Records of Achievement Process of Evaluation. London: Falmer Press.
Somekh, B, Tinklin, T, Edwards, L and Mackay, R (1996) The National Record of Achievement in Scotland: an Evaluation. Edinburgh: SCRE.
Taylor, S (1994) The Cost of Good Practice. Forum, 36, 3, pp 81-82.