Lesley Reid
Department of Primary Education, University of Strathclyde, email: lesley.c.reid@strath.ac.uk

Primary school teachers and secondary school teachers of English in Scotland have developed skills in using criterion referencing to assess the writing of pupils in the 5-14 age group, using the national testing criteria framework. This paper looks at how these skills can be further developed through assessment moderating sessions, meetings where teachers work cooperatively to refine and confirm assessment judgements. Evidence was gathered in the form of interviews of a staff tutor and 6 teachers from 3 schools (2 primaries and 1 secondary) who developed assessment rubrics using a critical skills approach, for a Local Moderation project in the Assessment is for Learning Programme. Further data was gathered in the form of field notes from project team meetings, and a cluster writing moderation meeting involving 10 schools. Teachers' views were sought on how implementing the formative assessment principle of sharing assessment criteria with children, and the experience of moderating the writing produced affected teachers' views of assessment. A model is proposed for the moderating process which suggests that it is beneficial for teachers to collaborate to move beyond criterion referencing to develop personal construct referencing frameworks. It is suggested that adopting such a model would help teachers resolve a perceived tension between summative and formative assessment demands in relation to the writing curriculum.

The aim of the research undertaken for this project was to explore the ways in which working reflectively and collaboratively to produce annotated exemplars of children's writing affected teachers' perceptions of using a criterion reference framework for assessment.

1. How does the experience of implementing formative assessment principles influence teacher's views of assessment?

2. What are the benefits of using the moderation process to work collaboratively cross school and cross sector in order to share assessment understandings about children's writing?

The researcher's role was a dual one of offering mentoring support to practitioners engaged in action research studies for the Local Moderation Project and investigating teachers' experience of participation in the project. (Bassey, 2003)

Interviews were conducted at the start of the projects and towards the end as teams were writing case study reports. Meeting notes and recordings were collected during the progress of the action research project (from October 2003-October 2004)

The Assessment is for Learning Programme aims to

• encourage teachers to use formative assessment to support pupil learning
• develop procedures to quality assure teachers summative assessment judgements

It is achieving these aims through a project based structure, with individual project teams pursuing action research case study investigations

Project 5 of the Assessment is for Learning Programme is entitled 'Local Moderation' and has 'sharing the standard' purpose, in line with the programme aim of assuring the quality of teachers' summative assessment judgements. Project 5 teams are producing local portfolios of exemplification of pupil attainment in a variety of curricular areas.

The project remit was to produce a portfolio of exemplars of pupil writing which would illustrate agreed standards of attainment at specified 5-14 levels. There was a desire expressed by the project team to have a teaching and learning focus as well as a curricular focus for their action research. This was achieved through discussions with school managers and the HEI representative, and was guided by the Kings College research (Black et al. 2002, Clarke, 2002) and the stated aims of the Assessment for Learning Programme (www.LTScotland.com/assess)

In order to fulfil the requirement of generating pupil exemplar materials and to help teachers engage with formative assessment principles, it was agreed that the team would also use the generation of exemplar material as an opportunity to progress their understanding of formative assessment principles, before they engaged with the moderating activity.

The teaching and learning focus for the project team was the 'sharing success criteria' with pupils. The action research team team consisted of four primary school teachers and two secondary school English teachers. The moderating meeting involved staff from all schools in the cluster and was focused on exemplars of pupil work from P7 and S1.

The teachers implemented formative assessment principles by developing assessment rubrics with pupils, based on adopting a critical skills approach. (Weatherley, 2003) A critical skills assessment rubric is a grid which shows criteria and a progression of skills devised by pupils and teachers working together using child friendly language in order to develop next steps. The assessment rubric is used during writing to provide support in the form of next steps to allow pupils the opportunity to improve the quality of their work, as they work.

In order to address the research questions specified, transcribed interview data and meeting notes were analysed in relation to a coding framework

This analysis allowed the first research question relating to implementing formative assessment principles to be addressed.

From this analysis conclusions were drawn about how teachers progressed their understanding of the role of assessment in the writing curriculum, firstly by developing a more detailed criterion referenced framework. (Brown, S. & Simpson, M.1988) They then applied the skills developed to specific writing tasks. This led to increased subject knowledge (Wyatt-Smith, 1999) and to increased craft knowledge about the social context of teaching and learning.

A shift in responsibility for learning and assessment occurred in the classroom as pupils assumed more responsibility for their own learning (Torrance, H. & Prior, J. 1998) and participated in interpreting the curriculum in relation to specific tasks set by teachers.

Further codes were developed from the interviews to allow the second research questions to be addressed. From this analysis, it is proposed that the development of teachers' subject knowledge base and their increased craft knowledge base were extended through the moderation process to become a construct referencing system (Marshall, 2001) This development led to the proposed a model for the moderation process.

Teachers used assessment rubrics as a technique to allow them to implement the formative assessment principle of sharing success criteria with pupils. Overt use of criteria for success became an integral part of all stages of the writing lesson.

Using assessment rubrics seemed to help teachers focus their teaching aims and perceive clearer links between those and assessment criteria. Both these aspects of the teaching and learning cycle were shared with pupils.

The value of the process of constructing rubrics jointly with children lay in the translation of these aims into 'child speak'. The negotiation of success criteria and the subsequent mechanisms developed for feedback therefore led to improved quality of dialogue between and among teachers and pupils.

Increased opportunities for peer and self assessment were a natural outcome of the use of negotiated rubrics. Teachers felt that they were more in touch with learning while it was happening in the classroom. They seemed to feel more in touch with pupils as writers rather than simply assessing the product of pupils' writing. Pupils' metacognitive skills also improved.

There was evidence that the rubrics enabled pupils to focus on specific aspects of writing while they were writing, thus acknowledging both the diagnostic and formative potential of assessment.

The linear structure of the rubric seemed to encourage children to identify their own next steps rather than put a ceiling on achievement. The relationship between learning intentions planned by the teacher and the success criteria by which their achievement would be judged, was clearer for children

Pupils found the negotiated rubric structure more helpful than previous forms of criterion referenced assessment, possibly because they had a sense of ownership of the 'grid'.

Teachers' diagnostic assessment of writing was enhanced. Teachers commented on specific improvements in aspects of writing, rather than in improved overall attainment: these improvements related to both compositional skills (choice and use of language, selection and organisation of ideas) and the technical skills of transcription (punctuation, syntax). The specific improvements related directly to the focus of the negotiated rubrics and were apparent across the ability range.

There was a sense from teachers that story writing was more coherent as a result of considered thought on the part of pupil writers.

The progression and continuity between primary and secondary was viewed as more robust for children and the assessment seen as contributing to an understanding of coherence and progression.

The criterion referenced framework provided by the national testing criteria for the assessment of writing was used as a structured starting point for the teachers' analysis of pupil exemplars.

During early moderation discussions, teachers sought to achieve more detailed interpretations of the criterion statements, in order to achieve increased specificity of judgement. These discussions led to the production, first of all of 'generic' rubrics for the assessment of Imaginative Writing. Collaborating with secondary staff on the interpretations of the performance indicators seemed to have benefits for Primary teachers in that they felt they gained increased subject knowledge from the secondary staff.

For secondary teachers, the discussions were also useful, but in a different way. Firstly, they felt that they helped them to articulate and share the subject knowledge they already had. Secondary teacher also perceived another benefit, that of being exposed during moderation discussions to the knowledge primary teachers have of teaching 'craft'.

A more detailed understanding of the assessment criteria also meant that teachers felt the quality of their feedback to children had improved and was taking place in the context of a shared understanding, between teacher and pupil.

It seemed to be the primary school teacher's ability to 'fill in the classroom context' of the exemplar of pupil work that was most helpful for the secondary teacher, particularly in relation to children who were working at the lower end of the ability range.

Teachers work at moderation meetings progressed from discussions of generic rubrics to discussions about task specific rubrics.

The teachers who had been involved in developing formative assessment principles in the generation of the exemplar materials found these discussions helped them to locate assessment more readily as an integral part of the teaching, learning cycle, rather than as a bolt-on addition to it.

It became clear that the development of task specific rubrics had great potential as a formative assessment tool, and that this represented a progression in teachers understanding of assessment from a summative, criterion referenced model.

There was an awareness that individual, task specific rubrics could become a resource which was accumulated over time to build up a holistic view of 'good writing', which might help resolve tensions between formative and summative assessment.

Conclusions

The use of assessment rubrics with children seemed to enable teachers to develop their understanding of criterion referencing assessment. Initially the generic rubrics developed led to increased teacher confidence in the criterion referenced statements of the writing assessment framework.

However, jointly constructing task specific success criteria with pupils (through task- specific rubrics) enabled them to move beyond ever more detailed criterion referenced interpretations of the curriculum. The task- specific rubrics allowed assessment to become a more effective part of the teaching and learning process.

Negotiating success criteria with children, and using them throughout the teaching and learning process meant that teachers' subject knowledge became part of the shared discourse of the classroom. This 'sharing' led to improved pupil engagement with leaning and improved attainment. The process of negotiating success criteria with pupils in the form of rubrics, therefore allowed teachers to share their increased subject knowledge base with their pupils. The improved quality of the focussed dialogues that arose between teachers and pupils, and between pupils, meant that children assumed more responsibility for their own learning. Assessment became a dynamic force that was part of the social context of the classroom rather than something that was done for pupils by teachers. Teachers welcomed the power shift in the classroom that resulted from the increase in pupil responsibility. Teachers positioned themselves at the interface of the learners and the curriculum, as learning was occurring. This resulted in a redefining of previous conceptions of assessment. Assessment became a dynamic part of the teaching and learning cycle, rather than a text based activity conducted in the absence of pupils. The dynamic quality of the process was activated by the social quality of the context, the prioritising of spoken discourse that arose from the negotiations between teachers and pupils.

Familiarity with a criterion referenced framework linked to a specific curricular area therefore seems to be a facilitating factor for the development of local moderation practices.

The benefits of the moderation process varied according to the school sector background of the staff: primary teachers increased their subject knowledge, secondary teachers enhanced their craft knowledge.

The predetermined focus of the team on the formative assessment principle of sharing assessment criteria with pupils, and the pupil exemplars generated by this focus, changed the nature of the moderating discussions.

Those teachers who were part of the practitioner research team and had engaged in development work in formative assessment found the moderating discussions about the use of task specific rubrics more helpful than those about generic rubrics, and saw increased relevance for their teaching. They moved in moderating sessions from a desire for increased accuracy of definition of criteria, to discussions which explored the relationships between the teachers' shared subject knowledge base, their knowledge of the curriculum and their craft knowledge of the teaching and learning process.

This exploration can be understood as teachers developing a construct referencing system, (such as that described William 1994, 1996, 1998 and further expanded by Marshall, 2000) rather than simply extending an already robust criterion referencing system, in an ever more detailed way. William sees construct referencing as a system which relies upon a construct or intuitive understanding of what it means to be competent in a particular domain. He views it as a more accurate description of what teachers actually do when they make assessment judgement, and as different from either norm referencing or criterion referencing. Marshall explores William's notion of construct referencing by looking for a match between secondary English teachers' subject philosophies and there assessment decisions. The project teachers' experience of using critical skills assessment rubrics with pupils and of moderating the resulting written texts and rubrics together, throws further light on the how teachers actually develop construct referencing frameworks and how these differ between school sectors.

The development of construct referencing through teachers moderation discussions, allows assessment to be more fully integrated into the teaching and learning process because it draws upon the social context in which the assessment evidence was produced. This social context, explained by the moderating teacher in the context of moderating meeting, illuminates the moderation discussion, rather than obscures or prejudices it. The value of the discussion focussed on specific exemplars is precisely that the teachers can relate the work to known classroom contexts,

The teachers taking the moderating decisions are familiar with the context of the tasks set, as they arise from their own classrooms. Because of this background context knowledge, moderating teachers can make links directly into the social context of the learning process.

They represent therefore resolutions of the tension perceived by teachers between formative and summative assessment decisions. They link directly to 5-14 curricular attainment targets, yet also to contextualised examples of pupil work. They are concrete realisations of the attainment target statements- the curriculum made real.

The acknowledged strengths and development needs analysed by participant teachers have a credibility which does not exist with assessments which rely purely on text based analysis.

Previous experience of working with teachers on sessions on the moderating of writing (eg Assessment of Achievement Programme survey marking, cluster group sessions, local authority CPD sessions and Education Faculty in-service provision) suggests that teachers find this a positive experience. The opportunity to articulate interpretations of the curriculum in the context of pupil work, while calling upon previous experience is an affirming one for experienced practitioners and a way of sharing craft knowledge with less experienced colleagues, who find it reassuring and informative.

The moderating process can therefore bring together summatively driven criterion referenced assessment and 'construct' referencing' which can be seen as formatively driven. Construct referencing (Williams 1994, 1996, 1998, Marshall 2000, Tchudi 1997, Wyatt Smith 1999) draws upon craft knowledge accumulated through teachers' experience of the teaching and learning cycle as it is played out daily in classrooms. It draws upon teachers' 'on - job' experiential knowledge of how children learn and how teacher interventions can help them learn.

A Proposed Model for the Moderating Process

Traditionally, the moderating process has often involved teachers using 'blind' exemplars of pupil work, to 'tease out' definitions of the performance descriptors embedded in the National Testing Criteria framework for writing, such as those supplied by the Scottish Qualifications Authority. This leads to a perception of the 'ladder' of the curriculum (Bryce, 2000) as one where the number of rungs to be 'ascended' by children is ever increasing and their positioning of the rungs closer and closer together.

This 'ladder' model of the curriculum assumes that there is only one route from level A to level E competence. Children tend, however, to pursue individualised learning journeys, which depend upon factors such as interest, motivation and learning style as well as ability, and which are in part determined by the social context of the classroom.

It is proposed that the moderation process can be used to develop teachers' construct referenced frameworks and that this would lead in turn to improved understanding of the existing criterion referenced framework for assessing writing, and in turn for improved support for pupils individual learning journeys.

The writing criteria framework, with its levelled performance descriptors seemed to provide a sound basis for teachers to work from. Only when teachers had a shared, detailed understanding of this was they able to move towards construct referencing. Exemplar materials which are annotated to supply information about the classroom context, the task and the pupil would seem to provide a better way of sharing domain and craft knowledge than 'blind' exemplars. The process of refining teachers' shared understandings in the light of these contextualised exemplars should be seen as the first stage in the moderation process.

A further and possibly more fruitful stage involves teachers bringing exemplars of pupil writing where assessment criteria have been negotiated with pupils as part of the writing task. These agreed criteria form the assessment rubric agreed between teacher and pupil and tend to be fewer in number than the generic assessment criteria. They are task specific and the task is focused on a particular aspect of writing (e.g. author's use of language in a character description,) Teachers are therefore moderating what they might see as partial texts, rather than pieces of extended writing. Engagement with formative assessment principles allowed teachers to bring to the moderating process, exemplars of pupil work which were normal classroom tasks. These represent the teacher's creative interpretation of how the knowledge and skills described in the curricular guidelines can be translated into 'practice', of how the implied learning might be achieved.

If moderating discussions are focused on these learning episodes, then the content of teacher discussion is concentrated on the learning process; exemplars of pupil work facilitate this focusing. The moderating discussions represent, therefore, resolutions of the tension perceived by teachers between formative and summative assessment decisions. They link through teachers' planning directly to 5-14 attainment targets, yet also to contextualised examples of pupil work, to specific learning tasks. The decisions taken during moderation therefore have a credibility which does not exist with assessment which relies purely on text based analysis.

Familiarity with the national Testing Criteria framework for the assessment of writing, enabled teachers to negotiate assessment criteria for specific tasks with pupils, in accordance with a critical skills approach. The pupils response to the tasks set in relation to the assessment criteria agreed, then fed back into the teachers growing construct reference framework of what were reasonable expectations of 'quality work'. Sharing these texts and expectations during moderation sessions meant that they became part of a shared discourse of expectations, which in turn informed summative assessment judgements for summative tasks. Teachers and pupils had a much more robust understanding of the 'bald' criterion statements. The 'community of interpreters' (Marshall 2000) developing construct referencing frameworks included both pupils and teachers.

One of the acknowledged benefits of embedding formative assessment principles into the teaching and learning cycle is that it leads to an improved quality of dialogue between teachers and pupils about learning. Just as pupils need to talk to others about their learning, so do teachers need to talk to others about their teaching; local moderation should provide a structure for this to happen.

This model for the moderating process develops teacher skills in both criterion referencing and construct referencing. It avoids teachers involved in moderating meetings becoming "bogged down by criteria" (Marshall 2000). It provides a mechanism for the sharing of both domain and craft knowledge and acknowledges the value of teachers' professional judgements.

The current changes to National Testing in Scotland in writing provide an opportunity to encourage teachers to adopt a construct referencing approach alongside their existing understanding of criterion referencing. One of the 3 pieces of writing that teachers will submit as evidence of a child's achievement of a level in writing is a short, focused writer's craft piece. This should encourage teachers to plan a wide variety of writing lessons for their pupils: some will focus on producing extended texts in a variety of genre; others may focus on developing aspects of writer's craft. These focused writing lessons could provide ideal opportunities to for teachers and pupils to use critical skills' rubrics to develop construct referenced assessment frameworks, and exemplars of pupil work to use in moderating sessions. It is suggested that the resulting deepening of teachers' domain knowledge would lead to improved understanding of the summative criteria in the National Testing framework and therefore a better 'sharing of standards' as well as improved teaching and learning in the classroom.

Bassey. (2003) A vision for the future of educational research, Research Intelligence, 84, (12-15), BERA, South well, Notts*

Black, P. & William, D. (1998) Assessment and Classroom Learning, Assessment in Education 5, 1, London, Carfax

Black, P, Harrison, C, Lee C, Marshall B and Willliam, D (2002) Working inside the Black Box: Assessment for Learning in the classroom, London, School of Education, King's College

Black, P, Harrison, C, Lee C, Marshall B and Willliam, D (2002) Working inside the Black Box: Assessment for Learning in the classroom

Brown, S. (1988) Assessment, a changing practice, Edinburgh, Scottish Academic Press

Bryce, T., & Humes, W. (eds), (2000) Scottish Education, Edinburgh, Edinburgh University Press

Clarke, Shirley (2001) Unlocking Formative Assessment, London, Hodder and Staughton

Marshall, B. (2000) A rough guide to English teachers, English in Education 34 (1)

Marshall, B. (2000) English Teachers - The Unofficial Guide: Researching the Philosophies of English Teachers, London, Routledge Falmer

Marshall, B. (2001) Teachers' subject philosophies related to their assessment practice, English in Education 35 (3)

(www.LTScotland.com/assess)

Simpson, M (1999) Diagnostic and Formative Assessment in the Scottish Classroom, Scottish Education, Edinburgh, Edinburgh University Press

Torrance, H. & Prior, J. (1998) Investigating Formative Assessment Open University Press, Buckinghamshire

Weatherley, C. Bonney, B. Kerr, J. Morrison, J. (2003) Transforming Teaching and Learning, Developing Critical Skills for living and working in the 21^st century. Network Educational Press, Stafford ISBN1 85539080 www.networkpress.co.uk

William, D. (1994) Assessing Authentic Tasks: Alternatives to marking schemes. Nordic Studies in Mathematics Education, 2 (1) 48-68

William, D. (1996) 'Standards in Education: a matter of trust', The Curriculum Journal, vol.7, No. 3 pp 293-306William, D. (1998) The Validity of Teachers' Assessments

Paper presented at the 22^nd annual conference of the International Group for the Psychology of Mathematics Education, Stellenbosch, South Africa.

Wyatt- Smith, C. (1999) Reading for Assessment: how teachers ascribe meaning and value to student writing, Assessment in Education 6, 2, pp 195-223.

Wyatt- Smith, C. Teaching and assessing writing, an Australian perspective, English in Education 31 (3)

This document was added to the Education-line database on 19 February 2004